Agreeing with experts and "research grade"

This is a great question. And while I’m sure it has been discussed before it merits more discussion.

I would wager that there are a significant number of research grade observations similar to what you describe. The issue here is a tradeoff between speed and accuracy. On the side of speed, one could argue that developing a body of RG observations quickly is useful because those RG observations then become an important source of information for other users who can use them to become good identifiers.

On the side of accuracy, I am certain that experts make mistakes. And what good are RG observations if they include errors? If they are honest with themselves, I’m sure experts would welcome having other observers provide careful second opinions about the identity of observations. The entire scientific review process is built on the idea that truths are objective and revealed through careful consideration of the evidence–in fact many papers are now reviewed blind, i.e. with their authors’ names obscured so that the reviewers aren’t unduly influenced by their reputations.

This topic raises some bigger questions about how iNaturalist challenges the historical paradigm of taxonomic expertise. Science is a collective project that requires a community of practitioners who share a set of criteria and who communicate often. Science cannot be done in isolation. However, professional taxonomists have generally specialized on a narrow range of organisms, and shared their work in niche publications that are not always widely read and that are often difficult for others to find–even if you have access to a world class university library. Specialization is obviously a necessary aspect of taxonomy, but the risk is that if everyone has a narrow specialization this reduces the number of other people that can review each others work. And, as I keep repeating myself (sorry!) that’s key to doing science. In fact, when two experts work on the same taxa, they quite often end up getting into big arguments! (Ask anyone who knows taxonomists).

I’m excited about iNaturalist because it opens up this process to a much wider range of people from the beginning. Experts have an obligation within this framework to justify their identifications with objective criteria, just like everyone else. While we can learn a LOT from experts–I don’t want to discount that (I LOVE EXPERTS don’t get me wrong!)–learning from experts is different from simply deferring to their authority without critically reviewing the evidence. Experts can sometimes learn a little from us non-experts too!

15 Likes

I think it’s definitely situational, and can be somewhat dependent on how many experts are reviewing a particular taxa. For leafhoppers and planthoppers for example, there are a few of us regularly identifying (and that includes making sure the research grade observations really belong where they’re situated), so we sort of keep each other in check. For psyllids though, this sort of peer-review checks and balances thing isn’t in place because I’m essentially the only person identifying them. Because of this, I choose to often give ID explanations especially on the more difficult taxa. I let the observer know what is important for the ID, and if the observer agrees that what I’ve outlined applies to their observation, they are free to agree (or disagree! no expert is infallible, especially when it comes to photo identification). Likewise, experts that give me an explanation or reference backing up their IDs to my observations I am more likely to attempt to check and agree with if I think it’s appropriate, as I do love to learn about the things I find. Otherwise, I’ll leave it to the experts.

8 Likes

Personally, no, I never do that. My identifications represent my own opinions, and I think blindly agreeing with experts just amplifies problems when the experts make mistakes. I don’t know why people get so hung up on making things “Research Grade.” I wish we’d just called it “Arbitrary Green Label.”

My only caveat is that I do consider an expert identification to be one of several sources of information I might consider in making my own identification. E.g. if Mark Egger (a Castilleja expert) says something is Castilleja applegatei and after consulting Jepson and Calflora I’ve narrowed it down to C. applegatei and C. affinis and I’m just not sure how glandular the hairs have to be to make it C. applegatei, the fact that Mark ID’d it that ways suggests to me that the degree of glandularness in this particular photo is glandular enough for that species. Though frankly I’d probably still ask Mark to confirm that.

21 Likes

I agree with @kueda. I want my IDs to represent my own knowledge/experience and not blind agreement. I will agree in situations where I IDed to genus conservatively but had a strong guess as to species and an expert confirms that or where I had my ID down to two species and one was confirmed.

One other scenario that I will “agree” in is when an expert points out features that support the ID that I can confirm myself (like, “this individual has a white eyering, indicating it is species X”.

Otherwise, agreeing without any specific knowledge just to get an observation to “RG” doesn’t seem to be of long-term benefit.

2 Likes

On the other hand, people can blindly agree with obs on the Identify page. It is not possible to see whether the identifier is an expert or whether the identifier provided explanatory notes. Yet it is one click for what is a very superficial agreement.

3 Likes

Have you seen https://forum.inaturalist.org/t/remove-agree-button-on-identify-thumbnails-in-certain-cases/3040?

As someone who IDs a lot of taxa that no other expert might ever see, I always appreciate when the observer adds an agreement to make the ID research grade. Mostly, my goal is to add training data for the auto ID tool, which requires a lot of RG observations. The tool often performs quite poorly in invertebrates, because they are diverse and there’s usually a serious lack of taxon experts on iNat. Sure, there will be mistakes, but IMO “blind” agreements makes the whole process go a lot faster.

3 Likes

If someone provides an ID for me in a group of animals I’m not very familiar with, I’ll often end up agreeing with it. But before doing so, I’ll do enough research to make sure the ID seems at least reasonably likely, typically by Googling the suggestion to learn more about the appearance of that species. Often I will ask the person who provided the ID which details of the photograph made them decide on that ID. I’ll definitely consider how much of an expert the person seems to be as one of my criteria for agreement, but I won’t just blindly agree.

6 Likes

The other angle here is at the current time if the expert deletes their account and no one else has added the ID it just vanishes and you lose the ID. Not just gets crossed off but actually vanished. This has happened at least once and means I’m more likely to agree with the expert if the ID looks reasonable. That’s my drive more than research grade.

Relatedly, I am teaching a coworker/friend plants in the field and they are adding them to inat and I am sometimes unsure whether it’s double dipping to go and agree with his IDs if I am the primary source. But he already knows lots of plants too so it’s a kinda odd grey area from the other side.

5 Likes

Does everything a user did get deleted if they delete their account - including their observations?

1 Like

Observations, comments, and IDs are deleted. Some other things, like taxa they may have created, or flags they made, aren’t. More discussion about that here: https://forum.inaturalist.org/t/dealing-with-account-deletion/93

A post was merged into an existing topic: Dealing with Account Deletion

I feel strongly that if someone doesn’t know how to definitively ID a plant, that I want them to refrain from identifying things.

Now, there are times when I see a post and it’s a species that I don’t know how to ID to species level, but perhaps I could ID it to family or genus…and then I see the previous user’s ID and I’m able to then look up all the possible options, and how to ID them, and I can verify that the ID is correct, and in the process, learn how to ID that species. (For example, if it’s a plant, I might check BONAP’s range maps and the Plants of Pennsylvania book, which I own, and go over all possible species and read their descriptions.)

Or…in some cases the species is very distinctive and not easily confused with anything, but I just wasn’t familiar with it, and when I see the user’s ID, I’m able to consult a few sources and verify that there is enough for me to independently ID it, and now I’ve learned how to ID a new species, and in this case I will then agree with the identification.

If people are doing this sort of thing, that’s great. Not only are they contributing to iNaturalist, but they’re becoming better at ID and learning new species or other taxa!

But if they aren’t doing this work, I’d rather they just sit back and let other people do the IDing. If they have a hunch or a guess, but are not sure, they can share this verbally in the comments.

7 Likes

As someone who makes observations and identifications, as well as actually using the data (at a non-professional level), I’ve seen the different ways this affects records.

As an identifier, I’m not always going to be 100% certain when I make an ID, especially if my ID isn’t going to make the record Research Grade (I apply a higher standard when my ID is the deciding one). As a rule, I’ll either be uncertain, and say so in the comment, or confident, or certain. Unless I’m certain, it’s a bit of a concern when the observer (seemingly blindly) agrees with my ID (especially when their previous ID was Class level, or spectacularly wrong). If I am certain, I will occasionally provide links and descriptions to help the observer confirm my ID more thoroughly, and I think this helps when there are only a few identifiers for that taxon (it also gives other identifiers a frame of reference for when they disagree, or if I disagree with them).

As an observer, I will often have a more specific taxon in mind when I ID my own observation. If someone I know to be an expert confirms my thoughts, then I will agree. Otherwise I will leave my initial ID as-is, or hit up external sources and confirm that way. I don’t agree unless I have managed to convince myself, one way or another.

Finally, as someone who uses the data (in an AI-based side project), and knowing how Research Grade works, I always run my eye over the data that goes in, because I know the system often produces misidentifications, even at research grade. It would be nice to not have to do this, but I can see that improving the system will be difficult.

So how can it be improved? You could try increasing the requirements for Research Grade, but that will cause issues when there are few experts for a taxon. One thing I would suggest is adding a confirmation for when an observer tries to agree, with a short blurb on when it is appropriate (ie, “to reduce the chance of misidentification, please confirm you have taken steps to validate this ID” - Yes/Cancel). It makes it less streamlined, but I’d argue this is one situation where that’s not a bad thing.

1 Like

Thanks everyone, this is very helpful! I really appreciate the discussion. A couple of comments:

I can’t help but think that those folks who never “agree” with an experts ID unless they could make the ID themselves (to paraphrase anyway) are being a bit too purist for me. After all, I’m posting the observation to try to identify the bug, specifically hoping that someone with expertise will help me identify it, and then when a internationally known expert who has seen thousands of these critters and written the key papers on how to ID them weighs in, I shouldn’t “agree” with them??? Isn’t that sort of crazy? :) Yes, experts can be wrong, of course, but it’s pretty much as good an ID as I will ever get, unless I collected the critter and sent it to them. And there are unfortunately a number of obscure groups of insects where there may really be only one or two aging experts, so getting two folks may be too much to ask…

@kueda says that “research grade” is an arbitrary designation/name, but doesn’t “research grade” have consequences for the observation (i.e. sent over to GBIF and other databases) versus non-research grade? So it’s not just a funny name, but means something in terms of what happens to the data, right? Or am I missing something there?

Thanks all,
Ken

4 Likes

I don’t think it’s necessarily purist as such. If I ID something, I’m not bothered whether or not the observer agrees with me if they don’t really know themselves. I think a simple thanks is better, or a question along the lines of “why did you choose that?”, giving me a chance to share my knowledge. I’m even happy with no response. Why would it be purism to expect people not to agree if they don’t have any real idea of what they’re agreeing to?

I see it more as working within the system that has been built, where “Research Grade”, though arbitrary, is now used as a legitimate indicator of whether something is what it says it is. We should be working towards that understanding of Research Grade, but also remembering that it is a social/collaborative platform and we shouldn’t be too militant about it.

2 Likes

My thoughts would differ a bit depending on if you’re another identifier or the observer (this also coming from a rather niche insect identifier).

As an identifier on someone else’s observation, my suggestion on etiquette would be to refrain from ID unless you know you can rule out other related species, especially any cryptics (or know that the expert did so). So keeping rather conservative on these IDs. Example: you probably shouldn’t become a top identifier if you’re just agreeing with experts and can’t independently verify the ID.

If you’re the observer, it might be permissible to be just a tad more liberal. Have you been convinced that it’s likely this particular species? Have you at least looked at other members of the genus or family from your area? Are any details presented that further rule out possibilities (sometimes experts will make comments on this)? Would you follow notifications on the observation in case a reason why it’s some other species is noted?

2 Likes

As a sort of mid-level iNat user (neither novice nor expert identifier), I do one of the following when someone adds an ID to my observation -

  1. If it’s something I know quite a bit about, or an ID that I had a hunch of (but didn’t add because of my uncertainty and probability of biasing others’ inputs) then I will agree with it after a very quick, cursory Google search.

  2. If it’s something I know a little about and can do with a little more expertise myself, I will ask the identifier to point out the key characters, then agree to it once I know what I see. I may tag another identifier if I’m still confused.

  3. If it’s something I’m absolutely clueless about (to the point that additional info about the ID will be of little use to me), I’ll thank the identifier and tag another one from the list of top identifiers for that taxon, available on the right hand side. I will keep an observation of this kind from turning RG from my agreement, even if the first identifier is very confident.

When I’m the identifier of an observation and my input is based on a very strong but non-100% guess, I add the ID with a sentence asking the observer not to agree unless they are basing it on other strong inputs (either via other identifier(s) or via their own background reading).

Also, this is very directly connected to what RG means, or should mean - https://forum.inaturalist.org/t/rename-research-grade-discussion-and-polls/590 (you might have seen this already). I sometimes find myself wishing I could express agreement with an ID, without having the action push the observation to RG. :) I guess that’s a stretch, though.

3 Likes

That’s correct, but what is and isn’t “Research Grade,” and the choice of terminology, is somewhat arbitrary, e.g. why is it a greater than 2/3 majority and not a greater than 5/7ths majority, and why do we call it “Research Grade” and not “complete and probably accurate and we’ll share it with some other websites if they want it I guess”?

If an expert identifies my observation and I know they’re an expert, then from my perspective, it is identified. It would be nice if it got to GBIF, but I don’t care too much if it does. I’m happy to learn a name, and maybe begin to see the pattern that name describes. If my own identification is somehow contradicting the expert’s ID, e.g Mark says it’s Castilleja and my previous ID was Triphysaria, I’ll usually withdraw my own ID.

8 Likes

I’m with this guy. (And props for acknowledging the wizard that is @markegger!) I have seen posts with a chain of confirmations, and as soon as someone makes a different ID the whole chain follows suit. We discussed this elsewhere recently, but I still am very suspicious of that and attribute it to what sociologists call “confirmation bias”, basically that humans tend to favor agreement with a claim over disagreement.

I get the motivation to make things Research Grade. I want 100% of my observations to get there, and nearly as much for my region. Sometimes I don’t post things I’m not sure will get to Research Grade, not on purpose, but because of that mysterious inner compulsion. However, I don’t think it is to anyone’s advantage to compromise accuracy for agreement. Better accuracy makes the data more valuable. It’s not useful for reference or research to pull all of the RG posts of a species and find a bunch of them are incorrect, and erodes the credibility of the whole project.

I’m in favor of a conservative approach, to ID as far as I am able with the reference material at hand. Sometimes an “expert” is a part of that reference material, but with caution, because there is a level of scrutiny and effort (called scholarship) that goes into the creation of a species description, that is not universally applied by an expert scrolling through their iNat feed, looking at cell phone photos.

The counterpoint is that taxonomic work is imperfect and incomplete, and reference material only gets you a certain kind of knowledge. I’m not arguing that you shouldn’t ID unless you are somehow free of all doubt, and certain of the validity of the taxonomic structure and description. You only know what you know, and we are all still learning, so we ID away as best we can and should be satisfied with that. However, before agreeing with the proposed ID, one should at least do some due diligence to learn about that species and rule out similar-looking ones before clicking “that big green button.”

2 Likes