I’ve been thinking about ways to catch research grade observations that may have been inaccurately elevated to that status for any number or reasons over the years. I was curious if it would be possible to have the computer vision review existing research grade observations of a particular species and to flag observations that might be inappropriate. It would be a sort of “which of these things don’t belong with the others?” sort of exercise. I imagine the computer vision might provide a likelihood for each observation, and any observations that fall below a particular threshold (e.g., 50% likelihood) would be flagged for expert reviewers to look more closely at. Is such an approach possible or feasible? I did a search on the forum for similar proposals and haven’t found anyone offering the approach yet. It seems pretty straightforward, so perhaps it has already been tried and ruled out? It seems like it could be a more and more useful approach as the database grows and the computer vision relies on more and more examples to make its suggestions.
I assume you mean to use CV in some automated fashion to identify potentially wrong IDs? I sometimes go through RG and will check CV suggestions if I’m suspicious that there may be a misidentification. Often I find observations ID’d and confirmed based on old CV suggestions that have been superseded by new ones e.g. where more species in that genus were added into the training set. So by now the CV is suggesting a different species than it did back when the observation was first ID’d. It would be great if those were easier to find. So far it’s all manual checking through observations to try to catch these.
Right now the CV doesn’t look at an image unless a human asks it too. What you’re thinking sounds like it would be theoretically possible as long as iNat could pay for that much extra computing power. However when I check RG observations (or captive plants that would be RG if they weren’t captive), if I find wrong ones usually they are wrong because humans have mistakenly agreed to an incorrect CV suggestion.
There’s an old request to have the CV identify unknowns (or if not officially identify then to add some kind of searchable tag, etc) which seems to me like a better use of computing power. iNaturalist staff is usually pretty against “machine generated content” so either way it would probably include the need for a human to review.
Yes, that’s exactly what I mean. Mistakenly agreeing with CV suggestions is one way observations have gotten elevated to research grade. This seems to be especially most likely for older observations that were ID’ed when the CV had fewer species it would suggest. Other research grade observations have been made such by simple human error, for example when an identifier offers a potential taxon, which the observer simply agrees with (perhaps to fulfill the understandable satisfaction to have a research grade observation).
You can use the anomaly detector, which uses the geomodel to find observations that have an incorrect location.
Here are RG observations sorted by lowest-to-highest geomodel scores: https://www.inaturalist.org/observations/identify?quality_grade=research&order_by=geo_score&order=asc&with_private_location=false&expected_nearby=false&acc_below_or_unknown=1000
However, do keep in mind that just because something has a low geomodel score doesn’t necessarily mean the ID is incorrect, but it’s a good way to surface potential errors.
In case CV can’t be used automatically to find potential misidentifications based on outdated CV suggestions, is there a way to look up species lists included in CV based on location and CV model version? I know there are the blog posts about CV updates with some links but those only show the additions.
although it is technically possible to write a program that will automate the process of getting iNat computer vision suggestions for observations (as is done by some folks who put unknown observations into various taxon projects), this sort of thing strikes me as a misapplicaton of the computer vision API, if not explicitly a violation of the terms of use, and i would not recommend doing something like that.
theoretically, you could train your own computer vision model to assess observations. tracking the results of your own computer vision outside of the system and using that as a basis for human assessment seems fine to me, but if you add ids, comments, or observation fields, or if you associate observations with projects, based solely on the results of your computer vision, then that seems to me like a violation of the machine generated content clause in the terms of service.
…
one thing that you could do that does not even involve computer vision is to just scan look at the set of research grade observations of your desired taxon, looking for cases where none of the identifiers (or reviewers) are well-established identifiers of that taxon, or where well-established identifiers of that taxon have maverick identifications. generally, i think these would be the kinds of observations that are most likely to be misidentified.
There’s a group of projects for observations with low CV scores, maybe that would help to find IDs that are more likely to be mistaken?
https://www.inaturalist.org/projects/low-computer-vision-scores-umbrella-project
There are also projects grouping “Unknown” observations into families, orders, and so on, which seems to do pretty much what you are suggesting:
https://www.inaturalist.org/posts/73398-phylogenetic-projects-for-unknown-observations
Yes, iNaturalist staff never expressed interest in doing it so @jeanphilippeb did it himself. I applaud his work.
Similar questions have been raised before:
Include computer vision model version in identification metadata - Feature Requests - iNaturalist Community Forum
Access to former versions of the Computer Vision model? - General - iNaturalist Community Forum
In that sense, CV should at least be able avoid suggesting wrong ids to enthusiasts or improve it’s guessings by having some kind of “filters”, since a lot of the errors committed in iNat from what I experienced came from the CV itself rather than enthuasiasts tentative ids. For instance, most of the time people cannot get to id something to genus-level by themselves, but CV can. By itself, the CV does a lot of the filtering in the observations by providing a “starting point” id, but in most cases, a more general id would be better and easier to correct by specialists and even enthusiasts that are more familiarized with that taxa than one that got to genus or species-level, because of the priority each status have over the other. Something that the CV doesn’t even take into account, that could help prevent wrong ids being suggested is to import occurence data for each organism group for example. Since CV it’s an algorhythm, it’s learning from the countries that have more users posting observations. However, different localities/regions tends to harbour different taxa, that might look similar but are not related at all. As a researcher from Brazil, what I gather from the errors in the group taxa I’m into, its that most of the time, wrong ids come from extremelly specific and risky id suggestions that CV gave taking into account observations made in U.S. for example, which could be right at least in genus-level sometimes but certanly not in species level. Couldn’t CV at least prioritize observations ids made in South America, that are in the same continent instead of prioritizing id from observations made in North America? Shouldn’t potencial spacial distribution and occurence be a basic concept to be taken into account when the subject is biodiversity and species identification?
It does this already. If the observation has a location entered, the CV will by default start by trying to find good matches that are expected nearby, based on iNat observations. (See: Introducing the iNaturalist Geomodel, though the CV included location-based suggestions even before this.) If there is no location entered or it can’t find anything that is a good fit, it will sometimes search globally for suggestions, or you can turn off the option to only show “expected nearby”. This is likely what is happening if you are seeing observations with out-of-range CV suggestions.
In what I see, the far bigger problem is where two superficially similar species share a range, and people often pick the wrong one. I suspect the CV could flag at least some of such errors, but the geomodel probably won’t be a lot of help. I’ve gone through RG observations of a few species where I see consistent errors, but there are an awful lot of species (in case you hadn’t ever noticed 
). Having potential mistaken IDs flagged could be helpful in such cases.
The problem is particularly pronounced where CV previously only recognized one of the two species, which resulted in almost all IDs made with CV being for that species. It takes true expert knowledge to say “no, the CV suggestion is wrong, there is another species that it just doesn’t know to suggest yet.” Even after enough observations of the second species have been ID’d and confirmed to make it into the training set, a lot of the wrong ones still persist, potentially continuing to affect CV suggestions if still pulled into the training set for the wrong species.
I fear it does not necessarily get better once both similar species are in the training set – the CV will suggest both and users will select one more-or-less arbitrarily based on some difference that they believe they have recognized in the taxon photos.
(In my review of European Xylocopas, two of which are extremely difficult to distinguish, I’ve sometimes asked users about their choice and about half the time their response indicates that they do not know the standard ID criteria, but they have concluded that this characteristic or that characteristic in the taxon photos looks more like their specimen, even though it is due to the light or the angle or whether a male or female is the first photo.)
Case in point is the “Acorn Moth”. There are dozens of species in the Blastobasid family that the CV currently recognizes as “Acorn Moth”, despite most of them requiring dissection to separate. But we’ve gotten to a point where there are thousands of “Acorn Moths” on iNat, and many of them are Research Grade due to users clicking “Agree” without knowing about the ID challenges. Many of the species involved have no “internet presence” at all, meaning they were described based on pinned specimens, defined based on genitalia differences, and haven’t been re-discovered by the modern moth photography community, so they fly under the radar and aren’t even considered as options. They’re probably being seen and photographed all the time, but they’re all just sitting under the name “Acorn Moth” online.
Coleophora is another excellent example. There are hundreds of undescribed species in the USA and Canada alone, and those that have been described have mainly been named based on larval characters. Identifying the adults from photos is a fool’s errand based on our current knowledge of the genus- even with dissection, most of them just turn out to be “yet another undescribed entity”. Forbes famously stated in Lepidoptera of New York that naming Coleophora based on adult wing patterns is “nothing short of criminal”. But you wouldn’t know that by looking at the iNat records, where all the vaguely streaky-looking ones get called “cratipennella”, the streaky-but-pale-thorax ones get called “tiliaefoliella”, etc. (and not just on iNat- some of the BAMONA coordinators put names on them haphazardly as well) A few of us try to stay on top of kicking them back to genus, so the CV hasn’t had a chance to “learn” any of the names yet. But once the CV gets one of these species into its training model, that’s going to be the one and only species name that gets applied to everything in the genus from then on.
I think part of the solution is to encourage users to be okay with genus-level IDs. Suggesting an alternate species ID for something that’s RG is usually met with thanks, but going through 100 of someone’s species-level RG observations and disagreeing back to genus for all of them with the comment “this cannot possibly be ID’d to species from this photo” often rubs people the wrong way. It’s what ought to be done in some cases… but I don’t have the emotional bandwidth to deal with the fallout from doing it.
I’ve noticed this too- the USA has a lot of iNat users, so a lot more US species are in the CV model. So any time I’m curating a North American endemic species, I’ll see a bunch of seemingly random dots all over the rest of the world where the CV suggested the Nearctic option because it’s the one that gets uploaded the most. The reverse doesn’t seem to be true as often- i.e. the South American species doesn’t get mistakenly suggested very often for a North American observation. These are situations where the sort: geo_score Ascending option really helps to pick out the outliers.
I guess in answer to the original question, my method for picking out errors is usually:
-use sort: geo_score Ascending for a large taxonomic group I’m familiar with
-find species being frequently incorrectly suggested far out-of-range
-if it’s being incorrectly suggested out-of-range it’s probably being incorrectly suggested in-range too
-go to all RG observations for the species and start picking out the errors
Given the rate at which computer vision capabilities are improving, computer vision generated research data from existing observations will get cheaper. I hope that at some point people using computer vision can look at the ecosystem photographed in observations, and notice things and draw conclusions that we aren’t picking up manually scrolling through observations…