Force computer vision to back off on the specificity of suggested IDs in regions with cryptic or hard-to-identify species

Feature Requests
,
1

Platform(s), such as mobile, website, API, other: Both mobile and website, as well as Seek

URLs (aka web addresses) of any pages, if relevant: N/A

Description of need:
The specific examples I list will come from my wheelhouse (spiders), but it is by no means unique to that taxon. Arthropods in general appear to be a problem point.

As discussed in many forum threads such as this one and this one, once a species has enough observations to make it into the CV training dataset and geomodel, the species will begin to be suggested for observations and locales where the CV expects that species to be found. In taxa with species that are either cryptic or otherwise incredibly morphologically similar (for brevity’s sake, I will use “cryptic” as a catch-all term in this feature request), this causes issues with overly specific identifications in the following possible ways:

  • if the initial batch of species IDs were made via observation of details either not visible on the observation photos used in the CV training set or too small for the CV algorithm to notice (such as small or subtle morphological features, microscopic examination, DNA analysis, etc.), that context is lost as the observation photos are fed to the CV algorithm, and the CV will start suggesting the species for any observation of a morphologically similar organism, despite identifiers knowing that a species ID is not warranted in most situations.
  • If not all of a cryptic set of species have enough identified observations to make it into the CV training set, the CV will only suggest the species it is trained on, regardless of whether or not it is possible to confirm the species ID. Many users, in their search for a species-level ID, will pick these options.
    • One way this can happen is where cryptic species only occur in parts of the genus’ range. Examples include:
      • Agelenopsis potteri (populations in Europe and Canada where there are few to no cryptic species, causing A. potteri to be constantly suggested for grass spiders in the USA, where multiple cryptic species exist across much of its range)
      • the Eratigena atrica species complex (E. atrica and E. duellica have geographically distinct populations in the US that allow for species ID, allowing the CV to suggest the very visually similar E. atrica and E. duellica, but not the equally similar E. saeva in Europe, where all 3 species are found. Ideally, in Europe, the species complex should be the baseline ID)
  • There are also hard-to-identify taxa that do not undergo routine & comprehensive cleanup or policing efforts from identifiers (the array of similar-looking species within genus Tetragnatha comes to mind), where uncorrected overly specific IDs have led to species being added to the CV model, causing a feedback loop as overly specific IDs continue to be suggested, accepted, and then confirmed by users who don’t know any better.

A change to the CV status quo at the software level is required to address these systematic issues.

Putting the burden on volunteer identifiers to manually and continually clean this up in perpetuity is simply not a reasonable request, due to the sheer number of observations increasing as iNaturalist constantly seeks to expand its userbase. In many taxa that are harder to identify or parts of the world where identifiers are lacking, identifier bandwidth is simply not keeping up. The constant janitorial work needed to clean up and constantly combat overly specific CV IDs is also damaging to identifier morale and motivation, further worsening the issue.

Feature request details:
In this comment, I laid out several potential avenues for software-level fixes to temper the CV’s tendency to suggest overly specific IDs. Suggestions 1 and 3 could be programmed in to the CV logic, while 2 would require additional human input to generate the “rules” for when to back off. Those suggestions are reproduced below.

  1. ID feedback - if a suggested ID keeps getting disagreed with back to a higher rank in a given region (genus, species complex, subfamily, etc), then the algorithm takes the hint and stops suggesting that species at the species level in that region.
  2. Manual flags - put in taxon-by-taxon instructions that say things like “hey, don’t suggest to species level in this region.“
  3. If a ton of observations are stuck at a higher taxon in general in a given region (especially if RGed at those higher taxa) while there’s a proportionally smaller amount of species IDs in the species within that taxon, have the CV take that into account and start deprioritizing species suggestions in favor of genus/species complex/subfamily in that region.
2

This is desperately needed.

17 Likes
3

This is such a great idea.

It’s not only the users selecting the (wrong) suggestions but mainly the overconfident and overly specific CV suggestions that seem to have increased lately. Addressing this problem via a (regional) feedback loop could prevent disagreements due to wrong specific IDs.

13 Likes
4

Just for some additional context - in the span of about an hour, over a half-dozen unconfirmable Agelenopsis potteri CV IDs popped up in the US.

5 Likes
6

Perhaps a better idea would be, providing a pop-up guide (or side note) that could significantly improve the accuracy of their submissions. This guide could offer areas of focus and highlight key characteristics for commonly misidentified species in a specific region, such as suggesting users take photos of certain body regions. I know that this is a feature with the online iNaturalist platform but, I have not seen this on mobile.

It’s a common human tendency to want to narrow down an identification with great specificity. Rather than focusing on CV, a more effective approach is to empower them with the knowledge to make a better-informed choice/judgment.

Professional entomologists understand this margin of error with the average persons’ observations and or donated specimens. The initial identification from a user is a good starting point/guideline, but it’s rarely definitive. Entomologists in Museums, Agriculture, and other related fields rely on verified specimens and established literature for definitive identification. I believe that working with the public on how to identify species should be at the forefront of this issue. AI or CV are an after-thought and really should be used as just guidelines not primary methods of ID’ing species.

8 Likes
7

I know the iPhone app shows a percentage of how correct it thinks the suggestions are. If that were available on the website, that might help users know how poor a suggestion might be. Assuming they are accurate enough to use, those percentages could also be used to determine what should be shown. If a species does not reach a certain percent, it would automatically show complex, genus, family, etc.

Disagreements with suggestions the current CV offers could knock down those percentages overall and/or by region. So, each CV model would be learning/adjusting until the next is ready to use.

8 Likes
8

I think one of the issues is, that with these difficult groups, those accuracy numbers are not correct at all–they represent over-specific ID’s that basically can never be made from low magnification images. Original poster mentioned Agelenopsis potteri above. My understanding is that many members of that genus can be identified only by very close examination of the male palps or other structures. Here is an example of the sort of images that are needed for definitive ID of that species.

The computer vision (cv) system is an amazing tool, and works really well in many groups of organisms. However for some groups, especially certain invertebrates, the assumptions underlying the model are incorrect. For these groups, a whole chain of unjustified, yet verified, identifications builds up–what I call the “Sorcerer’s Apprentice” problem. Some more robust guard rails are needed. It seems to be a difficult issue to resolve in the current framework of cv model and its interaction with human observers.

7 Likes
9

Regarding manual flags, each taxon could have a field that would be the rank at which CV suggestions could be accurate with a check box to apply to all taxa within that taxon. That could force CV to only ID certain groups to certain levels. The downside of this is that people would likely inappropriately call some groups cryptic that aren’t really. There are lots of plant taxa that you need very specific parts (often microscopic) to key, but the gestalt of the taxa may still be different enough for an expert or CV to pick up on.

As mentioned in some other posts, right now a bad geomodel is causing a lot of wildly inaccurate “expected nearby” suggestions as well as a lack of suggestions of taxa that are actually well documented nearby. As staff have said they are working on fixing the recently introduced geomodel problems, the next geomodel could reduce a lot of the current regional issues.

5 Likes
10

I just do not believe every organism on earth can be identified from imagery. Even if you include microscope imagery, convert audio to a spectrogram image. You can’t accurately train an image matcher for everything. Images leave out smell, chemical tests, DNA, weight, movement, firefly flash patterns, and so many other things that could factor into identifying a species.

Not having the ability to make exceptions even for extreme cases and remove certain taxa from being learned does not make sense.

14 Likes
11

If technically feasible, I think this would be the most painless thing to integrate. It could be a taxon-level flag available only to curators. I am sure the Diptera curators would love to be able to file genus-level flags telling the CV never to suggest anything more specific than genus. Same goes for many, many groups of (near)cryptic arthropods and I’m sure fungi and other things too. There are lots of great ideas/improvements that could be made in this area but I think this one would have the most impact with the least amount of development.

9 Likes
12

As a Diptera person myself, I’d much prefer an automated system - even just a traffic light system - based on the proportion of suggestions of that ID that have been disagreed with. No more extra curatorial work please!

6 Likes
13

The problem with an automated system is that in many cases it would require more work. If, say, it were set up so that if 30% of CV suggested ID’s in a region were needed to force it to stop suggesting that species in a region, you would have to go through every region and add IDs to ensure that the threshold is met.

Whereas what I propose is that flags would be set at the taxon level and apply universally throughout iNaturalist where that taxon (or all daughter taxa) are explicitly blocked from CV output. I’m not sure that having a geographic component is necessary, if a taxon can’t be identified reliably from photos in one part of it’s range, why would we want it to still be suggested by CV in others parts?

With flags, all it takes is one curator clicking one button and boom, no more CV suggestions of that species. I’d say that is much less work than an automatic system and way easier to implement for the iNat staff since they would just block species when they make an export from model training rather than having a whole complicated system which constantly scans observations and recalculates the percentage of incorrect CV ids.

5 Likes
14

@alex Can you please chime in?

There is a desperate need for the community to have more input on what taxa are included/excluded in the CV as it is causing so much identifier burn out and polluting the iNat dataset with bad identifications. We are constantly fighting a firehose of poor CV suggestions on taxa which cannot be reliably identified in 99% of the photos available on iNaturalist. Furthermore, the number is exponentially expanding with more observations and more taxa included in the CV. I am personally very frustrated as threads like this have been coming up on the forum for years now, with no significant changes implemented by the staff. Can you please be involved with this discussion and begin taking serious steps to address the issues which the CV is causing.

8 Likes
15

Because inferred iNat ranges are not always correct. For many fungi the names have been applied incorrectly outside their true range. The obs reach RG and are suggested by the CV in a self-re-enforcing loop. In this case the flag is to convey “this species does not occur here - so please stop suggesting it”.

6 Likes
16

The benefit of one of the two automated options is that whenever the model is updated it will automatically update whether or not it suggests IDs for relevant taxa without extra curatorial work. The negative is that it will take a lot of work experimenting with ratios and algorithms and testing them out on different taxa to figure out how to do it in a way that isn’t counterproductive. If we can get any hint from the hybrid reintroduction thread, trying out complex changes to the CV is a slow process.

The benefit of a manual process is that theoretically it should involve less up-front programming work and can be implemented sooner. The negative is that it will require constant upkeep by curators, and there are so many potential problem species that there’s still a good chance of species being missed and accumulating faulty IDs. We don’t have specialist IDers for all taxa, and even where we do not all of them will know about or optimally utilize this flag option. There will also certainly be cases where a problem species is resolved and having it flagged will be creating extra unnecessary ID work. We know that as IDs are fixed and the CV model is tinkered with, many problem species have been resolved.

Given the above, if my understanding is accurate, I would propose starting with a flagging system if that can be implemented faster, while waiting for an automated system to be developed. But there are a lot of things I don’t know about what happens behind the scenes that could change this. Long-term an automated process seems more sustainable and higher priority.

5 Likes
17

I also think it is very valuable for staff involvement because, none of us truly know how the CV is coded besides those that work with the CV. While there are many ideas here, some likely have much more costs than others, or possibly even are impossible without a complete rebuild of the CV codebase.

What solutions do we have available? It would be lovely to hear from the engineers as there are serious issues that continue to pop up in forum threads.

6 Likes
18

I believe that the geomodel has partially solved this issue no? If all the records of a taxon outside of it’s native range are corrected, the geomodel should update and stop predicting it in that area. This has worked for me for a few plants. It will definitely take work to reidentify everything back to genus, but I think we do have the existing tools to solve this one. If there are thousands of records, it would be a good candidate for a flag to discuss a taxon split to bulk update these.

1 Like
19

Geomodels can be curated, but it can take a lot of effort depending on circumstances. One subfamily of Chironomidae I’ve had to spend a large amount of time validating observations and correcting every single one to get the range map to more accurately reflect its true range.

Once you get a taxon fixed, you still need to put some maintenance in from time to time as misidentifications if allowed to pile up could alter the geomodel you curated. Spread across many taxa, it could get exhausting.

Sometimes the initial struggle is the hardest part. If there are 1000s of observations out of range, that is a lot of work to correct. If the flow of new observations is also high, it can feel like a never ending battle.

3 Likes
20

For fungi this problem lies with many hundreds of species across many different regions. Historical morphology-based species concepts were associated with very broad ranges. Modern phylogenetically-based concepts have shifted the game to much narrower ranges of morphologically cryptic species - thousands of them. When the relatively few experts active on iNat try and point this out and re-identify obs through large-scale taxon swaps the backlash can be significant. In many cases it is difficult to point to a suitable published revision to support shifts but they are true nevertheless and reduced real ranges known to the experts, although proof of absence can never really be definite.

People don’t like their obs being pushed back to genus. And, anyway, as soon as a shift has been carried out the problem starts again with those who didn’t get the message. Far better not to mislead people in the first place with known incorrect suggestions and slowly tackle the backlog through education.

I’ve managed to keep on top of this until recently for my isolated and relativley small patch which is New Zealand by reviewing every fungal observation and telling people why it isn’t species X even though it looks like it. Here the geo-model still suggests incorrect species as ‘expected nearby’ because of incorrect observations in Australia, which not exactly nearby.

A problem I see with the suggestion here is who gets to decide that a particular taxon should be supressed. That requires expertise (not necessarily synonymous with curator status), and iNat has no real mechanism for recognising expertise. I’m not suggesting there should be either.

5 Likes
21

Probably the same way most things are decided. From taxon swaps, community guideline enforcement, name disagreements, DQA gray areas, etc. The community.

One does not need to be an expert to realize a taxon is a major issue in the CV. I don’t consider myself an expert, but I can confidently say the site would be in a better place for Chironomid suggestions if certain Chironomid taxa were not allowed to be learned.

While the concern is very valid, i see it not so different from similar questions like, who gets to decide taxon swaps? Who gets to decide a user is suspended when the guidelines may not be enough to make a decision? Who decides what taxonomy we follow? INaturalist is a community based site.

Ideally this also would be reversible and not strictly permanent. One could make a flag to reactivate a taxon for the CV. Perhaps a taxon deactivated from the CV could even have a public badge just like taxa that are in the CV. Maybe it would say this taxon has been excluded for x reasons.

4 Likes