Better use of location in Computer Vision suggestions

This suggestion was made 18 months ago but hasn’t been implemented, and is surely one of the biggest weaknesses of the system. While suggestions for some taxonomic groups are good, some are impossible. For example, many, or even most, British Hoverflies (Syrphidae), even with very clear images, get suggestions of species or genus that are only American.

4 Likes

I’m going to share my 2 cents here.

As far as I see it, the main priority of CV is for helping and providing appropriate identifications. It has a second role of speeding up IDs, as well (often times waiting for CV to load is far quicker than typing a name and selecting it from the list).

There have been many times personally where CV suggested a species that was not “seen nearby” and it has been correct. So it’s actually been a big help especially when I was in Australia; a lot of the species were Asian and weedy there, but not otherwise reported on iNat yet.

My ideal vision for CV would incorporate all of these outcomes. The difficult question is “how can you control what is an appropriate suggestion that isn’t seen nearby?”, while retaining the remaining functionality. And I think it could work by using a priority list for how it displays results. Not just cutting “not seen nearby” taxa entirely, because that would be foolish and it removes the ability for CV to provide very helpful input on what something looks like without location bias. One of the main problems I see is the whole “it hasn’t been reported here, so therefore it isn’t”, and people have lower expectations about new count record or “rare” options than they should. But at the same time this still needs to meet the most important focus of CV, which is not a replacement for ID skills, but accurate suggestions.

Here’s what I feel could work. CV runs the images, and develops a number of suggestions, that would be displayed in the following order of priority.

1st priority. Genus name for the top suggestion if that species is “seen nearby”, and if that genus is not monotypic (if it is monotypic, move to 2nd priority). I say genus and not species, because often times the genus ID is vastly more accurate (by proxy of it being less specific). Species IDs tend to incur the most incorrect suggestions in my experience, especially in genera with a lot of local options.

2nd priority. Species name for the top suggestions, if that species is “seen nearby”. This provides a support to the 1st entry, but without dismissing species suggestion entirely. Very important in my eyes.

3rd priority. This is what I’ll call the “general” suggestion. This would be the family ID of the most similar taxa. By default this would incorporate “seen nearby” taxa, but if not, then it’ll just choose the next most likely decision without location bias. For instance, it looks like Ericameria and Isocoma, which are both Asteraceae, so Asteraceae will be chosen as on option. This is also helpful because for someone like me who might know the plants, and can see that the top option is wrong but is at least mostly right, it is useful having the family ID quickly available to click on.

4th priority. Other suggestions that are similar, but not “seen nearby”. For instance, I post a mint and it closely resembles a species from Europe, that isn’t known from California.

The 2nd and 4th priority lists are not restricted to one entry, though 1st and 3rd would be. At least in my head this makes the most sense in terms of how CV works best.

4 Likes

In general I find one of the biggest cons of CV is that it tends to fixate on species and genus IDs, which I understand happens most often when only one species is in the “memory”, and not others. For instance, groundselbush beetle is the only CV-approved Trirhabda species, so for all Trirhabda it chooses that as the top option, when it should be choosing genus. This is leading to a very large amount of incorrect IDed obs in places where that one species is not found.

I think some measure so that the CV can be more conservative in these cases, by at least jumping back to genus or family if the nearest option is not “seen nearby”, could help mitigate some of the taxon selection mistakes. IDing a beetle as a species from Africa is one mistake, but to genus is less of a mistake, and family is usually globally safe.

10 Likes

Suggestions for an observation of a Senna in Hong Kong:

Analysis:

Senna atomaria and Senna multiglandulosa are suggested, despite not a single one has been seen in Asia on iNaturalist.

If the observation looks like Senna pendula, already seen nearby, there is no reason to suggest in the top-8 that this observation could be the 1st occurence of Senna atomaria in Asia.

Suggestion of rules:

If a “species” visually similar to the observation has been seen nearby and is suggested, then do not suggest other “species” in the same “genus” that have not been seen nearby.

If a “genus” visually similar to the observation has been seen nearby and is suggested, then do not suggest other “genus or species” in the same “tribe” that have not been seen nearby.

(Such rules need not impact the Computer Vision itself. They may be implemented as a post-treatment, by selecting a top-8 out of, say, a top-20 resulting from the Computer Vision).

Possible beneficial effects:

Excluding a few unlikely suggestions from the top-8 might enable some better suggestions instead.

In the example above, Senna bicapsularis and Senna pendula are very similar species, but Senna bicapsularis is not suggested (despite the visual similarity with the observation and the favorable location). Senna bicapsularis might be in the top-8, if Senna atomaria and Senna multiglandulosa were not anymore in the top-8.

5 Likes

I’m pretty sure the reason Senna bicapsularis wasn’t suggested is because this species wasn’t available to Computer Vision. AFAIK, for a taxon to be included in the current Computer Vision model it had to have 100 verifiable observations (of which at least 50 had a community ID0 as of 29 September 2019. From what I can see, there were about 61 such verifiable observations by that date.

Fortunately, there are now 176 such observations. It looks like the 100-observation threshold was passed in May 2020, so any model using data after that point should be able to suggest Senna bicapsularis as an option.

I do agree about the broader point that suggesting taxa not recorded anywhere nearby is generally unhelpful. However, when CV fails to suggest the right species it’s probably not that the correct suggestion was crowded out by implausible cousins, but that CV knows nothing about the correct organism because it didn’t make the cut for inclusion in CV training.

8 Likes

Specific to the scenario you raise, the CV suggestion rules already adjust the “raw” list of CV matches to “insert” other sister species seen nearby. From this post by @kueda, it seems that the suggestion algorithm currently:

  1. finds the common ancestor for the top 3 raw results,
  2. searches for additional taxa descending from that ancestor that have been observed within 100 km of the observation’s location, and
  3. inserts those taxa into the list of raw results based on the frequency of nearby observations.

My guess is that this “insertion” process may be failing for Trirhabda observations because the raw CV results do not contain 3 closely related species. There are currently 2,905 putative Trihabda observations. Of these, 1,195 are identified just as being genus Trihabda. iNat recognizes 26 total species in the genus. Of these, there are 7 species that have no observations at present.

There are 2-3 Trirhabda species I would expect to be covered by the CV model. The first is Trirhabda bacharidis (currently with 665 observations) which had about 335 verifiable observations when the most recent training dataset was collected on 29 September 2019. CV also should be aware of Trirhabda flaviolimbata which had about 410 verifiable observations by the cut-off date. The third possible species is Trirhabda canadensis, which had about 120 verifiable observations by 29 September 2019. However, it’s possible that fewer than 50 of these had a community ID, which would have excluded the species.

So, when someone uploads an observation, there’s a maximum of 2 or 3 Trirhabda species that could be returned in the result set. For the insertion process to search for other species under Trirhabda, the raw result set would need to ID all those Trirhabda species as the top 3 results. Failing that, the insertion process could kick in at the Family level, if the top 3 results are all in Chrysomelidae, but that spans a huge number of genera and species, so I doubt this would result in additional Trirhabda species being inserted.

So in summary it could be that suggestions for Trirhabda will improve quite a bit once there are 4 or 5 species covered by CV.

But your scenario does suggest that it’s worth looking for any a logic tweaks that would better handle Trirhabda observations without degrading suggestions for other scenarios.

Back on your broader proposal, I see benefits for the prioritization you suggest, but this order does cause me concern:

For a lot of taxa I work with, the species-level suggestions are comprehensive and accurate, even within genera of 5 - 20 plant species. I’m concerned that making the genus-level suggestion more prominent than a high-confidence species ID will result in lots of observations with genus-level initial IDs where in fact CV did a fine job of finding the right species. That creates a lot more work for identifiers.

I would support prioritizing the genus just for those observations where the algorithm can identify factors that call into question the reliability of a good visual match. These might be:

  1. Many related species that are not in scope for CV.
  2. High rate of previous misidentifications.
  3. Low CV coverage rate for this geography.
  4. Some variable factor that reflects how amenable each iconic taxon is to image-based ID (e.g. it’s realistic to identify many flowering plants to species or even subspecies level based on photographs, but for many arthropods a genus- or family-level ID is the best that is reasonable).
3 Likes

The figure you linked to is just a simplified cartoon to provide a general example of what’s happening. It’s actually the top 10 results. If you want to dig into the code, here’s the “common ancestor” calculation. That said, I suspect your explanation for what went wrong with that obs is probably correct.

2 Likes

Here in Ontario, T. bacharidis does not occur but T. canadensis does. On a couple observations of the genus I just tested, genus-level Trirhabda is the top suggested option, which is great. The first species-level suggestion below that is T. bacharidis (even though it doesn’t look like any of our species), and T. canadensis is below that. Because T. bacharidis is the top species option and people like going to species, there are always a couple observation here ID’d as it. There are a lot of T. canadensis observations, I think again because of the CV because there are 3 other species here that are very similar. I would like to be able to go through Ontario’s Trirhabda observations and sort them out, but the 4 species are all similar and variable enough that I haven’t figured out to what extent it’s possible. I doubt most of the species will be able to meet the CV criteria. So there probably shouldn’t be enough observations identified as T. canadensis to put it in the CV options, but it’s better than it not being there because it means that the genus is the top suggestion instead of T. bacharidis. I’m guessing all this is the case for the rest of the range of T. canadensis as well.

1 Like

The link above opens a page with 176 results instead of the 61 results expected. The date filter is well defined but disabled by default when the page is loaded. Bug report.

About Senna bicapsularis, there is one specimen (or possibly a few specimens) with 100 observations (still increasing, 115 observations 4 days later) that will likely contribute much to the next Computer Vision learning session. It should not be an issue, provided the CV is not biased by recurring artifacts such as the braided flowerpot or the red fence.

1 Like

I would believe the contrary, that identifiers are given more work because the species-level IDs are far too common or incorrect. I think this depends a lot on both the taxon in question, and on the location. But for instance here, things like Deinandra and Phacelia I’ve seen to be incorrect with a fair frequency.

3 Likes

I agree with you on both these points.

Let’s think about this balance of work between identifiers having to correct computer vision IDs that are over-specific vs computer vision IDs that are too timid. I think this balance is different dependent on how amenable the taxon is to image-based identification and how well iNat’s computer vision is trained on the taxon.

Let’s start with taxa that are amenable to image-based ID and well represented in iNat. For a plant genus such as Dichelostemma or Triteleia, I would guess that maybe 5-10% of times the top species-level CV ID is wrong. I would also say that almost all these observations (with correct or incorrect CV suggestions) are in principle identifiable to species from typical images.

Very roughly, prioritizing the species ID suggestion might have resulted in 4,000 species level Dichelostemma IDs during 2020 and 3,000 Triteleia IDs, of which 400 and 300 respectively might have required correction from a human identifier. In contrast, if the genus had been prioritized, we could expect most or all of those 7,000 observations to require a refining ID from a human identifier. So, the work required to get an accurate species-level ID would be about 10 times greater.

As a counter-example, we can use Trirhabda. Roughly 1,000 observations in 2020. Using the aggressive CV approach, we could expect most of these to receive species-level IDs, and with current data many of those IDs would be wrong. A less aggressive CV approach would result mostly in genus-level IDs, which might be mostly correct. Given the level of detail needed for a species-level ID, it’s possible that the right level of precision would actually be a genus-level ID. Clearly, the aggressive CV approach is creating a lot more work for Trirhabda identifiers, at least in the short term.

So the aggressive CV suggestion approach reduces identifier workload in the Dichelostemma case and increases it in the Trirhabda case. How do we square that?

To improve this I think we need to look at two separate factors. Trirhabda apparently is hard for CV to correctly suggest a species-level ID both because few species in the genus are covered by the CV model and because the genus has several lookalike species that can often not be correctly ID’ed from typical images.

The low-coverage issue should be addressed as iNat accumulates more observations and knowledgable IDs. There’s some scope for tweaking the algorithm to favor genus-level suggestions when many of the subordinate taxa are outside the CV training.

The issue of taxa that can only be reliably IDed to genus or family level from typical images is less tractable. There is a strong argument that iNat should default to different ID levels for different types of organisms. For many arthropods, suggesting a species-level ID is just creating work for human identifiers.

3 Likes

Thanks for that extra insight. That really shows how these two issues interact. Trirhabda observations in Canada are getting incorrect T. bacharidis IDs partly because T. canadensis isn’t something CV knows about. But the incorrect IDs are also partly due to the built-in assumption that it’s reasonable to suggest species-level IDs.

If it were just the first issue, this should be solved as more observations and correct IDs are added. The latter issue doesn’t appear to be something CV is going to handle well. I guess if there were lots of genus-level Trirhabda observations marked “No, it’s as good as it can be” then the CV training and suggestion algorithm could pay attention to that and prioritize genus-level IDs. Does anyone know if that actually happens?

1 Like

The CV does know T. canadensis, so I’m really not sure why it suggests bacharidis here. But otherwise yeah.

I don’t believe so… Here’s a related staff response: https://forum.inaturalist.org/t/helping-the-computer-vision-is-this-wrong/13825/26

1 Like

I’ve brought this up several times on the forums, and I’ll do so again here… in its current implementation, the Computer Vision IDs greatly diminish the accuracy of the data on iNaturalist, such that experts are dissuaded from contributing. Here’s a twitter thread from a contributor, Dr. Derek Hennen, expressing this same sentiment and pointing out how he’s using this site less because of the problems caused by inaccurate IDs.

I know this is an issue that’s been discussed, but is there any progress on improving the CV IDs, particularly with respect to biogeography?

[edit: this was merged into this thread, from a separate thread originally titled "Computer Vision IDs are hurting the reputation of iNaturalist.]

2 Likes

I just wanted to give a quick update on functionality changes to better use location in CV suggestions.

On iNaturalist, we currently use location data to boost visually similar species that are also seen nearby, but we don’t do anything to demote visually similar species that aren’t seen nearby
Ken-ichi gives a good overview of this in this talk he recently gave to TDWG:
https://www.youtube.com/watch?v=xfbabznYFV0

We’ve learned from model evaluation experiments that demoting visually similar species provides better predictions on average than our current approach of not doing that. But we’ve held off because we want iNaturalist to also work well in situations where location data might not help, such as a garden filled with ornamentals or a remote location without much nearby data to draw from.

We’re currently working on altering the CV suggestions on iNaturalist so that by default it will demote visually similar species that aren’t seen nearby. But there will be a new toggle to have the CV ignore location data to accommodate these situations where location data doesn’t help (e.g. gardens, captivity).

We’re rolling this out in Android first as part of a new more elaborate ‘species chooser’ which we’re currently testing internally and hope to have in beta some time in the next month. Why Android? That’s just where we have the development resources right now. Once we’ve figured out how to make it work there, we’ll move on to changing the default/adding the toggle on the website and getting it on to the iOS app in some form.

On Seek, we currently don’t incorporate location data into the CV suggestions because Seek suggestions work offline and doing so requires getting location data on-device (there are a few exceptions related to the camera roll and older versions of iOS which use ‘online’ CV and thus location from the server). We’ve recently made progress on getting location data incorporated into the offline Seek CV suggestions (we have a working Android version) but we don’t yet have a release date. When this update is released, Seek will work in the same way as our plan for iNaturalist: i.e. demote species not seen nearby by default and have an option to ignore location data.

Thanks for bearing with us and your patience. We hope these features will help towards reducing the number of wrong IDs suggested by the CV and will thus help alleviate identifier burnout.

25 Likes

There are two competing streams for ID on iNat.

  1. A garden plant, not marked as Casual, so iNat offers visually similar and nearby - which is not necessarily helpful or useful. Especially to someone new to iNat.

  2. A (genuine) wild plant - where nearby is more useful than visually similar.

I see weird things on the distribution maps - which I try to tweak where I can.

2 Likes

One simple thing that may help in the interim is removing the language “We’re pretty sure this is…”

To inexperienced users, seeing that from the AI is difficult to distinguish from “This is…”

Pretty much every time I upload a pile of images I get a wildly incorrect “We’re pretty sure this is…”

Maybe something like, “Our top suggestion” would encourage fewer blind acceptances.

All that said, I greatly appreciate how quickly iNat has improved, and can hardly imagine how impressed we all would have been with the AI not so many years ago.

6 Likes

100% agree with this; I think a lot of people associate any kind of AI or computer software with infallibility, so when given a list of 10 suggestions they assume that one of them has to be correct

6 Likes

@loarie, thanks for sharing about this update. This seems to me like a reasonable solution, and I’m looking forward to seeing the implementation. It will help a lot! I like the idea of having the button to override location restrictions - that may also help for IDs of common (mostly introduced) species in parts of the world that haven’t accumulated many local observations yet. Just hopefully random people won’t be pressing that button too frequently.

1 Like

Thank you. It is funny to me when I see an ID for Sierra Juniper (J. grandis) in Virginia or in Poland as well as a suggested ID of J. chinensis in the middle of Idaho.