I like the idea over-all, but it would have to be implemented with extreme care.
It should not exclude species based on location, as there is an ever increasing amount of invasive and non-native species spreading all through the world.
The ID suggestion should include non-local species, but perhaps with a geographical note along with it.
Something like: “This looks like XXYZ, which is generally found in eastern North America”
Has anything been done regarding this issue yet? Since the “seen nearby” feature already exists, wouldn’t it be an easy fix to restrict auto-suggestions to these ones only? If the users can tell that it’s something not on the list, they can still search it normally. The problem is that an overwhelming amount of users just pick the first thing on the list, which is often something from the wrong continent. As it stands, the system prioritizes species from high population centers (California, etc.) over anything else, regardless of the “seen nearby” function. The algorithm is clearly not good enough to detect an invasive species unless it is an extremely common species somewhere else anyway, so I wouldn’t worry about that being an issue. That is more of a rare occurrence and best left up to individual users.
Hi @saucegandhi, welcome to the iNat Forum. The staff recently responded here in this topic with this update - see above:
Thanks, I missed that one.
The ‘Seen Nearby’ is also skewed bycultivated plants not being marked as casual. Then a domino effect from that first wrong one …
This suggestion was made 18 months ago but hasn’t been implemented, and is surely one of the biggest weaknesses of the system. While suggestions for some taxonomic groups are good, some are impossible. For example, many, or even most, British Hoverflies (Syrphidae), even with very clear images, get suggestions of species or genus that are only American.
I’m going to share my 2 cents here.
As far as I see it, the main priority of CV is for helping and providing appropriate identifications. It has a second role of speeding up IDs, as well (often times waiting for CV to load is far quicker than typing a name and selecting it from the list).
There have been many times personally where CV suggested a species that was not “seen nearby” and it has been correct. So it’s actually been a big help especially when I was in Australia; a lot of the species were Asian and weedy there, but not otherwise reported on iNat yet.
My ideal vision for CV would incorporate all of these outcomes. The difficult question is “how can you control what is an appropriate suggestion that isn’t seen nearby?”, while retaining the remaining functionality. And I think it could work by using a priority list for how it displays results. Not just cutting “not seen nearby” taxa entirely, because that would be foolish and it removes the ability for CV to provide very helpful input on what something looks like without location bias. One of the main problems I see is the whole “it hasn’t been reported here, so therefore it isn’t”, and people have lower expectations about new count record or “rare” options than they should. But at the same time this still needs to meet the most important focus of CV, which is not a replacement for ID skills, but accurate suggestions.
Here’s what I feel could work. CV runs the images, and develops a number of suggestions, that would be displayed in the following order of priority.
1st priority. Genus name for the top suggestion if that species is “seen nearby”, and if that genus is not monotypic (if it is monotypic, move to 2nd priority). I say genus and not species, because often times the genus ID is vastly more accurate (by proxy of it being less specific). Species IDs tend to incur the most incorrect suggestions in my experience, especially in genera with a lot of local options.
2nd priority. Species name for the top suggestions, if that species is “seen nearby”. This provides a support to the 1st entry, but without dismissing species suggestion entirely. Very important in my eyes.
3rd priority. This is what I’ll call the “general” suggestion. This would be the family ID of the most similar taxa. By default this would incorporate “seen nearby” taxa, but if not, then it’ll just choose the next most likely decision without location bias. For instance, it looks like Ericameria and Isocoma, which are both Asteraceae, so Asteraceae will be chosen as on option. This is also helpful because for someone like me who might know the plants, and can see that the top option is wrong but is at least mostly right, it is useful having the family ID quickly available to click on.
4th priority. Other suggestions that are similar, but not “seen nearby”. For instance, I post a mint and it closely resembles a species from Europe, that isn’t known from California.
The 2nd and 4th priority lists are not restricted to one entry, though 1st and 3rd would be. At least in my head this makes the most sense in terms of how CV works best.
In general I find one of the biggest cons of CV is that it tends to fixate on species and genus IDs, which I understand happens most often when only one species is in the “memory”, and not others. For instance, groundselbush beetle is the only CV-approved Trirhabda species, so for all Trirhabda it chooses that as the top option, when it should be choosing genus. This is leading to a very large amount of incorrect IDed obs in places where that one species is not found.
I think some measure so that the CV can be more conservative in these cases, by at least jumping back to genus or family if the nearest option is not “seen nearby”, could help mitigate some of the taxon selection mistakes. IDing a beetle as a species from Africa is one mistake, but to genus is less of a mistake, and family is usually globally safe.
Suggestions for an observation of a Senna in Hong Kong:
Analysis:
Senna atomaria and Senna multiglandulosa are suggested, despite not a single one has been seen in Asia on iNaturalist.
If the observation looks like Senna pendula, already seen nearby, there is no reason to suggest in the top-8 that this observation could be the 1st occurence of Senna atomaria in Asia.
Suggestion of rules:
If a “species” visually similar to the observation has been seen nearby and is suggested, then do not suggest other “species” in the same “genus” that have not been seen nearby.
If a “genus” visually similar to the observation has been seen nearby and is suggested, then do not suggest other “genus or species” in the same “tribe” that have not been seen nearby.
(Such rules need not impact the Computer Vision itself. They may be implemented as a post-treatment, by selecting a top-8 out of, say, a top-20 resulting from the Computer Vision).
Possible beneficial effects:
Excluding a few unlikely suggestions from the top-8 might enable some better suggestions instead.
In the example above, Senna bicapsularis and Senna pendula are very similar species, but Senna bicapsularis is not suggested (despite the visual similarity with the observation and the favorable location). Senna bicapsularis might be in the top-8, if Senna atomaria and Senna multiglandulosa were not anymore in the top-8.
I’m pretty sure the reason Senna bicapsularis wasn’t suggested is because this species wasn’t available to Computer Vision. AFAIK, for a taxon to be included in the current Computer Vision model it had to have 100 verifiable observations (of which at least 50 had a community ID0 as of 29 September 2019. From what I can see, there were about 61 such verifiable observations by that date.
Fortunately, there are now 176 such observations. It looks like the 100-observation threshold was passed in May 2020, so any model using data after that point should be able to suggest Senna bicapsularis as an option.
I do agree about the broader point that suggesting taxa not recorded anywhere nearby is generally unhelpful. However, when CV fails to suggest the right species it’s probably not that the correct suggestion was crowded out by implausible cousins, but that CV knows nothing about the correct organism because it didn’t make the cut for inclusion in CV training.
Specific to the scenario you raise, the CV suggestion rules already adjust the “raw” list of CV matches to “insert” other sister species seen nearby. From this post by @kueda, it seems that the suggestion algorithm currently:
My guess is that this “insertion” process may be failing for Trirhabda observations because the raw CV results do not contain 3 closely related species. There are currently 2,905 putative Trihabda observations. Of these, 1,195 are identified just as being genus Trihabda. iNat recognizes 26 total species in the genus. Of these, there are 7 species that have no observations at present.
There are 2-3 Trirhabda species I would expect to be covered by the CV model. The first is Trirhabda bacharidis (currently with 665 observations) which had about 335 verifiable observations when the most recent training dataset was collected on 29 September 2019. CV also should be aware of Trirhabda flaviolimbata which had about 410 verifiable observations by the cut-off date. The third possible species is Trirhabda canadensis, which had about 120 verifiable observations by 29 September 2019. However, it’s possible that fewer than 50 of these had a community ID, which would have excluded the species.
So, when someone uploads an observation, there’s a maximum of 2 or 3 Trirhabda species that could be returned in the result set. For the insertion process to search for other species under Trirhabda, the raw result set would need to ID all those Trirhabda species as the top 3 results. Failing that, the insertion process could kick in at the Family level, if the top 3 results are all in Chrysomelidae, but that spans a huge number of genera and species, so I doubt this would result in additional Trirhabda species being inserted.
So in summary it could be that suggestions for Trirhabda will improve quite a bit once there are 4 or 5 species covered by CV.
But your scenario does suggest that it’s worth looking for any a logic tweaks that would better handle Trirhabda observations without degrading suggestions for other scenarios.
Back on your broader proposal, I see benefits for the prioritization you suggest, but this order does cause me concern:
For a lot of taxa I work with, the species-level suggestions are comprehensive and accurate, even within genera of 5 - 20 plant species. I’m concerned that making the genus-level suggestion more prominent than a high-confidence species ID will result in lots of observations with genus-level initial IDs where in fact CV did a fine job of finding the right species. That creates a lot more work for identifiers.
I would support prioritizing the genus just for those observations where the algorithm can identify factors that call into question the reliability of a good visual match. These might be:
The figure you linked to is just a simplified cartoon to provide a general example of what’s happening. It’s actually the top 10 results. If you want to dig into the code, here’s the “common ancestor” calculation. That said, I suspect your explanation for what went wrong with that obs is probably correct.
Here in Ontario, T. bacharidis does not occur but T. canadensis does. On a couple observations of the genus I just tested, genus-level Trirhabda is the top suggested option, which is great. The first species-level suggestion below that is T. bacharidis (even though it doesn’t look like any of our species), and T. canadensis is below that. Because T. bacharidis is the top species option and people like going to species, there are always a couple observation here ID’d as it. There are a lot of T. canadensis observations, I think again because of the CV because there are 3 other species here that are very similar. I would like to be able to go through Ontario’s Trirhabda observations and sort them out, but the 4 species are all similar and variable enough that I haven’t figured out to what extent it’s possible. I doubt most of the species will be able to meet the CV criteria. So there probably shouldn’t be enough observations identified as T. canadensis to put it in the CV options, but it’s better than it not being there because it means that the genus is the top suggestion instead of T. bacharidis. I’m guessing all this is the case for the rest of the range of T. canadensis as well.
The link above opens a page with 176 results instead of the 61 results expected. The date filter is well defined but disabled by default when the page is loaded. Bug report.
About Senna bicapsularis, there is one specimen (or possibly a few specimens) with 100 observations (still increasing, 115 observations 4 days later) that will likely contribute much to the next Computer Vision learning session. It should not be an issue, provided the CV is not biased by recurring artifacts such as the braided flowerpot or the red fence.
I would believe the contrary, that identifiers are given more work because the species-level IDs are far too common or incorrect. I think this depends a lot on both the taxon in question, and on the location. But for instance here, things like Deinandra and Phacelia I’ve seen to be incorrect with a fair frequency.
I agree with you on both these points.
Let’s think about this balance of work between identifiers having to correct computer vision IDs that are over-specific vs computer vision IDs that are too timid. I think this balance is different dependent on how amenable the taxon is to image-based identification and how well iNat’s computer vision is trained on the taxon.
Let’s start with taxa that are amenable to image-based ID and well represented in iNat. For a plant genus such as Dichelostemma or Triteleia, I would guess that maybe 5-10% of times the top species-level CV ID is wrong. I would also say that almost all these observations (with correct or incorrect CV suggestions) are in principle identifiable to species from typical images.
Very roughly, prioritizing the species ID suggestion might have resulted in 4,000 species level Dichelostemma IDs during 2020 and 3,000 Triteleia IDs, of which 400 and 300 respectively might have required correction from a human identifier. In contrast, if the genus had been prioritized, we could expect most or all of those 7,000 observations to require a refining ID from a human identifier. So, the work required to get an accurate species-level ID would be about 10 times greater.
As a counter-example, we can use Trirhabda. Roughly 1,000 observations in 2020. Using the aggressive CV approach, we could expect most of these to receive species-level IDs, and with current data many of those IDs would be wrong. A less aggressive CV approach would result mostly in genus-level IDs, which might be mostly correct. Given the level of detail needed for a species-level ID, it’s possible that the right level of precision would actually be a genus-level ID. Clearly, the aggressive CV approach is creating a lot more work for Trirhabda identifiers, at least in the short term.
So the aggressive CV suggestion approach reduces identifier workload in the Dichelostemma case and increases it in the Trirhabda case. How do we square that?
To improve this I think we need to look at two separate factors. Trirhabda apparently is hard for CV to correctly suggest a species-level ID both because few species in the genus are covered by the CV model and because the genus has several lookalike species that can often not be correctly ID’ed from typical images.
The low-coverage issue should be addressed as iNat accumulates more observations and knowledgable IDs. There’s some scope for tweaking the algorithm to favor genus-level suggestions when many of the subordinate taxa are outside the CV training.
The issue of taxa that can only be reliably IDed to genus or family level from typical images is less tractable. There is a strong argument that iNat should default to different ID levels for different types of organisms. For many arthropods, suggesting a species-level ID is just creating work for human identifiers.
Thanks for that extra insight. That really shows how these two issues interact. Trirhabda observations in Canada are getting incorrect T. bacharidis IDs partly because T. canadensis isn’t something CV knows about. But the incorrect IDs are also partly due to the built-in assumption that it’s reasonable to suggest species-level IDs.
If it were just the first issue, this should be solved as more observations and correct IDs are added. The latter issue doesn’t appear to be something CV is going to handle well. I guess if there were lots of genus-level Trirhabda observations marked “No, it’s as good as it can be” then the CV training and suggestion algorithm could pay attention to that and prioritize genus-level IDs. Does anyone know if that actually happens?
The CV does know T. canadensis, so I’m really not sure why it suggests bacharidis here. But otherwise yeah.
I don’t believe so… Here’s a related staff response: https://forum.inaturalist.org/t/helping-the-computer-vision-is-this-wrong/13825/26
I’ve brought this up several times on the forums, and I’ll do so again here… in its current implementation, the Computer Vision IDs greatly diminish the accuracy of the data on iNaturalist, such that experts are dissuaded from contributing. Here’s a twitter thread from a contributor, Dr. Derek Hennen, expressing this same sentiment and pointing out how he’s using this site less because of the problems caused by inaccurate IDs.
I know this is an issue that’s been discussed, but is there any progress on improving the CV IDs, particularly with respect to biogeography?
[edit: this was merged into this thread, from a separate thread originally titled "Computer Vision IDs are hurting the reputation of iNaturalist.]
I just wanted to give a quick update on functionality changes to better use location in CV suggestions.
On iNaturalist, we currently use location data to boost visually similar species that are also seen nearby, but we don’t do anything to demote visually similar species that aren’t seen nearby
Ken-ichi gives a good overview of this in this talk he recently gave to TDWG:
https://www.youtube.com/watch?v=xfbabznYFV0
We’ve learned from model evaluation experiments that demoting visually similar species provides better predictions on average than our current approach of not doing that. But we’ve held off because we want iNaturalist to also work well in situations where location data might not help, such as a garden filled with ornamentals or a remote location without much nearby data to draw from.
We’re currently working on altering the CV suggestions on iNaturalist so that by default it will demote visually similar species that aren’t seen nearby. But there will be a new toggle to have the CV ignore location data to accommodate these situations where location data doesn’t help (e.g. gardens, captivity).
We’re rolling this out in Android first as part of a new more elaborate ‘species chooser’ which we’re currently testing internally and hope to have in beta some time in the next month. Why Android? That’s just where we have the development resources right now. Once we’ve figured out how to make it work there, we’ll move on to changing the default/adding the toggle on the website and getting it on to the iOS app in some form.
On Seek, we currently don’t incorporate location data into the CV suggestions because Seek suggestions work offline and doing so requires getting location data on-device (there are a few exceptions related to the camera roll and older versions of iOS which use ‘online’ CV and thus location from the server). We’ve recently made progress on getting location data incorporated into the offline Seek CV suggestions (we have a working Android version) but we don’t yet have a release date. When this update is released, Seek will work in the same way as our plan for iNaturalist: i.e. demote species not seen nearby by default and have an option to ignore location data.
Thanks for bearing with us and your patience. We hope these features will help towards reducing the number of wrong IDs suggested by the CV and will thus help alleviate identifier burnout.
