I think this is crucial. You will find the symbols in most of my observations at my initial ID, but it is far from meaning that it was one of the first suggestions. I would mostly use it as a quick way to select the taxon if it happens to be among the suggestions.
1 Like
What I asked for, and was turned down.
The date when this taxon was Included - to appear on the taxon page.
Plenty of space on that ‘Included’ box to add the date.
If you are going to evaluate the CV suggestion, then enquiring minds would like to know - if it is Included, and if so, when - before or after this ID.
Our only access to that date is the blog post with a selection of taxa, each time the CV is updated. And even those embedded links do not include everything from this update.
I leave a comment on my own obs - included in the August 2024 CV update.
2 Likes
People don’t seem to realize how badly iNat’s CV works (or really doesn’t work at all) for clavaroid fungi. I made a small study and compiled some hard data.
-
two days ago I exported all identifications of Phaeoclavulina myceliosa made between 2024-08-07 and 2024-09-07 (1 month). This resulted in 137 P. myceliosa identification over 1 month. I went through these and made correct annotations (link to data in Google sheets).
-
Of these 67.4% had the CV badge, another 5.9% both had a CV badge and confirmed by comment that CV was used. Notably 6.7% of observations did not have a CV badge even though the user confirmed that CV was used (!!). In total (see reasoning in bullet point three 3) we can conclude that 80% of all P. myceliosa identifications are made by blindly trusting CV (as most observations identified as P. myceliosa in fact belong to Ramaria subgenus Laeticolora).
-
I have made correct annotations of all identifications of P. myceliosa made with CV during this 30 day time period. 78% are in fact different flavours of Ramaria (!), 5.6% lack enough information to put them at the genus level and 14% actually belong to the genus Phaeoclavulina.
The argument keeps being made that some experts use CV as a kind of auto complete, which is a valid use case. P. myceliosa and Ramaria subgenus Laeticolora, however, share no visual similarity and can’t be mixed up by anybody with a vague familiarity of either and if a textbook was consulted there would be no doubt. Even if it the ID was intended to be an auto complete, I’d argue that the person got an incorrect notion of P. myceliosa originally because of iNaturalist’s faulty CV suggestions. Therefore, the majority observations with a CV badge in this clavaroid dataset can be assumed to be based of CV and not ‘auto complete’. We can thus conclude that 80% of P. myceliosa observations were made with CV. This gives a subset of 105 observations.
- next is a breakdown of correct identifications.This pie chart shows that of the IDs made with CV: 81.5% got the wrong genus (!!), 5.6% got the correct genus but the wrong species, 3.7% of observations did not have enough information to confirm the genus, and only 2.8% of observations could plausibly be P. myceliosa.
-
P. myceliosa observations made with CV: 88.9% are incorrect, 7.4% are probably incorrect and 2.8% are plausibly correct.
-
Conlusions. iNaturalist’s CV has a probable incorrect identificaiton rate of 96.3% (!!!) of P. myceliosa. Thus, on average, iNaturalist’s CV contributes with 100 incorrect identifications of P. myceliosa per month. There are in total 3634 observations with at least one active identification tag of P. myceliosa. This might mean that iNats CV has single-handedly contributed with 3500 (3634*0.963) incorrect P. myceliosa identifications(!). Regardless of exact numbers, this shows that iNat’s CV has significant flaws for certain groups of clavaroid fungi. Publicly releasing the version numbers made with each ID is a small step to maintain any kind of credibility of iNat’s CV model(s). Arguing that experts should plow through literally thousands of completely incorrect identification is, while not intentional, a rather inconsiderate expectation of how my time should be spent on iNaturalist. Suggesting that an expert’s identification should be considered as accurate as the identification of a deeply flawed CV model (with a hidden version number) is wild.
Here are two examples of obviously incorrect identifications that iNat’s CV contributed into making into research grade observations, which in turn will be used to train the next CV model of P. myceliosa. Suddenly the CV regards Ramaria as a synonym to P. myceliosa!
https://www.inaturalist.org/observations/235758997
https://www.inaturalist.org/observations/232783586
3 Likes
This is interesting, and gels with my experience of working with people new to iNaturalist that they don’t really understand how the CV suggestions work, but will click on the first one, or one with a photo that looks somewhat like what they’ve seen. Even if there are various ways of using the CV suggestions, including as an autocomplete shortcut, uncritical use does seem pretty common, as you’ve shown here.
I wonder if the most effective way to turn off the influx of these observations would be a Feature Request for more frequent re-training of the Seek model? The costs and benefits of that could then be discussed. Including the CV model version would be interesting, but ultimately it won’t do much to solve this issue.
1 Like
This isn’t possible as the CV model used by some apps is currently over 450 days out of date. Older mobile app versions that people still use arguable use even older versions. And going back two years, who know how big of a version difference there was between the main website and a specific app. This is not documented anywhere. A couple of your other concerns has been discussed further up in the thread as well.
It’s not true that this information is “mostly meaningless in most situations”. See my previous comment providing hard data that the opposite is very common.
This is not true. Multiple version of CV models are in concurrent use. For example, the model used in the latest Seek app is over 450 days out of date (see discussion earlier in this thread). Mobile app versions that haven’t been updated presumable use even older built in CV model versions(!).
I’ve shown in my previous comment, that for certain diverse clavaroid genera, that close to 100% of identifications with CV badges are incorrect.
This sadly doesn’t document when species are removed from CV. And these blog posts are spread out over many places, not really optimal tbf.
What has still not been explained is how knowing the CV version number will change this. Are you trying to argue that the pie charts would look different if they were generated separately for each version of the CV? Are you thinking in terms of allocating your time as an expert identifier based on which CV version was used?
4 Likes
Seek definitely uses a local (and simplified) version of the computer vision, but as far as i understand it, the standard iNat apps and the website ping the same server-based version of the computer vision to get suggestions, although different versions of the app and website may prepare and send the images to be evaluated slightly differently during initial upload.
i’m not sure whether iNat Next uses a local model for its AR camera.
1 Like
Do you have any evidence for this? I’ve never heard that this is the case.
As I and other users above noted, the Seek model is separate and it is currently possible to determine which observations have been made with Seek, so there’s no issue here.
3 Likes
Exactly. I could list off all the taxa that the CV consistently misidentifies in my “little brown moth” universe too, and some of them have 90%+ error rates for sure. But the CV icon has no impact on how I treat these misidentifications, which is to correct them with a correct ID… and knowing which version of the CV made the mistake doesn’t impact my workflow when doing this. I’m just failing to see how the point “the CV is bad at some taxa” connects to “life as an identifier would benefit from seeing CV versions on the icon”.
Same with some moth genera, but how would that be improved by a CV version icon? If you know they’re incorrect, you know they’re incorrect… If someone calls their Coleophora moth a Plutella, I know they’re wrong and I add an ID of Coleophora. This is true regardless of where their wrong ID came from. I’d be willing to bet it came from the CV, because this is one of its common errors, but the ID I add doesn’t need to take into account where their wrong ID came from.
I’ve read through this thread twice now and I’m really trying to connect “the CV is usually wrong about some taxa” to “adding a CV version icon would help solve this problem”, and I’m just not seeing it. Am I to assume that if this icon change were implemented, identifiers who currently fail to correct CV misidentifications which they know to be wrong would do so if they knew what CV version made the error? Or that observers who are making mistakes by currently relying on the CV to identify fungi would stop doing so if older observations allowed you to mouse over the icon and see the CV version?
As I’ve said on other threads, the way to fix the CV making consistent errors in a taxon is to keep up with ID corrections so the next version of the CV will be trained on a better data set. When I started as an identifier, I did about 20,000 ID corrections on Acrolophus moths, which the CV was really bad at, and now the CV is better trained on the various species and doesn’t make as many mistakes. I’m up to about 30,000 Acrolophus IDs now, and in no cases has knowing the CV version been relevant to how I treat an observation. I don’t think I’d ever muse, “this looks just right for popeanella… but wait, this was ID’d by 2.11 which was bad at recognizing popeanella, I’d better not click Agree!” Or conversely, “well the palps on that one are way too long for arcanella… oh but this was ID’d by 2.12, which usually nails arcanella, I’d better hold off on my correction!”
I guess I’m just waiting to see this thread come full-circle and explain how the proposed feature change would solve the problem of the CV being bad at certain things, because I’m not seeing the connection.
11 Likes
I haven’t heard a strong argument for why this would be beneficial enough to be worked on so I’m going to close the request.
1 Like