Yes, some of the IDs with the Computer Vision icon are actual IDs, where the person could actually identify the species without the CV suggestion. The issue is that those are currently not differentiable from suggestions where the user is just agreeing with the Computer Vision suggestion, which is very common.
My suggestion is a way to differentiate between these two things, and it would certainly help solve this problem. It might also bring up issues of its own, which is why I brought it up for feedback. But so far I haven’t seen anyone suggest any issues with it, other than it might slightly slow down IDs for some users. But, as I already explained in some detail, I don’t think that it is a worthwhile tradeoff to remove the value of a very large number of IDs just to gain a few additional IDs on a few observations that must be rather easy to ID in the first place (if you are ID’ing it so fast that an extra second or two makes a difference).
I’m not certain how many more ways I can reword this, so I think that I am going to stop.
Unfortunately, we are stuck with the humans. So, we need to fix the tool.
I’m not sure what to make of this comment. It does not at all. It presents visually similar photos that are in iNaturalist. Only a small fraction of species will have photos taken and posted on iNaturalist. For some taxa, this might include all the visually similar species, but there is no way for the Computer Vision to know this. It is telling you that here are some visually similar species, some of which may be present in the area, many of which are not. And there are also likely a bunch of other species in the area that look similar, or maybe not, which it isn’t presenting.
for that I will go to their profile. If they are a scientist currently working on the taxonomy of …
Or they have many obs of that taxon
Or many species IDs
But if it is from someone new, I will wait for IDs to roll in from people I have learnt to trust. Probably from their helpful comments.
On 2 of the 3 platforms (website and Android) unless the user intentionally turns it off, the suggestions are ones seen in the area. Hopefully the ios app is soon brought to parity in this. If anything the reverse is an issue in that because of the +/- 45 day window used species that are local may be excluded.
Yes it shows the most commonly seen and reported taxa, but those are also the ones most likely being added as records anyways.
The last time the site did something to remove the most effective way to do id’s which was to remove the agree button, the outrage was spectacular.
…or the most common mis-IDs. I’ve come across multiple regional examples, where one confirmed record shows up as “Visually Similar \\ Seen Nearby” and turns into to 100 incorrect observations of an out-of-range, visually similar species.
Among all this back and forth, I think there is general agreement that people need to better understand the advantages and limitations of CV and mostly stick with the first “We’re pretty sure” option or a higher level for the initial ID.
I will sometimes seek out those unlikely ‘Seen Nearby’ or the random way out of range obs on a distribution map - then try to tweak some back into line. Sometimes the volume is too daunting to tackle.
I’d bet these were almost all before the change in the past couple of months to by default only present seen nearby options. When the default was anything visually similar, yes they could proliferate.
Now with the change in default options presented (again with the hopefully soon fixed ios app) to those seen nearby an incorrect record would have to be both within 50km and +/- 45 calendar days to get returned as an option.
Agreed. But I’ve been impressed by the general accuracy of the iNat CV for plants and since I’m a lousy botanist I rely on it quite a bit. But I use other sources of information also. I have the PictureThis app on my phone (I know, some are not fans of this app) and will often take a photo of the plant on my computer screen to see if that app agrees with iNat (at least to genus). If it does, that’s encouraging, if not I need to do more homework. Regardless of what either CV says, I check my local plant field guides, iNat records from the area, and a comprehensive current list of plant taxa verified from my state to make sure the field guide taxa are still in use. Only then do I settle on an ID at species or genus level for my record. And then the local botanists on iNat get to weigh in on my ID. It takes work, using all the tools I have at hand, but the process has increased my ID accuracy. And hopefully reduces my contribution to badly-IDed records.
Some interesting comments from everyone. A few more thoughts:
I fully agree with everyone saying something along those lines - we should not change the CV suggestions in such a way that it disadvantages people who use it as a quick way to input something that they already know. And I don’t see a way to distinguish between people who know the ID but use CV because it’s fast, and people who don’t know it and choose the CV suggestion because the computer suggested it. This is one of the main reasons why I think the only real things we can do here are to change the wording, or possibly to make a pop-up window or something the first few times a new user uses CV (or perhaps disable them from selecting CV until they have completed a little tutorial or something? I’m not sure). But we shouldn’t disadvantage people who are using it to quickly add hundreds of correct identifications.
This to me is the main problem that needs fixing. The interface and the wording suggest to inexperienced users that the CV is far more accurate than it is, and that you can just take the AI’s suggestion at face value without checking it.
I’ve already ‘mined’ through all of the Australian Mantodea sightings and sadly that’s it for now. People keep uploading them though and we will reach the threshold soon though! I agree that the CV is working as well as it can in this situation, but the problem is not that the CV is inaccurate, it’s that the wording suggests it’s more accurate than it is. The vast vast majority of taxa do not meet the inclusion threshold, so the CV will not suggest them, and currently there is no nuance in the wording of the CV suggestions to say that the AI can only identify relatively common species. I hope that one day the AI will be able to identify most, if not all taxa, but it’s nowhere near there yet and the current wording does not reflect this.
I still disagree with you here. As I said, in the taxa I work with it is accurate less than a quarter of the time. I assume though that with the taxa you are familiar with, it gives you a much higher success rate. We are both biased by our own experiences and need hard data to give an accurate answer.
I did a little ‘test’ of the CV to see how good it was. It still obviously has some flaws but I think it is a pretty good general idea of how good the CV is. I have this project that collects all the sightings of all taxa from four of my favourite places. These places are all quite different from each other and the sightings represent the typical spread of taxa that you would expect to see almost anywhere. So I took the first fifty sightings that I was sure the ID was correct for (yes, I know this in itself is a biased approach but see below), and had a look at the CV suggestions. Going by the results, the AI is actually a lot better than I thought it was. But it’s not majority correct. I got 23 correct (comprised of: 14 where the ‘pretty sure’ suggestion is correct and at least one of the ‘top suggestions’ is also correct, 6 where the ‘pretty sure’ suggestion is correct and all of the ‘top suggestions’ are incorrect, and 3 where no ‘pretty sure’ suggestion was given and at least one of the ‘top suggestions’ is correct) and 27 incorrect (comprised of: 14 where the ‘pretty sure’ suggestion was incorrect and all of the ‘top suggestions’ are incorrect, and 13 where no ‘pretty sure’ suggestion was given and all of the ‘top suggestions’ were incorrect) (46% success rate).
Now clearly there are caveats, the obvious one being that this is only for sightings that I personally know the correct ID of, or I was confident that the people who added IDs to the sightings were correct. Overall though I am happy to say that I did know the vast majority of the sightings, and I had to exclude fewer than 10 (mostly blurry photos where nobody had offered any supporting identification etc.). The other caveat though is that this is only what the CV suggested, not which name was chosen. I think for new users mostly the ‘pretty sure’ option is the one they go with, but of the ‘top suggestions’ they are likely to just choose the first option, and in the sightings I looked at, the correct option was not at the top in the majority of cases. The 46% success rate there is only if the person choosing the ID knows exactly which one it is, and this certainly isn’t the case for new users. So the actual proportion of correct IDs will be significantly less than 46%.
This is not my experience - I have never touched the CV options because I don’t use the CV, but it frequently seems to swap between only local suggestions and suggestions from everywhere. Have a look at the examples I put in my original post - they were just a few minutes apart, and I had changed no settings, but one is only of nearby taxa and one is of taxa from everywhere.
I think one of you is considering accuracy across all observations in iNat, and the other is considering accuracy across all taxa in iNat.
Across all taxa, the accuracy is decidedly low (at least at species level), because only a small fraction of species-level taxa are included in the CV model.
Across all observations, the accuracy may well be above 50%, because the observation set is skewed so strongly toward the taxa that are frequently enough observed/identified to be included in the CV model.
Have a look at the bit right after I said that in my comment! I took what I would say is a fairly representative sample of observations and got a 46% accuracy rate for the CV using a very leniant approach to what constitutes the CV being correct. Certainly the accuracy rate across taxa is extremely low, but across observations it is also not above 50%. It’s close, but it’s not a majority and it’s far from “pretty sure”.
I’m certainly talking about observations, not taxa. Last time I checked there were something like 980000 distinct species level taxa on the site. Only what 38000 is it are in the training model. So high 90 percent are not even trained on.
However the species it is trained on are not evenly distributed either. They are skewed towards frequently seen, easy to photograph etc ones. Of the research grade records globally the top 5 most observed taxa alone are more than 1 percent of all records submitted to the site.
If I remember the top 50 or it might be top 100 most observed taxa are roughly a third of all site records.
And on these the cv does really well. The one fault I see is unlike humans i don’t think it can zoom into a photo so pics of distant birds can get incorrect results.
23 records from 4 small places in Australia is not a representative dataset. A representative dataset would reflect the composition of submissions globally on which the cv is being run.
Okay, but our own personal observations of the CV’s accuracy are not representative datasets either. I don’t trust anyone’s anecdotal reports of its accuracy, especially not my own, so actually looking at a few observations to get an idea of accuracy is the best estimate I can get, unless someone has better data. 50 representative observations from four very different places was just a little test; it’s certainly not accurate for all the observations but it’s a standard representation that gives a decent idea of the right ballpark for accuracy. We’re not going to get a super accurate idea of how good it is unless someone collects huge amounts of data. But really the exact accuracy is not relevant, because whether it’s 40% or 60% or even 70%, it’s not as high as inexperienced users expect it to be or would reasonably think it is from the current wording. The point I’m trying to make is that the CV is fantastic, but the current wording greatly overstates its ability and that is causing problems that could be fixed if it was reworded.
User uploads observation as usual. User is promoted to identify organism without aid of CV. At this point user types in their identification be it species, family, genus or has an option to enter “unknown”.
At this point their selection is “locked in”
After their identification the CV automatically identifies the observation
but the CV’s identification does not count towards Research Grade.
So after “saving” observation you would have 2 ids.
For example:
User: Unknown
Computer Vision: Mallard Duck
Or
User: Mallard Duck
Computer Vision: Mallard Duck
Or
User: Duck
Computer Vision: Mallard Duck
Only the Users ID would count towards Research Grade. As a default you could have the Computer Vision’s ID displayed as a placeholder name if the user selects “unknown”
The problem here though is that this still unfairly disadvantages people who use the CV suggestions to speed up their IDs when they already know what they are