I agree picking higher order taxa is fine, but I think OP is criticising the behaviour of picking from the selected species. This is displaying some confidence because if there is none they do not suggest any species (I think? I’m sure seek does). But you are right in that they are not declaring their confidence explicitly, nor warning about it. I have a hunch they are somewhat ordered too, at least I typically see a correct cv identification at the top of the list.
I think if people did only select the recommended genus it would be fine?
Thanks. Though I do still feel the explanation ought to be different than the one that pops up. It does nothing to address the issue which this thread is about…the limitations and inaccuracies of relying on the AI.
I think @anasacuta did a great job laying out thoughts I’ve been having as well that the CV ID problem is a confluence of two main factors:
Near-automatic/uncritical selection of CV suggestions for IDs by some (even experienced) users.
and
Overzealous agree-ers.
It’s the combination of these two things that potentially leads to positive feedback cycles where the AI promotes large groups of erroneously-IDed observations for certain taxa/locations.
So I think the big question is: can we change something with the way the CV offers suggestions or the way CV is trained (or some other aspect of iNat) to break this bad combination of factors/cycle.
I’m personally sceptical of doing much to fix overenthusiastic agree-ers. This has been a problem on iNat for a while, and, while better onboarding may help, I think it’s an issue that is here to stay to some degree no matter what.
I can think of two suggestions, both of which would probably be unpopular to some degree, but which might be helpful. Both suggestions involve how the system treats observations which have an initial ID made by AI.
Make it so that training sets for future CV models do not include observations which have been IDed by AI that don’t also include at least two other identifications (ie, non-AI identifications). Also, do not include these AI-made observations when determining “seen-nearby” for the CV suggestions.
This change should be pretty easy to implement and would function as a bit of a circuit-breaker to prevent positive feedback loops of AI mis-IDs.
Don’t count AI suggestions towards reaching Research Grade. Make it so that an observation requires two non-AI identifications to reach RG. This would allow the benefits of AI with initial IDs to get observations to experts who can ID them, but require two independent humans to verify to get to RG. This would also help break the positive feedback cycle, and at least philosophically aligns with iNat’s identification philosophy to only agree to what you are sure of. For instance, if I enter an AI ID for something I have little to no knowledge of, that ID doesn’t necessarily mean anything as far as my own expertise. That ID may only signify “I trust this computer program”, which is a very different thing.
This would obviously be controversial, so I just throw it out there as an idea to consider, not necessarily strongly backing it. I know a lot of people (myself included) use AI as a shortcut to save time when typing stuff in. That’s definitely convenient, but it’s something we didn’t have before the AI, and I’d personally be willing to give it up if it meant a big reduction in AI mis-IDs.
Actually, that would do just the opposite of what I prefer. When I know the ID, but don’t want to type it on my phone, why should I be penalized for choosing it from the CV suggestions? Autofill is too useful a tool.
There is an easy solution to that, there just needs to be an interface that allows you to say whether or not you actually know what it is, or are relying on the AI. It would just be one extra click of the mouse or tap of your finger, and would solve a lot of problems. I think that it would also help change people’s relationship with the AI to something that is more healthy for iNaturalist. It is very useful, but it is not a person, and it knows nothing about the organism that it is identifying.
The example above with the amber snail Succinea is a good one, and I said it in other threads and will probably repeat it in the future: what I consider the most efficient way is the CV to learn taxa which are difficult to ID by factoring in disagreements.
My wishlist:
change the wording
<<we are PRETTY SURE>>
to something like
<<the COMPUTER ALGORITHM suggests>>
(because who at all is ‘we’ in the first place and how sure is ‘pretty sure’?)
have an onboarding process where at least once there is a pop-up when using a CV suggestion for the first time describing how to use it carefully
and most importantly: a self-reflecting CV learning process
One major point: it will simply not always be possible to provide enough observations of similar species to have the CV learn the differences - because in many Arthropod taxa there is just no way to tell the species apart on photos (at least on lower resolution cellphone photos).
We had the ‘famous’ situation with the flesh fly Sarcophaga carnaria, which only got resolved not because there were more photos of other species so that the CV could learn the difference, but merely because there was a joint effort to push all the observations back on a higher level to have it below the threshold before the next learning round.
But what would happen when some experts now will upload enough correctly IDed S. carnaria (including both in situ photos and microscopy images)? The CV will again start to suggest this one species for all the blurry flesh fly photos posted on the side.
Similar situation with the Amber Snail Succinea: most species cannot be IDed on photos, but with my suggested process, provided enough disagreements, the CV could learn to be more careful with species suggestions for this taxon in North America, and there would be no need to provide the algorithm with alternative suggestions
I could be happy with that, having to choose between “I’m pretty sure” and “I’m relying on suggestions”. That should probably be a choice when identifying other people’s observations on the website, as well.
I agree that this would be a convenience trade-off (which I did note in my proposal). But the “regular entry” of typing a name also does autofill (at least on the web) so that I often need to type only 5-6 characters to autofill to the option I want. In my own experience, for IDs that I know before uploading, it’s usually a second or two faster to type and select than waiting for the CV.
But maybe it’s like Dumbledore said, “We must all face the choice between what is right and what is easy.”
I have seen this as well: users who state the correct name in the caption or as a comment, but then use the (wrong) CV suggestion for the actual ID, as if they don’t know they can type in an ID or they’re not aware an ID is different than a comment or caption.
Figuring out a way to reduce errors initiated by the CV suggestion would be useful, of course. Getting a pop-up that I have to answer for each ID? Let’s see, that’s an additional mouse movement plus click for each ID, hundreds and hundreds of times. NOT appealing. I’ll live with it if I have to, but it would be annoying. Admittedly right now it would a waste of movement, not time, because iNaturalist is being slow again. However.
One small workaround is to change the main photo of a taxon, because it seems to matter in people’s choices. When people are selecting a choice that the CV gives, they have a tiny square icon for each taxon, which may be the only thing they’re working with when they decide which ID to pick. I try to change the “profile picture” for each taxon to show the flower/leaves/most diagnostic character as big and clearly as possible, since it’ll be shrunk down to a thumbnail.
For example: Haworthiopsis attenuata is commonly grown as a succulent. But until the last CV update, Haworthia was usually the top choice, with Haworthiopsis the second or third. When I changed Haworthiopsis’ ID photo to this close-up photo, more people selected Haworthiopsis which was correct. This is only anecdotal - I don’t have hard evidence, but it seems to work.
I wish that were true. As a beginner I try to work on the backlog of easy ids of species I am personally familiar with and have a confirmed id. The number of instances of “chicory” that have nothing but the original observers guess is high. And I know how many of my observations with decent pictures have not been id’d (although I hope it will improve in winter when entry slows down in northern areas). Of course, not every picture is good enough, but even a comment that a seashell needs a picture of the interior to reliably identify is very helpful.
One thing that I notice often is the situation where someone did just that (selected a specific taxon solely on the basis of the computer vision suggestion), and then someone comes along and adds an ID of “Life,” without realizing (or caring, possibly) that that ID is treated on iNaturalist not only as a disagreement with the first ID but with every taxon between the two IDs. These observations then get stuck in the State of Matter Life category, sometimes for years. So I can see an advantage in being more conservative and opting for a less specific ID that won’t invite the “Life” ID in response.
That’s interesting - I am quite the common culprit in selecting optimistic (and wrong) families based on CV, but this has never happened to me. My experience with follow-up IDs on my guesses is nothing but great (this concerns mostly arthropods though, so it may be different in other classes).
I usually see it happen where the first ID is a species ID rather than something like a family ID. For example a plant species gets responded to with “Life” instead of “Asteraceae,” or even “Plantae.”
Interesting, thanks for the comments everyone, and interesting to hear everyone else’s experiences as well. I use the website and have no experience at all with the app where it would seem that it’s more tempting to use the CV suggestion rather than type anything. Personally, I don’t think we will have any luck in changing the behaviour of people, and I don’t really think a weighting system would be a fantastic solution given how much effort it would involve.
For me, I just think that the wording and even the accessibility of the CV suggestions, at least for new users, needs to be drastically changed to cut down on these problems. I still like the idea of a popup window the first few times CV is used, or even something more drastic like removing it completely until a tutorial is completed. Other suggestions I can think of would be something like making people input their own ID suggestion before being able to view the CV suggestions, or enabling the suggestions to be visible but unable to be selected (again only for new users) so that people whose sole reason for coming to iNat is to use the CV to get an ID can still get a possible answer, but can’t directly input it for their sighting without more effort.
I think a lot of the problem just seems to be that people put too much trust in the CV suggestions because they aren’t really aware of the fact that they are almost always very wrong for most groups. A simple change of wording could go a long way to solving this. But I think part of the problem at least is what was suggested by @arboretum_amy:
A lot of the time new users really do seem to not be aware that they can input their own suggestion. It’s the only real possibility that I can think of as to why this sort of thing happens. I think in this case the user interface needs to be modified, at least for new users, to encourage them to use their own IDs as opposed to just the CV suggestions.
I’m trying very hard to think of suggestions that don’t impact experienced users much, because I know for many people CV is very useful for saving time. I don’t think any experienced users would need any of these ‘restrictions’, because we all understand how the AI works and what its limits are. But for new users, these would be very helpful I think and would help with both teaching new users and cutting back on incorrect IDs. So if we could have one system for inexperienced users and one for experienced, then that would be fantastic. But of course how do you separate the two categories? Anything automated is bound to be gamed by some people, and anything based directly on the views of other users could lead to even more problems (e.g. discrimination or people automatically ‘approving’ their friends).
I think we need to remember that a great many new users come to iNaturalist precisely so they can get an identification, so delaying that would push them away.
Also, although we’re very aware of how bad CV can be, we need to remember that it’s very often right! A test on the older version suggested about 85% right, and the new one is probably better. (After all, people mostly post the common species, which the CV has been abundantly trained on.) Making the system even better is good! But we have something fairly good now, too.
