A few weeks ago I started to notice sometimes I get very unusual suggestions for the species ID. For example this pretty clear image of a crane fly on a wall getting both a plant and bird suggestion which looks nothing like it; or a beetle getting the suggestion for a deer
Platform iOS
App version number 3.3.4 build 719, ios 17.4.1
URLs (aka web addresses) of any relevant observations or pages:
In both cases it still suggested a correct broad group (genus/subfamily) but the individual species suggestions didnāt make sense. Iām not sure if this is a ābugā or just a by product of the computer vision having a ton material to work with that it can find pics of mule deer that look like beetles.
Iām not sure how to reliably reproduce this, it just happens randomly. I just tried re-uploading the beetle pic and it made a better suggestion for Ocean Spray, which is pretty close to the plant. Trying the crane fly again and it make no species level suggestions, so yeah, not sure how to reproduce this.
I donāt see how the āseen nearbyā functionality would explain suggestions that have no relation to the photo. A deer suggested for a beetle just seems incorrect when there are plenty of beetles seen nearby which are a closer visual match.
Your links seem related to computer vision not making a good enough suggestion. My post is about it just making completely wrong suggestions. If it was still recommending beetles or various crane flies seen nearby I wouldnāt be bringing it up here.
Turning that setting off it still suggests the same bird and plant for the crane fly, but it also suggests more crane flies. (Iām not sure why it had the same results this time, this the third time I tested the same photo, 1st and 3rd have these incorrect suggestions, 2nd time none at all. All 3 times I tried it with Show nearby suggestions only on so itās not that the settings are changing.)
With or without that setting it no longer suggests the deer for the beetle.
To be clear: no suggestion at all would be fine, I donāt expect it to always find the right species. But it shouldnāt make suggestions that are way off mark just because they are in the expected range.
as noted in some of the other threads i referenced, even if you start with one image, you could take several different paths to get to CV suggestions, and each path will lead to potentially different sets of suggestions. the fact that you get different suggestions when you go down different paths is not evidence of a bug.
specifically in this case, once an observation is already identified, subsequent suggestions for the observation will be limited to the observation taxonās iconic taxon.
but during upload, when no identification has been made, you could get suggestions from any iconic taxon.
the way the images are processed in the upload screen is also different from the way the image will be processed from an existing observation, but thatās probably a lesser source of differences here.
thereās no evidence of a bug in this thread. itās time to close it.
thatās not evidence of a bug either. weird suggestions from a human perspective donāt necessarily mean that the suggestion was not logical for the computer vision.
take a picture of a hummingbird feeder with no hummingbird, and the CV will invariably suggest a hummingbird.
The point for me is not about considering irrelevant a suggestion for some Sauria for a photo showing only plants (BTW, I agree with your explanation about the bird feeder), the point is to know if the suggestions for the same photo have [dramatically] changed over time (from some [hypothetical] Sauria suggestion [that we didnāt see], to what we see today, no suggestion for Sauria). I have no record of the data at the time this observation was pushed to the project āUnknown / Sauriaā, so I have no [direct] evidence of a cv suggestion bug, I just say I have no other logical explanation.
Another explanation could be the observer changing the 1st photo of the obervation, but this was not mentioned by the observer.
From now on, I keep a record of the data (photo IDs and cv suggestions) at the time an observation is pushed to a phylogenetic project (yellow label project).
I donāt think thereās a specific actionable bug here, but I do think this is a behavior that should be on your radar and I havenāt seen any similar posts about it yet.
In my opinion the noteworthy detail is Iāve noticed this just recently, only the last couple months or so- so I suspect itās related to the last couple rounds on the AI vision model and not the Seen Nearby feature Iāve been using longer than Iāve seen this happening. The incorrect suggestions stand out because Iāve posted so frequently for so long that Iām pretty used to the suggestions I will get and this feels like a something has changed (though still rare).
It might just be that the tolerances of how the model makes visual connections was loosened and maybe itās worth some bad suggestions in the long run.
you may not have seen similar posts, but plenty of them exist. no model will ever give what humans perceive as reasonable suggestions every single time, and there are plenty of existing threads detailing all sorts of CV suggestions that folks perceive to be strange. besides some of the threads iāve already noted, hereās another one that includes an explanation that probably applies in your examples: https://forum.inaturalist.org/t/cv-suggestions-are-bizarre-not-visually-similar-at-all/38025/43.
logging another bug thread and keeping it open even though thereās likely nothing that can be done just adds to all the existing clutter at some point.
the staff have already done a series of vision accuracy experiments and will continue to do these, and thatās probably the effort that will actually drive improvement in these kinds of cases in the long run.
this is not new. itās just another version of the photograph-a-hummingbird-feeder-get-a-hummingbird-suggestion situation i talked about.
any time you get suggestions accompanied by the āweāre not confident enough to make a suggestionā disclaimer, you shouldnāt put much faith in those suggestions.
tying this back to the original post, i think itās also worth noting that you often shouldnāt put much faith in the suggestions that fall lower on the CV suggestions list either. for example, hereās a view of the suggestions for @brnhnās crane fly photo, including things not expected nearby and also showing the underlying scores:
notice that the higher-than-species suggestion gets a score of about 84 (out of 100), but the Great Mullein suggestionās score falls below 1 (out of 100). so the quality of the Great Mullein suggestion here is much lower than the quality of the Crane Fly suggestion.
itās currently not obvious there is such a disparity in the quality of the suggestions because the scores are not displayed to the end user, but there are signs that this will be rectified in the future.
ā¦ and i guess this is probably something that @jeanphilippeb should note, if heās not already incorporating it into his Unidentified project effort. if the top CV suggestionās score falls below some threshold, you probably shouldnāt use that suggestion as the basis for including an observation into a project.
Specialized identifiers might not spend much time reviewing these high rank projects, but I need them because the whole set of phylogenetic projects is used to filter out observations already treated by the program (else the program would run into an infinite loop, treating the same observations again and again). Technically, the āUnknown / Lifeā project is the most important one, because it is the default one, where to push an observation with insignificant c.v. suggestions.
