Allow users to constrain Computer Vision suggestions to given taxa

Right now, the computer vision suggestions change to reflect location, but there’s no other way to provide it with useful information. I think it would increase the utility of the CV substantially if you could limit its suggestions to a particular taxon. This would solve two unrelated problems.

  1. If I have a picture of two organisms together–a bee on a flower, a predator and prey, disease and host, etc–I often only get suggestions for the organism the CV decides you’re interested in. If I could eg just say “tell me all the plants this could be,” when it’s suggesting insects, the AI could be useful twice for the same picture.

  2. If I think I know what family or order an organism is in, I would like to see what the CV suggests within that particular order. This would make it useful in cases where you can recognize a high-level trait the CV isn’t aware of (yet). For instance, I posted this scat and got 10 mammal species suggested. Someone pointed out that it must be bird scat because of a white uric acid stain. It would be nice if I could then ask the CV to suggest only birds. You could then try a few possible families and see if the CV suggests anything convincing.

The obvious objection is that if the CV knew something well enough to ID it given a family, it would have suggested it in the first place, but algorithms are weird, and that might not be true–I’ve heard of cases where chess AIs have given up on a particular side only to immediately find a winning strategy for it once given a particular human move it hadn’t found. iNaturalist already capitalizes on the fact that AI and human expertise are better together than either alone, and this would let us take that synergy a bit farther.

PS – I’m slightly confused now because my bird poop example is now only suggesting birds, as if it’s taking my suggestion here, but in other cases (eg these ants on a dragonfly) it is still suggesting the things I don’t want. Is this feature already present and I’m just not grasping how it’s meant to be used?

On the observation detail page (individual observation page):

  • Click “Compare” next to any ID (or on the right side of the page in the Community ID section, if there isn’t an ID yet)
  • Change source to “Visually Similar”
  • Select the taxon you want to constrain the results to

On the Identify page:

  • Open the identify modal for an observation (click one of the thumbnail photos)
  • Go to the Suggestions tab
  • Change source to “Visually Similar”
  • Select the taxon you want to constrain the results to

6 Likes

+1 to this idea.

Often I know a plant is in a particular genus, and would like some suggestions from the AI in the field. Being able to constrain suggestions taxonomically would be a welcomed addition to the app.

I didn’t know you could do this in the “Compare” feature. Excited to try it out!

1 Like

Perfect, thanks for explaining how this works! I’d seen all that infrastructure before but never given it much thought :/