Computer Vision mistaking plants for arthropods

As more people upload and identify leaf-mining insects, plant gallers, etc., more of these species are getting incorporated into the Computer Vision models—and these are all great things of course, IMO.

However, as that happens, Computer Vision begins making only arthropod suggestions in some cases for pictures of plants that have often-reported arthropod associates.

For example, here are screenshots I took this month of the CV suggestions for perfectly fine and ordinary pictures of Trifolium repens, Arctium sp., and Asclepias syriaca (not showing any leaf damage, visible insects, or the like). In all three cases, the CV is suggesting only arthropods, no plant species.

Many, but not all, of these CV arthropod suggestions are photographed in association with the plants in question (so one can see where the CV suggestions are coming from), but none of them were present in the example observations, and I find it especially odd that the CV isn’t suggesting plants at all in the mix of options.

On one hand, there’s an obvious temporary solution for us identifiers, which is to periodically go through arthropod taxa of interest and knock out all the plant observations with another ID (which I do). But that doesn’t seem to be improving the underlying issue with the CV suggestions.

So the reason I’m posting is to draw attention to the issue for the developers and researchers working on the CV models and training. Are these errors always going to be with us? Will things get better over the years if we simply keep making correct identifications on top of all the bad choices people make when misled by CV?

Or are there ways CV can be improved in the short term to reduce these kinds of errors in the model results?

6 Likes

After an ID is added, the CV filters its suggestions to whatever iconic taxa the ID is under. If the ID is of a plant, the CV will suggest plants. If the ID is an insect, the CV will suggest insects. This means that after you correct an incorrect arthropod ID, the CV will stop suggesting arthropods. This also means that if an observer chooses a plant ID, it shouldn’t suggest insects

4 Likes

Can you please provide links to specific observations?

No leaf mines here
https://www.inaturalist.org/observations/213470234
An obs that @djringer retrieved, now trapped at Life

Yes, thanks for that example, Diana, and I think these were the burdock (https://www.inaturalist.org/observations/219191685) and milkweed (https://www.inaturalist.org/observations/176335494) examples. (Sorry, I thought I wasn’t supposed to link to specific observations.) However, as lj_lamera said, these are no longer returning arthropod-only results after my plant IDs, and in fact, the correct plant genera are now the “we’re pretty sure” results. So I guess what I’m seeing is due to a user picking an arthropod example as the initial ID, and thereby bounding the results subsequently (until another ID is added).

I am curious about what the users who do this see in their apps that lead them to pick options like a leafminer, when there’s no leaf damage, or beyond arthropods, a powdery mildew for a perfectly healthy rose bush (https://www.inaturalist.org/observations/228866540). This weekend, I’ll take some pictures myself of plants where I see this happening a lot and see what kind of results I get in CV for an initial ID.

1 Like

For new iNatters, who are politely accepting the ‘pretty sure’ suggestion. And probably overwhelmed by scientific names - they may be utterly mystified by What can You Mine in A Leaf?!
Trifolium can have white patches on the leaves, and at first glance those leaf mines could, be unusually large patches.

More examples generally can be found at Kingdom Disagreements (I clear the African ones daily, but there are still 599 pages for the Rest of The World)
PS 597 ;~))

2 Likes

As examples of technical issues it’s generally fine, but not to call out someone’s behavior.

4 Likes

FWIW, for https://www.inaturalist.org/observations/219191685 here’s what I get in the web uploader, after entering the exact coordinates of the original observation:

And when I import the photo into the iOS app and make the location as close as possible to that of the original observation:

https://www.inaturalist.org/observations/176335494 was created a year go, when the model would have been pretty different than it was today.

1 Like

When CV offers suggestions from more than one Kingdom, is it possible to add a (human) step?

If only a popup to prompt the identifier (or the observer when uploading) - is this for the insect or the flower ? Those obs can languish in Life for ever (10 years and counting).

I can see the CV logic of - this insect is usually seen on this plant, so suggest the insect for the plant picture etc. But why do all rose obs on iNat suffer from mildew by default - mildew or Rosa ? That adds an extra burden on persistent mildew identifiers. Circling back to the leaf miners we started with.

4 Likes

When observing or IDing from the Android app there is a filter on taxa. The identify page also has a filter. It would be nice if the observation pages also had a filter on suggestions.

Have a similar issue with this user’s observation of a very common butterfly being mistaken for a plant. All the “similar” options are also plants.

The person added the ID manually. The CV suggestions are only plants because the observer ID’ed it as a plant.

Suggestions before I corrected it:
IMG_3532

Suggestions after I identified it as a butterfly, making the community taxon “Life”:
IMG_3534

1 Like

Didn’t know it did that.

1 Like

It’s for when a photo has multiple organisms (let’s say a flower with a beetle on it). If the observer adds a plant identification, CV will identify the flower in the photo. If the observer adds an insect identification, the CV will identify the beetle on the flower.

You will get better CV suggestions if you start from ‘your best broad’ taxon with the location filled in.
Unknown needs some human input to start it in the right direction.

1 Like

I was unaware of this forum topic and just started a new one about the same issue, with a proposed solution or two: https://forum.inaturalist.org/t/leafminers-have-broken-the-cv/54378
My most recent comment gives two specific, very recent examples, one of which involves burdock:

Are there any solutions to this being explored? I’ve been seeing more and more examples of computer vision suggesting random assortments of arthropods for photos that are clearly plants. As the number of gallformer, leafminer, and other insect herbivory observations grows and gets incorporated into the CV model, I can see how this could start to seriously undermine CV for plants.

CV is already (understandably) overwhelmed by - It’s A Plant, duh!
If you can start from a plant family, we are all much happier.

Yes. Last week I was working on the yellow square project Fabales. It was exasperating how many obvious legumes were uploaded, not as “Dicots,” “Flowering Plants,” or even “Plants,” but as Unknown. Legumes as a family have some outstanding characteristics; often, I have no clue what species something is, but am certain that it is a legume.

1 Like

You’ve said that sometimes things that everyone else seems to know are not obvious to you. How about extending some compassion and understanding to others and recognize that your lifelong interest in nature provides you with a different knowledge base than someone who does not have this background?

You might feel that it is blatantly obvious that it is a legume – this does not mean that it is the case for everyone.

People who are new to nature and have never really looked at plants may not have any idea about what plant families exist or how different familiar vegetables are related to one another.

Please also consider that most users who upload observations without an ID do so not because they have no idea what they saw, but because they do not understand how to use iNat – they do not know that it is necessary to enter an ID or they do not know how to do so.

2 Likes