I’m wondering if the Computer Vision system has an unintentional bias toward animals rather than plants?
I’ve notices that when I upload an image that has both plants and animals in it the CV system tends to offer identification suggestions for the animals, even if the plants are clearly the primary subject.
An example is this observation of small flowers on a tree from a few days back. The flowers were the main subject, but as there are ants gathering the nectar the CV system tried to identify those instead.
I wonder if it might not be useful to have an option to tell the CV that the observation is for a plant, animal, fungus, etc to assist in how it applies its algorithm.
EDIT: For clarity’s sake I’m referring to the initial CV suggestion, not ones the CV system makes once an initial ID has been suggested by a user.
just based on what i’ve seen in the past, if you make an initial ID at a kingdom level, the subsequent computer vision suggestions do seem to be narrowed down to that kingdom.
I suspect in this case there may also be the effect that many observations of ants feature them on plants like this, so the photos that the CV has trained on for ants look like your photo
Yeah, I added an edit, I was referring to the initial CV suggestion, not ones made after a user inputs their own suggestion. The subsequent CV suggestions tend to follow the initial ID made.
That’s exactly what I mean by potential bias in the CV system. People are often more interested in animals than plants, so if there are lots of images of animal on plants that have a user ID as animals rather than plants, then when the CV system is trained it’ll have an ingrained bias to look for the animal in the photo rather than the plant in the photo.
This could probably be countered if every observation like that was duplicated and the plants given an ID (to research grade) as well, but that’s not likely yo happen.
but if this already works this way, why do you need an extra option? the initial ID would be the way you tell the computer vision that you want a suggestion for a plant, an animal, etc.
Yes, I’ve definitely noticed this as well. I recently was surprised when the algorithm identified a photo of common milkweed (the dead stems, photographed in winter, that is) as a Eurasian blue tit! I have also noticed a clear bias toward invasive species (e.g. anything brown and herbaceous looking in winter becomes common mugwort in my area), which I assume is because of the higher ratio of research grade observations of introduced versus native plants?
I think this CV can identify more taxa correctly than any one of us could. And how long would it take any one of us to learn as much as our friend CV knows?
I think an answer to this question will depend on how you define “bias”. You use the phrase “clearly the primary subject” to refer to the flower, but this is a subjective assessment in and of itself, and the kind of determination that can be difficult for a computer model to make.
The response of the CV model that you describe to a picture of this kind might indicate that humans uploading pictures like that (insects on flowers) are more often interested in the insect on the flower as opposed to the flower itself.
I don’t ID a ton of insects, but when I do, I notice that many of those observations are of insects on flowers (as pollinators are wont to be). I would also suspect that many (most?) observations of flowers don’t have pollinators on them. They are likely just the flower itself, either because it didn’t have pollinators at the time or the photographer caused them to leave by disturbing them!
So I would guess (totally a guess) that in many incoming observations, a general rule/pathway that would make good sense to/be favored by a computer model is:
Flower by itself -> Best ID for flower
Flower with something that looks like an insect on it -> Best ID for insect
Whether or not this is a “bias” would depend on how you define bias. If anything, it is likely a bias in the types of images uploaded to iNat and how they are IDed by humans. So I would guess that the CV model is just a reflection of what humans generally choose to care about/focus their IDs on in those types of pictures, and that the CV model is functioning primarily as intended.
The computer vision is fairly smart in that sense. When someone takes a photo of a daisy with a bee on it, so you really think their observation would be of the flower? When they could wait a few seconds for the bee to move away? The photo more likely was capturing the moment in time when the bee was available and still.
I’ve noticed that when people take photos with X having Y on X, then it is usually the Y they are interested in identifying… that could be like the mites on an animal, or the aphids on a plant, or the rust on a leaf. And the computer vision has probably just picked up on that (human) behavioural trend.
As a plant ecologist… my guess is that since the algorithm is trained by iNat users, it just reflects the bias of the majority of naturalists towards animals rather than plants. :)
I had an even more extreme example of this than an insect on a flower - earlier this month I uploaded an observation of a tree that had been heavily chewed by a beaver but not yet taken away. I was interested in IDing the tree itself and had multiple photos; not just of the chewed trunk but the branches, buds, bark, etc. Yet the first suggestion iNat gave me was for “Canadian beaver”, even though there was no animal in sight.
But as @charlie says above, I’m sure that was a reflection of the algorithm being trained rather than an intentional bias - most people who take photos of beaver-chewed trees are probably uploading for the beaver, not the tree.
Since computer vision looks at the lead image, and the lead image is quite ant heavy, I am not surprised. Since if you sent that image to me I would feel you are pointing out the ants. If image two or three were the lead image I feel the suggests would have been much more plant based.
I’d imagine this is just a consequence of the data that goes into the model. Many (terrestrial) animal observations will have plants in them, but the majority of plant photos are plant only.
Probably not a big enough issue to be worth developer’s time to fix, as it can be addressed by the user in the various ways suggested in this thread.
Since the CV system is trained on user observations it’s clear than any bias would be a result of how it’s trained and therefore a reflection of the bias in the user base rather than something intentionally programmed into it.
Even though it’s a reflection of how users interact with the site it’s always important to be aware of bias in anything, especially if it’s potentially going to be used for scientific purposes, as you need to understand the bias and how to factor it into any study done relying on the data.
An example of this is where I work. One of our focal species is a primate. The region is a mature karst area with dense vegetation which limits visibility, making it impossible to do extensive land-based surveys, and it’s an island. As a result the vast majority of survey work needs to be done by boat and, due to the steepness of the cliffs that means that it’s impossible to see into the interior in most areas. Therefore observations are biased to the periphery of the island. This means that all studies of behavior, feeding habits, sleeping site preference, etc have a similar inbuilt bias due to the limitations on the data. This in turn has implications for conservation of the species, the local economy as it’s a tourist area and certain areas need to be protected, politics and policy as this is a species of national importance, etc.
Bias is not necessarily a bad thing, it’s just something that we need to be cognizant of.
