Workings of iNat Artificial Intelligence (AI) aka Computer Vision (CV)

A couple of examples from my experience:

I made an observation of a Rock Wren (bird) on a granite boulder in a talus field. Computer Vision identified it as a Pika (mammal), because it has been trained on hundreds of images that contain granite boulders in talus fields that have been identified as Pika.

Another observation that I encountered that sticks in my mind had a photograph showing a flower as a blurry purplish blob in the foreground superimposed on a different but well-focused plant in the background. Computer Vision “correctly” recognized that the subject of the photograph was intended to be the fuzzy purple blob in the foreground, because iNat users (including me!) have uploaded so many blurry flower photos. Of course there was no detail in the flower that would allow anybody to identify it, but CV made a species-level suggestion for a plant that does not grow on this continent.

9 Likes

I think this video has a decent explanation of how basic image recognition algorithms work. The computer is asked to tell the difference between a rectangle and a circle, and you can see in the video that its method for doing so is nothing like how you would teach a human to tell the difference between a rectangle and a circle.

6 Likes

On iPhone app, you can toggle which photo is the first one in order to see the different CV suggestions. There are tiny buttons below the photos for toggling. The little buttons are very challenging to hit. Takes us multiple tries.

but, how often, people confuse binomials which are the same for a plant and an animal. So many kingdom disagreements because - how did that happen - people confuse for example, Erica who is a spider instead of Erica (heather). If people struggle, no wonder CV does.

1 Like

Yes, but I was pointing out that the android version works similarly to the web version (as Vireya pointed out), where you can compare CV to all pics without having to switch back and forth between screens to toggel which pic is first.

1 Like

PS instead of circle vs rectangle imagine the AI / CV working thru these
Kingdom Disagreements

1 Like

@twr61 explained it really well, but I’ll add one more example: pinned insects. Let’s say we trained the CV model on only pinned and spread photos of a certain species of moth. Those photos might contain all of the required diagnostic details for that species - maybe some special mark on the underwing. But they would be photos of that species that a) all had a white background and b) showed the moth in a position that almost no iNaturalist user could replicate when photographing the moth in situ.

So if I took a photo of that moth species on a leaf or on some bark, with its wings closed, my photo of the moth would look very different than any of the photos the model was trained on for that speices because the background isn’t white and the wings are closed. So the model might really struggle getting a proper ID from my photo.

7 Likes

The most obvious way that this manifests is that the CV has no concept of “subject” vs. “background”. If an insect is against human skin, it’s likely to suggest a mosquito, regardless of whether the insect looks anything like a mosquito, because most photos of insects against skin are of mosquitoes. (A good way to avoid this bias is to crop out as much of the background as is reasonable).

As other examples, if a lot of pictures of one species are blurry, then the CV may assume a blurry picture is likely to be that species. If a lot of pictures of a nocturnal species are very dark, it may be inclined to suggest that species for any dark picture.

These are all characteristics of the photo, rather than the organism. In some cases they may be pretty good clues for what is likely to be in the photo, but in other cases they can mislead the CV.

4 Likes

You didn’t include a scientific name, but I’m assuming you mean this speices. It’s not in the model (go to the About tab on the taxon page, look on the right-hand side). So you would have never gotten the correct species ID from CV.

Yeah, Seek doens’t use the current model that’s available on the website and the iNat mobile apps. It’s older (so fewer taxa), simplified, it has a higher confidence threshold (as it only shows one suggestion at a time), and it doesn’t take location into account when displaying suggestions.

Yeah, always good to keep in mind. It’s a suggestion, and iNat’s power is really in the community.

4 Likes

There are some mind-boggling examples there!

A proportion of them have arisen because someone accepted an obviously incorrect CV suggestion. Then it takes a bunch of intelligent humans to over-rule the CV.

1 Like

Yes, indeed. One almost feels sorry for the poor old AI/CV faced with untangling these. Not surprisingly, seaweeds, fungi, and lichens seem over-represented. As I have experienced, having wrongly identified something as a Red Algae and having it corrected to Brown Algae (or was it vice versa?) resulted in it being ‘elevated’ to ‘Life’.

1 Like

Thanks Jane. It would take a very clever algorithm and a lot more than pixels in a photo to replicate the power of the human brain!
Are there any measures for how the accuracy of the AI/CV ID’ing is improving over time as more images are added to the data base?

It has been documented that the accuracy is high (80-ish %), but there are relatively few taxa that get uploaded a lot (e.g., mallards, honeybees) and thousands of obscure species that will probably never have enough observations to be included in the model.

A personal tale on CV accuracy. Last year, I reviewed Penstemon strictus pictures on iNat. At the time, it was overwhelmingly the first suggestion out of the purple-flowering beardtongues. From the stats in the “Similar Species” tab, 1/3 of observations that used to say P strictus were wrong. Most of those species-level IDs were inspired by CV (users contribute too, but that’s been discussed to death).

also anecdotal evidence. When Cape Town did its first Bioblitz AI / CV couldn’t even recognise our king protea!

https://www.inaturalist.org/taxa/132848-Protea-cynaroides

Year by year I see steady and appreciated improvement. And motivation to get more obs and IDs for the … Pending to be included next time …

And also motivation to - flag for curation - please add this (African) species. The human curators’ work also gratefully appreciated each time.

1 Like

If you are referring to this post, the range mentioned is 60-80% but those numbers are very misleading in my opinion for the reasons detailed here. …so, I would take them with a pinch of salt.

I don’t believe there are any meaningful broader stats about AI/CV accuracy shared on the forum at present. But the number of species included in the model is growing fast and that is a solid metric for improvement! It increased from 25000-38000 in the update last summer - more details are visible on this blog post.

2 Likes

There is a check box that lets you move a picture to the first position in the iPhone, and then that picture is used for suggestions. The check box is only available to the person who made the observation.

Also remember that local species may not be in CV and thus may not appear in the suggestions. Other more broadly distributed species that are visually similar and overlap the range may appear instead.

1 Like

https://forum.inaturalist.org/t/workings-of-inat-artificial-intelligence-ai-aka-computer-vision-cv/30288/15?u=star3

The perceptron is cool for explaining because you can literally work out a small one by hand (I know, because I had to do this on a computational neuroscience exam in college) but rudimentary, which is why CV barely worked for most problems even 10 years ago, even though it was invented over 60 years ago. The CV that makes inat work is much more sophisticated and probably more similar internally to how the human brain works.

Its also debatable if the training is that much different from how you would teach a human in that you cannot see how your brain actually does the training of your neurons.

2 Likes

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.