Recently I have been getting a lot of hilariously bad suggestions and misidentifications when using the iNat app (iPhone, version 17.4.1). It seems that most pertain to insect photos, but not all. Screenshot examples below from past few weeks:
Does this appear to be a bug with the AI identification or something more localized? Thanks!
Eric Hough (iNat: eric_hough)
Here are URLs for some of the above observations:
https://www.inaturalist.org/observations/214013744
If you start CV from butterfly or beetle or at least insect … your suggestions will be better.
I had a few instances with the app (iOS) where for some reason or another, the GPS info was faulty or not attached to the file (thus no location) when I tried to upload in the app. I might have done a screen capture of something and tried to upload that rather than using the original photo. In such cases, especially with a complex photo, iNat’s CV is thrown into guessing mode because the visual clues were too complex (no focal organism) and there was no location to gage “expected nearby”. It was the digital equivalent of throwing its hands up in the air and saying, “Huh?”
Considering the totally reasonable image quality of these photos, I don’t think I’ve ever gotten CV suggestions that were that bad on Android at least – especially the Trichoton suggested as a Cooper’s Hawk. If you go back in to edit the observations on the app, do they still give those suggestions?
Edit: on Android, I’m actually getting some similarly terrible results. Not sure what’s going on here. On desktop, it correctly suggests Trichoton at genus level.
I can re-create the issue with the Juno Buckmoth too on Android – I get identical results as you. Maybe the mobile version is doing some kind of unexpected cropping before it runs the image through the model?
i suspected this is likely at least partially responsible for the unexpected results. see: https://forum.inaturalist.org/t/seems-like-the-cv-suggestions-in-the-web-upload-screen-might-be-based-on-a-cropped-image/51000
i supect that some of these images are already challenging for the computer vision to analyze in the first place for various reasons (ex. that first caterpillar really does sort of look like a cactus even to human eyes), but then if you add some unexpected cropping, you make things even more challenging for the CV.
for example, compare the results that the CV returns for the two images below. the uncropped image shows the entire outline of the beetle, but the square center crop makes the bottom edge of the beetle fall just outside of the bounds of the image, and look how much that changes the results. in particular, notice how the score of the first suggestion drops dramatically. (i assume this is because losing that bottom edge of the beetle introduces some uncertainty – ex. does it have a tail?)
uncropped:
square center cropped:
For me, it seems CV seems to do pretty well after I add an ID of an iconic taxa. Here’s its suggestions of a Euglesa shell before adding an ID:
Very off, with half of them in the complete wrong kingdom. Here’s the suggestions after adding the ID Euglesa:
Correct subfamily, at least, and has Euglesa is its second suggestion.
CV uses current identifications on an observation to narrow it down to an iconic taxon (mammals, insects, plants, etc.) to give more accurate suggestions. But it looks like without any iconic taxa to base its suggestions off of it can really struggle
this is part of the difference, but it’s deeper than that, i think.
when you ask for a CV suggestion during upload, the upload page on the website or the iNat app actually sends a version of the image toe the CV to be evaluated, and it seems like there’s some unexpected manipulation (cropping?) of the source image before sending it over to be evaluated.
but when you get a CV suggestion from an observation that has already been uploaded, then the suggestions are based on the image stored on the server, and in that situation, the CV seems to be taking the image as is (possibly only with a bit of resizing).
Here’s another one. https://www.inaturalist.org/observations/214645000
With nearby suggestions toggled on:
With nearby suggestions toggled off:
For what it’s worth, I frequently remove and redownload the iNat app due to the doubling of storage space that happens when uploading a bunch of images. So it wouldn’t appear to be an issue of having the app too long with excess storage space and bugs happening for some reason.
I have a background as a biologist and naturalist, so at least I can get most of my observations in the correct Order or Family at least, but I worry if this same issue is happening for novices using the app.
for this beetle, try uploading it again with just the first photo (not both), and see what suggestions you get. you don’t have to actually send it to iNat (or you can delete it afterward). i just would like to see what suggestions it gives you during the upload process.
That is very strange behavior. @pisum, I did a screen capture of the above beetle from the observation page and attempted a fresh test upload with only the photo–no date, no location, no suggested ID, etc. iNat’s CV seemed to recognize it fine! See attached screen capture of the suggested IDs. FWIW, this is on the website using an older version of Chrome (103.0.5060.134 (Official Build) (x86_64)) on a MacBook Air with OS 10.11.6.
yes, that’s why i asked eric_hough to try to repeat it on his app (to make sure it wasn’t something about the workflow during that particular upload that caused the unexpected results).
Next attempts: I tried to get an ID of the beetle from two pathways on my iPhone 14 (iOS 17.4.1).
(1) I took a photo of the beetle image (displayed on my laptop from the original observation) with the Camera app on the iPhone. I then did a test upload accessing the image in the Photo Library from within the iNat app and got bizarre suggestions–a different list that the @eric_hough. See the first screen capture below.
(2) I accessed the iPhone camera from within the iNaturalist app and then did a test upload with that. It comes up with a different erroneous set of suggested IDs. See 2nd screen capture.
So does this have something to do with some aspect of pulling images from the Photos app into iNat (iOS or other)?
Quick update: after some sleuthing by @pleary and @alex they did find a bug in how the iOS app resizes and crops small images before giving CV suggestions which can explain the strange list of species. We’re working on a fix. Thank you all for reporting it!
I agree that it has something to do with cropping (see the suggestions for my bald eagle with three different crops of the same image). But these bizarre suggestions only started happening to me a few months ago, if that makes a difference to sleuthing the problem. I haven’t changed how I’ve been cropping, and the suggestions used to be spot-on.
I usually crop my images (from either my Canon Powershot SX70 HS or my iPhone 13 Pro using iOS 16.3.1) on my iPhone using the photos app, and then submit them from my phone.
@carrieseltzer – do you know if the developers have also looked at possible unexpected cropping (or something like that) as a potential source of weirdness in the CV suggestions in the web uploader?
Is this iOS specific? I was seeing the same behavior as OP with the same images on Android.
There are two things happening:
“Cropping Change” is a slight modification to the way images are prepared before they are sent to the CV model that resulted in an average 2.1% improvement.
This is the center cropping that you correctly deduced. The previous method was basically a “squish” that can distort proportions (e.g. make an elongated leaf look like a wide one).
