(This might be a quirk more than a bug exactly, but I wanted to point it out to see if it sparks any thoughts. Maybe others can verify this behavior? I’m using a Win 11 laptop running Chromium Edge browser.)
I knew that the way the Computer Vision (CV) evaluates images on the web Upload screen is different from the way CV evaluation works on the Observation detail page, but I thought the differences were conceptually minor. I always thought both were effectively evaluating a resized image, but it seems like the web Upload page might be cropping the image (as a square, keeping the right side? center) before resizing and evaluating it.
Here’s an example case. Open the Observation detail page for this observation: https://www.inaturalist.org/observations/212423206. If you look at the suggestions that the CV provides on that page, they seem reasonable (Turkey Vulture):
Now I’m gong to upload different variants of the image in the web Upload screen and compare the suggestions:
square-cropped 2160px x 2160px image (keeping the right side of the image) → Swifts (same as suggestions for original image, but no higher-than-species suggestion):
The suggestions for the uncropped images are bad. The cropped-keeping-right image seems to match those bad suggestions more or less, and the center-cropped image suggestions seem to be similar, too. So it seems like the suggestions on the Upload page might be based on a cropped image of some sort.
(I didn’t test what happens for an image that is taller than wide, but I assume it’ll have similar results.)
I’d be interested to see, if you flipped that first photo horizontally (so the vulture is in the right side), does it indeed use the right side of the image to give better suggestions? That would be a way to test whether your suggestion that it’s keeping just the right side is correct.
that’s a good suggestion. it looks like after flipping, the CV suggestions are still bad. so it must be effectively a center crop. also, maybe it’s not a crop then resize. maybe it’s just a reize based on the minimum dimension (instead of based on the max) and then a crop after that. (i guess i could just look at the code to see exactly what it’s doing.)
it’s worth noting that the aspect ratio of the image is 16:9 (since it’s a still from a video). so it’s relatively wide versus tall. if it were a 3:2 or 4:3 aspect ratio instead, it would be much less likely that the subject would ever fall outside of the center square. so maybe that’s why the weirdness is hard to notice.
hmmm… i don’t have an iOS device to play around with, and it’s going to take me more time than i’m willing to spend to study the iOS app code to figure out exactly what it’s doing in your case. it looks like it’s using the same computer vision API to evaluate images as the web Upload screen, but i haven’t figured out exactly what the iOS app is sending to be evaluated. it looks like it’s trying to send a small cached version of the image, but i haven’t figured out what that means exactly.
i’m guessing it might be doing some sort of crop because if i do a crop to get the center 240px x 240px (without resizing), i get a mix of results akin to what you’re seeing:
however, the question in my mind is: why would you see a problem in this particular case and not in other cases? are you seeing the problem only here? if so, is there something different about this photo vs other photos? is there something unusual about the process you used to upload in that particular case?
ultimately, i think your example is probably a little different than whatever is going on in the web Upload screen, but in both cases, i’d guess that they’re probably doing unexpected (but probably different) transformations of the image that they’re sendng to the CV API to be evaluated.
I can’t be sure, but I think this is the first time I’ve had vertebrates suggested first on a moth photo. The all white background of the sheet confuses the CV a fair amount, but it usually associates it with some type of insect. Thanks for taking the time to look into this.
I don’t know about iOS, but it is possible to crop in the Android app. You have to add the photo to the observation, then click on it to open it in edit mode. I recently discovered that this can be done after upload, too, by opening the observation for editing.
That’s exactly how I crop my photos too (iPhone with iOS 16.3.1), before submitting them using the iNat app. Here are some screenshots of iNat suggestions for my moths (notice it got the Three-spotted Fillip right when I used an uncropped image).
