The iOS photo app has a “Look Up” feature that automatically labels photos with categories like “Piano,” “Human,” or organism names. In the past, it gave broad labels like “Spider” or “Bird.” Now, it often identifies organisms to genus and species, and it’s surprisingly accurate.
For example, it correctly identified a leaf-footed bug nymph to species:
However, when it’s not confident, it labels generally (like “bug”) or doesn’t try. It’s better at identifying birds, insects, and plants than fungi and mollusks. Unlike iNat’s CV, the iOS feature scans for the subject before identifying, which might explain its higher accuracy in some cases.
It identified all birds and plants I showed it correctly and most of the arthropods (which was fairly accurate given the variety of arthropods I was showing it). But it made mistakes, like calling a beetle a honeybee and an ant-mimic spider an ant. It struggled with mollusks and some fungi, misidentifying some inedible mushrooms as edible ones, which is concerning.
What do you think about this iOS feature’s accuracy and potential uses?
My experience of noticing the “Look Up” suggestions is that if the organism that it is “looking up” is unique an has the necessary characteristics on display then it can be accurate. Can very much being the key word there. However, I’ve found it to be wrong quite often. I don’t think it takes location data into account like the iNaturalist CV does and so I’ve had it suggest species not present in the UK (my location), even when what is photographed is quite obvious. I find the iNaturalist CV much more reliable. I was also put off by it when it first appeared on my iPhone when it popped up under a mushroom picture I took and it asked me if I wanted to look up the “plant”…
I also checked with a few more mushroom photos and it is indeed labeling them as “Plant”. Maybe someone who made the feature incorrectly thought that fungi were plants. I think someone manually puts categories under broader categories. For example, there is the “Bird of Prey” category and as I mentioned above, “Bug”, which includes insects, arachnids, and anything blurry enough to resemble a bug
I agree that iNat’s CV is much more reliable and consistent, but I’m curious to see where the iOS feature will go. I noticed that it occasionally suggests species that aren’t included in the CV, although I’m fairly sure those identifications aren’t fully correct. Either way, iNat is limited by what is observed and identified within the website
I don’t think it takes location data into account like the iNaturalist CV does
I don’t either - when I give it these pictures of a galled Tilia leaf it suggests Tilia americana for the underside (rarely grown here), and Malvaviscus arboreus (a subtropical American species!) for the underside. It’s probably Tilia x europaea (difficult genus though), okay, maybe not that far off on the first count.
When it’s a terrible image it’s not very good though. This Buddleja davidii was taken from a dirty train window - Look Up says it’s a “Brazilian peppertree” or “Christ’s Thorn Jujube”. Both are subtropical species that aren’t (can’t be) grown here. iNat does suggest the right species (B. davidii) but not first.
I notice with insects that iOS will assert completely different IDs if I check different pictures of the same organism. in fairness, I suppose iNat could do that, also. At least with iNat, I notice the CV will often say, it’s not sure, but it could be one of these ‘…’,
