I’m very impressed with each new iteration of the Computer Vision model. That said, I still manage to occasionally/regularly upload images for which CV has no confidence in identifying. HOWEVER, one curious thing happens sometimes: When CV is throwing out guesses as to what my obscure plant or animal might be, I am sometimes seeing taxa which are totally unrelated (not even the same kingdom) but which I have previously uploaded in hand-held images or images in which my hand or most often a few fingers are prominent in the image. At first I was a little confused, then it began to feel a little creepy. Could CV be “learning” about my hand/hand print/fingerprints and associating certain details in its identifications of taxa I’ve uploaded (particularly those for which I’ve uploaded many images)?
It’s discussed here https://forum.inaturalist.org/t/fingers-in-photos/30145
I don’t think it learnt any fingerprints, cv learns fingers as part of photos it learns on, so it’s possible that is the reason it suggests those taxa, but also there’re many weird suggestions it shows just because it can.
My guess would be that hands and fingers have been included in photos of so many different taxa that CV treats their pixel patterns as noise instead of signal.
I’ve noticed that if you upload a picture of an empty beach, the CV model will often “see” shorebirds. In a picture of a creek it will “see” frogs, in a city street it will “see” pigeons, even when no organism is present.
CV doesn’t just isolate the organism, it takes the entire picture into account, so I assume hands and fingers have at least a small effect on the prediction.
If the dominant number of images for a single taxon includes your fingerprints then it seems entirely plausible it could trigger the autosuggest.
But… so what if it did?
The model is trained to predict species, not use fingerprint data for nefarious reasons.
It “learns” only in the sense that it looks for patterns and uses these to create the autosuggest.
It doesn’t learn in any broader sense or equivalence to human intelligence.
So connecting the taxon to your fingerprint temporarily ( until better training data is provided ) is meaningless.
But if you don’t want photos of your fingerprints to be used for nefarious reasons, best just not to upload images including your fingerprints! Although the iNaturalist model isn’t harvesting them or using them in any bad way… the original photo, or set of photos could certainly in theory be used by someone, somewhere, at some point to do some-thing.
It would make sense that the CV would begin to associate fingers and hands with plants, fungi and insects as those are things that people are most likely to handle when taking a photograph. Live birds, mammals, large reptiles etc. are far less likely to include human fingers for obvious reasons.
Agree with the other commenters - it’s likely just part of the image recognition. We see a similar thing with spiders - there is a type of small spider called “Wall spiders” (Oecobius sp.) - common in North America and usually found on flat surfaces. Because of that, the Computer Vision likes to recommend Oecobius for anything with 8 legs on a light background. So any type of spider found on a wall is likely to get IDed as a “wall spider” because the CV is trained on the background of the images. I imagine a similar thing is happening with your hands/fingers.
I don’t know – I suspect iNat knows more about us than it’s letting on. Would make a nice little conspiracy theory for those so inclined.
@sbushes In truth, I’m not worried about any nefarious use of my fingerprint data. My concern is more general, for the proper functioning of CV for any images with hands/fingers in them. As @calibas has indicated, the “interference” of the background milieu of a photo may becoming a measurable distraction in the “mind” of the CV model. As image numbers increase and the breadth of CVs taxonomic schooling increases, the “finger issue” may began to inject a measurable bias into the suggested IDs–at least regarding any taxa for which it trains on some of my “digital” (pardon the pun) images.
As for keeping hands and fingers out of images, that’s often not an option. My common field equipment is one or another trusty point-and-hope camera (e.g. Canon PowerShot cameras). As a group (I’ve heard this complaint from just about everyone using them), these tend to be unable to focus on small or narrow objects in a wider field of view such an insect on a twig or an isolated flower, etc., so my fingers end up in thousands of my images as a focal point for the digital sensors in my little camera. Even with my hand-holding efforts, I lose probably 10 to 20% of my images because the camera wants to focus on something else in the distance. Were I to not put my fat fingers in the field of view, I would basically be unable to photograph 90 to 95% of small objects of interest to me (insects, flowers, etc.) which are separated from the background by any distance.
I would say that in the “mind” of the CV, there are no such things as distractions or subjects, just the image as a whole. The CV has no idea that there is a spider or a fingerprint, per se, it’s just matching patterns to correct classifications it receives.
The CV certainly does use (and has used in the past) information from the whole photo including the background. This will continue to be the case unless a different type of model is employed. In some cases this use of the “background” can be useful, as when the CV sometimes “sees” a small common pollinator in a picture of its common host plant. In this case, the CV “knows” that pictures of the host plant are associated with these two taxa.
In my experience the CV certainly does pick up on fingers/hands in some photos for taxa that are often pictured that way. And while fingerprint identification from social media posts does seem possible, this is with techniques specifically designed for it. I find it highly unlikely that the iNat CV would ever learn to distinguish between human fingerprints given the way that it works.
If there’s any risk, it is just from posting pics of fingerprints online in general, which could then be harvested and processed separately. I doubt iNat is any worse than any other site for that, and, of course, you can always crop or take other measures (gloves, etc) so that your fingerprints don’t show if that is a concern.
I too have a camera which sometimes can’t focus on a flower/insect/etc without some help. I hadn’t thought about whether using a field guide or piece of clothing or whatever instead of fingers might produce better or worse computer vision results, but I suppose it is an interesting question to ponder and maybe even experiment with.
It’s also worth noting that before images get fed into the CV, they are resized to 299 x 299 (source), small enough resolution that unless it’s a photo taken with a strong macro lens, the detail on the fingers should be washed away.
@kevinfaccenda That’s really important information to know, and frankly I’m quite surprised at that. IF the CV is ever to learn to ID similar small subjects like moths, it is unlikely to be able to discriminate between/among similar species (even if the subject fills the image) with that kind of resolution. I’m having a hard enough time trying to distinguish some of these small, plainly-marked species with the full resolution of original images displayed on my large computer screen!
I think they are down scaled so much because as the resolution increases, the time spent training the model increases exponentially. Somehow it does manage to work on small subjects though, and how it does that is honestly very impressive.
I carry a small card, which I use instead of fingers sometimes.
We have been told before that iNat is trained to look at ‘ordinary’ pictures of whatever. For the wall spiders it is trained on the pictures observers choose to take. Some that show stunning detail! Some that are just a blur or a smudge. The bulk lying somewhere in between. I need iNat to CV my photo - which won’t be a super macro in a laboratory setting, but the best I can do before my spider scuttles off.
No, cv learns moths perfectly.
As image numbers increase, it seems unlikely that shots with fingers in them will become dominant in a training set - more likely the opposite. The more photos without fingers, the better the CV will discern the true subject. If this is an issue at all, it’s likely a temporary one which will only get better with time.
It certainly happens that common habitus has influence ( e.g. dung can be a trigger for autosuggest of dung flies ). But unlike specific habitus, fingers are non-specific and used across a wide variety of taxa so more broadly, the CV should sift them out as background noise.
Note also that the CV trains predominantly on the 1st image in a set. When uploading, if you really want to feed it more training data without fingers, you should put an image without fingers first.
Indeed, that is what I try to accomplish in my uploads when I have multiple images of a plant including general habitus and close-ups (with fingers) of flowers, leaves, etc. I’m not always successful, but I try.
Most observations I have come across of “American pika” are just a mountain landscape of scree slopes, with no organism present.
This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.