Good afternoon.
I am curious if external objects in photos can influence the computer vision in an unwanted manner? For example, if a ruler is photographed beside a flower (in order to measure, say the phyllary length), will this negatively affect the CV? I am starting to get back into observing, and would like to incorporate a ruler into most photos for identification purposes, and am wanting to ensure that it is acceptable to do so.
plantman4
If you are worried you can upload a second picture without a ruler. But the CV is trained to use pictures – from various users – with various cameras / cellphones – and varying photographic skills. Real obs by real people.
CV is understandably confused by ‘this bird is often seen nectaring on that flower’. So an obs of the flower can be IDed as the bird. Up to the identifier to do quality control.
You should never censor your own images to suit the peculiarities of the CV. It will have to learn that sometimes there’s a ruler next to this flower, just the same as it has learned that sometimes there are insects or birds.
Agreeing with DS. The CV training looks for commonality between many observation photos (the training set). Other objects or organisms will only start to confuse the training if they consistently appear in photos of the organism. This is often an issue with host-specific parasites or diseases. For example, the CV will often suggest Dinocampus coccinellae (a parasitoid wasp) for any photo of a lady beetle, because nearly every photo of the wasp includes a host lady beetle.
I’ve posted a few pressed plant specimens with a ruler visible along the bottom of the picture. I don’t recall the CV having an issue with that, but then again I might’ve just entered the species name and not used the CV. (Does the CV recognize pressed plants very well?)
It seems to me that photoing a plant with a ruler in the picture is not much different than photoing a bird sitting on a fence. Both have artificial objects in the image. EDIT: Unless you’re posting many observations of the same species with a ruler in the image every time. Then it might be a problem for the CV. In which case, post the first image in the observation without the ruler.
It’s a great idea to include a scale in some of your photos. I wish more people would do it. Heck, I wish I would do it!
As others have said, you may want to make the first photo one that doesn’t include the ruler, so that there’s no chance it might bias the CV algorithm when you or others ask for ID suggestions. But I wouldn’t worry about how objects like that might affect the CV algorithm more generally, in the scenario where some of your images are used for CV training. I believe the CV training set is up to 1000 images per taxon, and the expectation is that particular extraneous objects will not appear consistently enough to influence the training.
Now if you used a blue ruler in every photo for 1,000 observations of one aster species and a red ruler in every photo for 1,000 observations of a different species, I can see that CV would likely suggest IDs based on the ruler color more than the plant’s appearance. Hopefully, we’re all much less systematic than that!
[Edited to correct training set to a maximum of 1,000. I remembered wrong and should have looked it up. Thanks, Tony!]
According to the help page, apparently there are up to 5 photos used per observation for the computer vision (source: https://help.inaturalist.org/en/support/solutions/articles/151000170368-which-taxa-are-included-in-the-computer-vision-suggestions- )
EDIT: This is a good suggestion, though, thank you all!
We should distinguish between two different ways photos are used in the computer vision algorithm.
First, thinking about which photos are used to train the model, this definitely is not limited to the first photo in an observation. The algorithm starts by identifying eligible observations and then for each taxon it builds a set of up to 1,000 associated training images, largely taken at random from those eligible observations. One limiting factor is that no more than 5 images are taken from any particular observation (to reduce the chance that the algorithm is biased in favor of what that organism looks like when photographed by a particular observer or in certain conditions). In this context, photos with rulers are just as likely as any others to get included in the training set, no matter whether they’re the first, fifth or tenth photo in the observation.
Things are different when we consider which photos are used when suggesting possible IDs for an observation, either to the observer or to identifiers. In most of those contexts, only the first photo in the observation is sent to the suggestions algorithm as its input; the output is a list of up to 10 taxa that best match that first photo (as modified by the geomodel). There are a few circumstances where you can get the CV algorithm to suggest IDs for second and subsequent images, but mostly it just looks at the first. So, if you avoid including a ruler or other extraneous object in that first photo then you eliminate the possibility that CV might suggest certain species because they also had photos with rulers.
Thank you for taking the time to write out this detailed explanation!