A species or clade could be tagged as “starkly sexually dimorphic” if it has been tagged in this way and if a record is marked “male” or “female” it would restrict the comparisons by sex.
Identifying male ants is hard. Most ants are female. Males can be more rare and they look very different from females. Sometimes people think they are flies or wasps. Further there are fewer species keys and information generally for male ants.
It seems like the current system tries to compare male ants to every ant and so if a species has a lot of confirmed photos of male ants then every new male ant sighted is assigned to that species.
I keep finding new observations where there is a male ant and an incrediblly specific species ID. Normally I’d think “maybe this person observed the nuptial flight and therefore knows what kind of male ant this could be” or I think “wow someone who can tell male ants apart they must be some kind of expert” but increasingly I think it’s the recommendations. Because I ask “did you see the nuptial flight? do you have any photos with workers too?” Nope.
At this point I only trust photos where there is a worker or queen so I can verify the male. I think a lot of noise is being added to the data by having species IDs on male ants like this.
Is it possible to compare males to males?
I bet this would help with other species where the differences are stark.
(It would be nice to also only compare queens to queens… but that’s less general and maybe too ant-centric… if you can imagine such a thing.)
Good point. I have seen a different but related issue with the classification model. I uploaded an image of a white-tailed deer that was slightly behind a tree. The deer was centered in the photo and still easy to see. However, the model suggested 10 species, all of which were birds.
There were no birds in the image. The photo contained only the deer, some dead branches, and a pine tree in the background. My assumption is that the model learned an association between birds and branches. As a result, it may interpret the presence of branches as evidence that a bird is present, even when there is no bird in the image.
This was not an isolated case. I have noticed the same behavior when documenting trees. The model will sometimes suggest a bird or another animal that commonly uses that tree, even though no animal is visible.
This can create a harmful feedback loop. The more often the model suggests a species, the more likely users are to select it. Those observations may then enter the training data, which reinforces the same error in future model versions.
My Ph.D. research had focused on prediction. Based on my experience, a better computer vision model may reduce the problem, but computer vision alone cannot fully solve it. This is primarily a data preprocessing and validation problem. There needs to be careful selection of which observations are included in the training set. The model could also be trained using soft labels that represent identification uncertainty, rather than treating every identification as equally reliable and definitively correct.
iNaturalist is currently hiring a computer vision engineer. Hopefully the new hire can help address this issue.
It’s also possible that the visible portion of the animal, say the head and ears, has similar cues for the CV as a bird does, so it may not be an association with the trees, but that since the entire animal is not visible the CV system treats different visible portions as belonging to different animals rather than one animal.
But careful selection of photos to include is likely to just mean it’s good on equivalent photos uploaded and hopeless on standard iNat photos (because your point that careful selection may be required implicitly suggests that the average photo won’t give ‘good’ results). Yes, the CV can make a complete and utter hash of things (I’d love to be able to select the kingdom when uploading to avoid the times it fixates on the plant the insect is on or vice versa) - but I don’t think changing the training data actually fixes anything. If anything, it may just get more confused on typical iNat photos! After all, it’s set so it will suggest something for any given photo, even if that photo contains nothing, or nothing living.
The fact of the matter is that the CV is purely pattern-matching (well, more or less - the geomodel comes into it too, and probably other things). It’s not doing what a human would in terms of counting legs or recognising the specific focus of a photo - it’s just finding the (local) species with photos that average most like this one (over-simplified, but my point stands). I think it does an amazingly good job, but that doesn’t mean either the observer or the identifier can turn off their brain and just select the top suggestion.
@grampianshiker I agree that if you use the current model to decide what is a good photo for the next training set, then you have a dangerous feedback loop. This is a great point as that could be a simple, but ineffective way to do data filtering. That is not what I am suggesting.
I am suggesting an entirely separate pre-processing pipeline. The purpose would be to construct a more selective dataset used for training, with confidence labels on each observation. Importantly, this pipeline would be completely independent from any of the CV models (past or present).
I also agree that CV is essentially advanced pattern-matching. If the model only relies on CV, then it can get good, but it will always have limitations. My recommendation would not be to make larger, more complicated CV models, but rather to have more intelligence built around it.
I wasn’t suggesting using the model to suggest photos for the training set either - merely pointing out that ANY selected, and therefore non-representative, sample of photos is more likely to struggle with ‘normal’ photos than what we currently have. Because let’s face it, many of the photos that get uploaded are a long way from the ideal training photos, mine definitely not excluded.
I agree that the CV performance on taxa where some individuals are more difficult to identify than others (certain life stages, certain sexes) tends to be poor, but I don’t think this would be an effective way to solve the problem.
The problem isn’t that the CV can’t cope with sexual dimorphism per se – there are many sexually dimorphic taxa that the CV has no problem recognizing in both sexes. A prominent example: birds. The CV is also capable of recognizing other species where there are multiple visual patterns (e.g., larvae vs. adults, flowers vs. leaves vs. seeds) – provided that the training set includes enough observations identified at species level of all of these different forms and appearances.
The problem you are seeing with ants is that there is an asymmetry in what is identifiable. Females get identified more precisely than males and the CV is trained on the narrowest possible taxa, so observations with a broader ID (for example: males; observations of nests without any close-up images of individuals) do not get included in the training. In a few cases there will be identifiable males, so these photos might get included in the training of the corresponding species and the CV will then over-suggest this species for males because it does not have other examples of males.
As I noted in the other thread, the CV isn’t currently trained on annotations – all images of a taxon go into the same training set and the CV looks for patterns among these images, but it doesn’t know what any of these patterns mean. I think incorporating annotations into the training would be an important step for improving the CV performance, particularly if it could be combined with training some taxa at higher levels. One obstacle here is that for a lot of taxa only a small percentage of observations are annotated, so there may not be enough labeled material for this to be feasible at present.
Another technical hindrance for your proposal is the fact that annotations can only be added after an observation has been uploaded, while most people are using the CV before upload.
Well, “amazingly good” depends on what taxa one is interested in, but I basically agree.
And I absolutely agree that encouraging users to be more conscious and critical when using the CV is an important part of the equation, and I’d like to see more efforts to systematically educate users on how to use it responsibly. It should be a tool, not a magical answer-machine.
Okay, I understand what you mean now. This is one reason I mentioned assigning confidence scores to the observation labels instead of assuming every label is equally reliable. Those confidence scores could then be incorporated into the training process using soft loss functions, which may lead to better-calibrated uncertainty estimates from the computer vision model.
The key idea behind my proposal is that there are two related, but distinct, questions:
What organism was actually present?
What identification can the photographs support?
A correct observation label does not necessarily mean the photographs contain enough visual evidence to confidently identify that organism. Likewise, the opposite can also occur: the photographs may provide strong evidence for an identification even if the observation label is incorrect.
I do not know the details of iNaturalist’s current training pipeline, so I cannot say whether these two questions are already treated separately. If they are not, I think explicitly modeling them could be an interesting direction for future research.
To be clear, I think iNaturalist’s current computer vision model is outstanding. It is remarkable how well it performs across such an enormous number of taxa. As a research scientist, my role is to think about how algorithms can be improved further. That does not diminish what the iNaturalist team has already accomplished on a very difficult problem.
Another big problem in this area is the taxon photos. It’s a constant annoyance when using the CV with larvae, for example, that the list of suggestions often only shows the adults for many taxa. I really wish we had separate sets of taxon photos for the life stages (and possibly also the sexes). The larvae/nymphs of many taxa go through several stages, often with each stage looking radically different to the previous ones. And as already noted, sexual dimorphism can also produce radically different adult forms. It would be very helpful to have a more structured way to represent all that variation in the taxon photos.
Ideally, if the CV ever got better support for annotations, it would automatically only show example photos for the relevant life-stage/sex so as to make comparisons much easier. But if that wasn’t possible, it would still be very nice to have some way to manually filter the example photos so that a more relevant subset could be shown (i.e. similar to the filters in the photo-browser).
Exactly what happened with Platypezidae where basically all males were just suggested to belong to Lindneromyia (probably still, I’ll find out in Autumn…)
I wonder whether there’s a significant difference in the pattern-matching CV does in these cases vs. not. Or generally in case of a mis-ID vs. a correct ID. And whether those differences could be used to limit ID-specificity at a broader taxon in these cases.
If it is possible to visualise, I would generally be interested to see what “features” of the image CV looks at for various taxa.
I agree that the CV could be improved for highly variant species, but I don’t think any new tags or annotations are required. Instead, this can probably be achieved by tweaking how the training sets are selected. If observations of a Lepidoptera species are 90% butterflies and 10% caterpillars, the CV training set might include only adults in some training rounds. To avoid this, when there are sufficient observations, the random training set selection could take annotations into account to ensure that there are representative samples of eggs/larvae/adults, females/males, fruits/leaves/flowers, etc. within the training set. i.e. the training set should be biased to include more of the less frequently observed variants.
This problem is referred to as “interpretability” and is still an area of active research in the machine learning (ML) community. ML models are largely black boxes, and the models (probably) aren’t identifying “features”, at least not in the way humans do. Figuring out and visualizing what CV models are actually doing is very challenging.
The problem is that certain individuals of some species aren’t represented at all (or only represented in observations that include genitalia or other characteristics that can’t be seen in field photos) because there is no known way to distinguish them from similar species in that sex or at that life stage based on normal field photos and thus the observations of males or caterpillars or nests have IDs some level broader than species.
I will continue to advocate for the inclusion of internode training. If only males are IDable to species, and females get stuck at genus. If it could learn the genus. The problem would likely be mostly solved. Though i doubt it would be perfect, nothing is.
This is a troublesome issue with Lycosidae. Some are significantly sexually dimorphic and need two different field ID guides. Another problem in dividing up observations is that juveniles are different from adults and need at least a third guide for late juveniles. Part of the problem is the observations themselves. Until more observations include good ventral views, it’s hard to know what most are. Perhaps it’s already possible to search for female/male or adult/juvenile. I know this can be done from the taxa page, but it would be useful to filter for in “Explore”. An additional problem for these is that the Taxa image which only shows one individual.
On the bright side, if they’re that starkly dimorphic, the CV is likely basing its ID of the females on their similarity to other female images in the training data. For comparison, there are a lot of images of deer poop, footprints, bones, and antlers in the training images for white-tailed deer. But these have not had any negative impact on the CV’s ability to recognize a photo of an actual deer. If the training sets of some ant species are contaminated with images of random males that look nothing like the females, those male images probably aren’t impacting the ID suggestions for females.
I imagine this same issue would happen even if the CV were trained separately on males and females. Once one species’ male has enough observations to get recognized, that’s where all the males will be sent if none of the others have enough male images to be included in the CV. In fact, the problem may even be exacerbated, since the CV won’t be trained on any images of the males of species with under 50 male images (the threshold to be included in the CV). At least now there are a few male images in the training set for a bunch of species.
It depends – if the CV is trained separately on different sexes, theoretically this could also use different CV inclusion criteria, i.e., males would be trained at whatever broader taxon (genus or subgenus or whatever) reaches the threshold of number of observations/photos, while females which are better able to be ID’d are trained at species level.
Agreed, which is why it once again comes back to the issue of genera being excluded from training once a species is included. If one species’ male reaches the threshold to be included, the CV won’t be trained on any of the male images at genus any more.