Georgia Tech ML Research Partnership — Calibrated Species ID for Backlogged Taxa (Starting with Amanita)

General
1

Hi everyone,

I’m reaching out from Georgia Tech, where a team of MS students and I are developing a calibrated ML classification pipeline that could help with the backlog of observations that still lack species-level IDs.

The problem we want to help with
Many taxa on iNaturalist have a large fraction of observations which are not research grade due to missing a species-level ID. Groups like the mushroom genus Amanita have thousands of observations that could potentially be identified but remain unreviewed, presumably because there simply aren’t enough expert identifiers to keep up.

What we’re building
Our system is designed around the key principle: only submit IDs when the model is confident enough to be right. Rather than optimizing for coverage, we’re building a calibrated system that withholds predictions unless it can achieve high-90s accuracy on the IDs it does make. The goal is to be a reliable contributor, rather than a noisy one. We can calibrate the system using existing and new species IDs from iNaturalist experts. From preliminary results, we expect the system to add IDs to the relatively common species and more clear-cut observations, and leave the rarer, more difficult cases to the experts.

We’re currently prototyping with the genus Amanita, chosen because it’s a large, well-photographed genus with a substantial fraction (a majority) of observations still needing species-level IDs. Additionally, it has many observations with multiple photos which we’re looking into ways to use to make more accurate predictions.

What we’re hoping for
We’d love to connect with iNaturalist staff and community members to discuss whether this kind of tool would be welcome and how it could fit into existing workflows. We are aware that the iNaturalist Community Guidelines (link) explicitly prohibits “Machine generated observations, identifications and comments”, so as law-abiding citizens, we won’t upload anything to iNaturalist unless we get a green light from iNaturalist staff.

We want to build this with the input of the community. If there are concerns, norms, or prior discussions about automated IDs that we should be aware of, we’re all ears. I see a couple posts in the last couple weeks about the problem of users relying on third-party GenAI systems as authoritative sources.

Happy to share more details about our approach, and we’re planning to open-source our work so others can build on it. Feel free to email me at mussmann@gatech.edu if that works better for you.

Thank you for reading,

Steve Mussmann
Assistant Professor
School of Computer Science (SCS)
Georgia Institute of Technology

https://steve.mussmann.us/

2

is lack of expert identifiers really the underlying reason most fungi don’t get identified to species readily?

i don’t think adding identifications using your AI would be a great idea. at the minimum, you would want to have a process where someone could readily respond when another identifier comes along and challenges your identification. ideally, that someone would be able to explain why the particular identification was made, other than saying that an AI decided that was the right identification. that communication is sort of fundamental to the ethos of iNaturalist.

also, many folks have a strong dislike of anything AI. so i think your process would need to account for this and include a way to exclude particular observers from getting caught up in your project.

that said, there are other ways to add information, other than making identifications. you could add comments, you could add observation fields, you could set up projects and addd observations to those projects. this doesn’t solve the machine-generated content issue, but i think this sort of machine-generated content is less bad than adding identifications.

maybe the best way to help the community and mostly avoid machine-generated content issues would be to provide folks a mushroom AI suggestion tool (browser extension or website) that folks could use if they chose to. that way, you still have a human making the final decision, and you could set up your tool to figure out when the human chose something different than the AI suggested.

7 Likes
3

https://www.inaturalist.org/pages/machine_generated_content

4 Likes
4

I was under the impression that Amanita are often difficult to ID from photos, and that many observations fail to attain Research Grade because the characteristics needed to ID them are not shown in the photos taken. Certainly starting with the genus that’s responsible for the most deaths worldwide seems… unwise.

The bigger issue is that iNat already has a Computer Vision tool to help ID taxa. Besides seemingly re-inventing something that iNat has already spent thousands (tens of thousands?) of hours working on seems like a duplication of effort. Adding another ML tool would potential just confirm the CV suggestion with another computer-aided suggestion, taking any human element out of IDing taxa. That is something that iNat has been against for a long time.

12 Likes
5

Amanita is not the best choice for this as other have said

6

While I appreciate the ideas presented in this proposal, there are some serious issues you should consider before moving ahead with it. First, the backlog of observations on iNaturalist that lack species-level IDs are not due to a lack of automated solutions, or in the case of Amanita, because of a lack of expert identifiers. In a huge proportion of cases, the species cannot be reliably identified from photos (or at least from the specific photos given). In fact, a lot of the work that expert iNaturalist volunteers now spend their time on is correcting species-level IDs (often suggested by an over-confident CV model) and changing them to genus-level IDs. The existing iNaturalist CV model is decent, but has some major problems. Since nothing in your post suggests awareness of those problems, I’m afraid your team would repeat the same mistakes.

The main problem any species-IDing ML tool will face is the fact that most of the described species on Earth have no published photographs and the vast majority of the species on Earth (~95%) don’t have enough published photos to train an ML model. This means that any ML species model will only know about a small fraction of the possible species, and its 99% confidence will not translate to 99% accuracy. Unless it’s a group like birds or mammals, IDing everything to species just isn’t possible. Even for common species that seem easy to ID, there are very often obscure closely related species that make it impossible to confidently ID the common species, especially if its just from photographs. For most types of organisms, ML models should ID to higher taxon ranks like genus or even family. If you can build a model that knows how and when to do that, you might build something useful. If you model doesn’t take those limitations into account, it will just be another toy creating garbage for volunteers to clean-up.

There are other discussions on these forums discussing more problems with the existing iNaturalist CV and I highly recommend seeking those discussions out before starting on any similar project. One other issue that comes to mind is the existing CV’s inability to distinguish foreground and background features which often causes overfitting to irrelevant background features. Anyway, I don’t mean to discourage your efforts, but please, please do your research before you charge ahead with this project and talk to some actual taxonomists, not just computer scientists. Computer scientists almost universally suffer from the Dunning–Kruger effect when it comes to the idea of identifying species. It’s a much messier and thornier problem than most people realize.

Oh, and as several people have mentioned, starting with Amanita is a very bad idea. Don’t do that. In fact, I would avoid fungi entirely since iNaturalist has plenty of expert fungi IDers (maybe more than any other group) and fungi taxonomy is in a state of total chaos currently. Plus you might accidently kill people.

8 Likes
7

Contacting staff using the iNat email would be a better way to connect with staff than posting in the forum.

As others have said, iNat’s policy on machine-generated content that specifically bans automated IDs would probably make this a no-go from the start. Other proposals, for example, to add automatically add a provisional CV ID for observations that were uploaded without an initial ID after some amount of time has passed, have been rejected.

iNat’s observation validation system is also based on the basic principle of observations being reviewed by at least one other person besides the observer. Using an algorithm to confirm observations would mean that this no longer happens.

If you and your team want to use machine learning to benefit iNaturalist/biodiversity recording and are not just looking at iNat as a convenient source of data for developing and testing machine learning applications, I recommend starting by familiarizing yourselves with iNaturalist.

The iNaturalist community has mixed feelings about the existing machine-learning ID suggestion algorithm (computer vision), as it creates a number of issues connected with validation of data, in particular for species that are difficult to ID from photos. There are also other frictions connected with the human-CV interface (e.g. how people use the CV suggestions and whether they use it uncritically or as a tool to be supplemented with their own research).

Your description of your project suggests that it would replicate rather than address these existing problems. This in particular gives me cause for skepticism:

“Confidence” is not a good measure of correctness. If the system is only trained on a subset of possibilities, it may be highly confident simply because it lacks information about alternatives. There is also the problem that a machine-learning algorithm can’t critically assess the image material because it doesn’t know what it is seeing and it does not have the option to say “none of the above” or “something doesn’t fit”.

If you really want to work on something that would benefit the iNaturalist community, I agree with the advice to read some of the discussions on existing challenges with the CV and see if there are ways you can tackle some of the issues which the current training does not adequately address.

It also seems to me that for a project like this you ideally want someone on your team who is not a computer scientist but is familiar with identifying difficult taxa (many insects, or fungi as in your prototype) and can help you determine what approaches are working or not working and can provide feedback about the sorts of things that are relevant for identifying their taxon. (While some problems with iNat’s CV are connected with the training limitations and broadly apply to many taxa, I suspect that particular taxa are pose specific challenges not present for other taxa – there may not be a one-size-fits-all-taxa solution.)

7 Likes
8

@stevemussmann - Have you considered the idea of building an ML tool that identifies organisms to genus instead of species? That would actually be really useful and would avoid most of the problems that come with doing species-level IDs. If iNaturalist’s primary goal was creating useful and accurate scientific data rather than maximizing engagement (“100 million naturalists by 2030”), I think they would have taken a more conservative approach with their CV, given many of the inherent limitations discussed above.

6 Likes
9

just to clarify, because many people have misinterpreted that goal/specific number or had some qualms about it: as I understand it, that 100 million figure includes all forms of engagement with iNaturalist/its data, including scientists using the data (whether through iNat directly or via GBIF, including in combination with other datasets), teachers + educators using data in classrooms, people casually browsing images, people liking iNat’s Facebook posts etc (with all of these user types being classed as ‘naturalists’). From memory there are already something like ~40 million people ‘engaged’ with iNaturalist in some way (although obviously difficult to calculate an exact number); a huge portion of these people are not, and are not intended to be, active observers or even have an iNat account at all, but rather people ‘indirectly’ engaging with the site and its data in the many many ways that is possible without having to upload a photo or make an ID. It is absolutely not a goal of iNat to have 100 million observers/users using the platform by the 2030, I can say that with 100% certainty. I definitely think this could be made clearer though in the messaging.

So I don’t think

and

are mutually exclusive from this particular perspective.

3 Likes
10

Luddite here (although an active iNat observer and identifier): what does ML stand for?

11

I agree with others who’ve noted some serious issues with this proposed approach.

The biggest is that iNat has a prohibition on machine-generated content as noted in the proposal. iNat is focused on human interaction with nature, not just assigning an ID to everything. iNat could easily add an automatically predicted ID (at any score threshold) to every observation if they so chose, make that ID automatically visible, etc. using the existing CV model. But they have chosen not to. I think it is highly unlikely that iNat would change to allow machine-generated content like this to be submitted as it doesn’t align with their mission and because users would likely be strongly opposed (based on previous discussions of the use of AI tools and iNat).

Additionally, it isn’t clear to me how the proposed approach (a bespoke model for Amanita) would be an improvement based on the summary. All the species/taxa within Amanita that meet iNat’s thresholds for inclusion in the CV model are already in the model. iNat’s CV model also generates a confidence score (though the number itself intentionally isn’t shown to users, it is accessible) so the proposal isn’t novel here either. Is there a reason that the proposed model would be significantly more accurate than iNat’s current CV or preliminary data to indicate this?

As others have noted, Amanita would be a particularly challenging set of observations for ML modelling for several reasons. I think the biggest issue for making an ML tool for this genus is that the current iNat data (which would be both training and test) is impacted by several structural issue. One big one is that fungal taxonomy in general is very in flux, so the reliability of the training data on iNat is uncertain in that way - different fungal groups are in various stages of revision/knowledge about their diversity. There are lots of undescribed taxa that only a few experts with DNA sequences know of, and this knowledge is variably incorporated into iNat (sometimes with observation fields, not at all, etc.).

Experts know that many species/taxa are various degrees of impossible to distinguish visually/based on photos alone, and so they don’t add IDs (or only add them to few observations that meet the conditions in which a reliable ID is possible). It generally isn’t possible to readily distinguish whether an observation hasn’t been IDed because a) an expert reviewer hasn’t looked at it or b) lots of experts have looked at it and not added an ID because they’ve assessed it isn’t possible to accurately ID further. An additional issue that others have noted is that a fair amount of existing RG observations in fungi generally may be incorrect. These RG IDs are often the product of less experienced and/or overconfident IDers, often with at least one of them using iNat’s existing CV suggestion (which can create overconfidence loops). Fungi experts are often bumping back overly specific IDs that aren’t supported by the evidence to higher levels as opposed to adding specific IDs. As in the thread above, fungal experts often wish the CV to be less specific rather than suggesting species level IDs. All of these factors together (and maybe others) create a situation in which getting a reliable training dataset and finding an unbiased test dataset will be very challenging.

I don’t mean this too negatively, but the initial post here and the choice of Amanita for testing/developing a tool indicate to me that there are some major issues with understanding how iNat works. I think it would probably be beneficial to get more experience using iNat and its features and the community before trying to implement a project like this (not a personal criticism, but assuming your iNat username is similar to the forum, it looks like you’ve only been on iNat for a few months and made <10 observations/IDs).

One final constructive suggestion - I do think that there are probably some CV-focused projects that could be useful to iNat. If there are fundamentally/structurally different approaches to incorporating different types of data (location, phenology, comments, something else) into training a model that lead to demonstrated performance gains, that could be very useful. Another long-standing desire of many users for the current CV implementation is to suggest higher level taxa more frequently (e.g., when there is imperfect knowledge/inclusion of its descendants) to prevent overconfident specific IDs that experts need to push back.

6 Likes
12

Machine Learning

13

Ah, thanks!

14

I’m not clear on what this offers that the current CV does not offer, and furthermore, a big issue for many clades is that you need really, really specific information to ID something down to species, information that just isn’t present in the photos a lot of observations come with, or that just isn’t possible to capture in the morphology in a photo. For example, as far as I know, there is no way to morphologically distinguish Araucaria heterophylla vs columnaris seedlings. I am told that it is basically impossible to distinguish several species of mallow when they are not in flower, so basically none of my Californian mallow observations from last November and December are identifiable to species because they happened to not be flowering. I hear that for many insects, you need to have specific photos of the genitals, so if the observation doesn’t have that (likely, since insect genitals are very small to the point you might need a microscope)

1 Like
15

As I read it, they propose to insert AI generated IDs. The current CV does not, leaving that to actual humans. There being a backlog based around qualified humans, they propose these are replaced with machine learning / “a calibrated system” .

(Again, as I, a human, read it.)

16

@stevemussman You only just recently joined iNaturalist. As of now you have very few observations and perhaps more to the point you have contributed no identifications for others at all. May I suggest to use the site the traditional way for a little in order to get experience with what the process looks like for the rest of the community.

7 Likes
17

I suspect there are a number of invertebrate taxa where some species in the genus can reliably be identified from photos, and the theoretical identification knowledge exists on iNat, but there’s still a large backlog due to a poor ratio of identifiers to observations. My impression is that this is even increasingly becoming the case with birds, but there is still plenty of identification effort in that direction. So I think there’s probably a genus where this could be tested and theoretically be helpful, but as others have said it’s unlikely to be in fungi.

A lot of backlog for many obscure taxa consists of overconfident CV suggestions that need to be pushed back to a higher level, or reidentified to a similar species with care (outdated list with many such examples here). Theoretically AI/ML-assisted identifications could be helpful with this if they are more cautious and careful than the naive observer/default CV combination that created the original identifications.

1 Like
18

Do experts exist which will be able to identify, with near 100% certainty, which species of Amanita each photograph represents? Without that, how will the ML model be constructed?

6 Likes
19

For Amanita the problem may actually be even worse, because I can for sure see people using iNaturalist to try to identify edible mushrooms, and there are a lot of mushrooms that look very very similar between edible and deadly species…it seems some species need genetic sequencing or other lab techniques to be identified, or at the very least other factors like time of year/season and spore print. If people are potentially using a computer to try to tell if this mushroom will kill them or not, we need a very, very high level of certainty that they will not die from using it.

5 Likes
20

Can you explain this some more, please? How does the model decide how confident it is when it is looking at observations which have not been identified yet?