Thank you for the constructive feedback and ideas!
In terms of the reasons for why most fungi don’t get identified to species readily, from a small sample, roughly 2/3 have multiple identifiers but not a species level identification (perhaps not possible to tell from the photos) but 1/3 have one person giving a species level identification but no corroboration.
Thank you for the link. I’m aware of the policy which is why I mentioned “we won’t upload anything to iNaturalist unless we get a green light from iNaturalist staff”.
Thank you for sharing your thoughts! I did a small sample and found that of the ones that “need ID”, roughly 2/3 have multiple identifiers but not a species level identification (perhaps due to the reason you mentioned) but 1/3 have one person giving a species level identification but without corroboration.
I completely agree with the duplication of effort. We wanted to scope out a decently large project that could be scoped/specified clearly since it would be hard to work on smaller tweaks externally to iNaturalist. We thought the calibration/confidence aspect would be different enough, but I agree there is a large amount of overlap. If you have any other directions for projects, we’d love to hear them!
Thank you for your detailed feedback!
Your first point is very helpful. From a small sample of observations from a year ago that “needs ID”, roughly 2/3 have multiple identifiers but not a species level identification (perhaps not possible to tell from the photos as you mention) but 1/3 have one person giving a species level identification but no corroboration. We found Amanita by looking for groups with a large proportion of “needs ID”, but your point is very helpful that we should use a more refined criterion.
We completely agree that 99% confidence doesn’t translate to 99% accuracy (models are very often over-confident), but this doesn’t preclude calibration by examining the accuracy of their confident predictions on unseen examples (something like Platt scaling).
Thank you for the ideas around (1) classification to higher taxon ranks, and (2) reliance on background features. Unfortunately, I completely agree with your evaluation of computer scientists and the Dunning–Kruger effect. I’ve been working with a couple biologists at Georgia Tech for guidance and any additional guidance (e.g., from this post) is very helpful. It’s way too easy to create projects that “make sense” to the creators, but for some subtle reason, are actually pointless.
Thank you for your detailed response!
I agree that posting on this forum isn’t ideal. I’ve reached out to the iNat help email (4 weeks ago) and cold-messaged an iNat staff member (2 weeks ago), but unfortunately haven’t received any reply. If you know a person or email that would be a better fit, that would be very helpful!
Thank you for mentioning the policy. We’re aware of it which is why “we won’t upload anything to iNaturalist unless we get a green light from iNaturalist staff”.
Thank you for the ideas around the current machine-learning ID suggestion algorithm. Since we’re external to iNat, it’s difficult to focus on specific feature updates and thus we need to scope out a decently large project that can is a bit more independent from the existing systems. This is definitely something we’re thinking about since we don’t want to duplicate work.
I agree that confidence doesn’t imply correctness. Our plan would be to perform calibration with unseen data (something like Platt scaling) since many ML models are over-confident.
Thank you for the advice on team composition. We’ve been working with a couple biologists at Georgia Tech for guidance, though they aren’t taxonomists, which I agree would be better. If you have any thoughts on any taxa that you’re familiar with, I’d be interested to hear them.
Thank you for this idea! We hadn’t considered this but it’s useful information that several other responders have mentioned as well.
Thank you for your response!
To clarify, the difference would be that we would use calibration techniques (and re-run them to monitor performance degradation) with the ML system to ensure high accuracy. As other responses have pointed out, ML models are often over-confident so confidence by itself doesn’t imply accuracy. However, I agree that there is substantial overlap with the existing iNat system which reduces the usefulness of the proposed system.
You’re correct that I don’t have much activity on iNat. I didn’t create this project direction as an iNat user but rather through collaborations with biologists at Georgia Tech. From talking with them, citizen science data isn’t at the level of quality needed for use in their studies, but they saw an opportunity for data science and ML to potentially bridge the gap. It’s only later that I created an iNat account to poke around and learn more about the platform.
Thank you for the suggestions about the other data sources and higher-level taxa predictions, which are echoed by others on this post!
Thank you for your response! Your information about the limitations of IDing organisms from photos is very helpful.
Thank you for your response. I agree that I have recently joined iNaturalist and I haven’t contributed identifications since I am not sufficiently familiar with any taxa. For more context, this project idea is not from me as an iNat user, but rather as a collaborator with biologists at Georgia Tech who see an opportunity for ML and data science to be useful to biological studies by organizing/modelling citizen science data. I only created an iNat account later to learn more about the platform.
Thank you for your response!
This is very helpful advice that is echoed by others on this post. If you have any thoughts on particular invertebrate taxa to take a look at, that would be very appreciated! From your response and others, it seems that fungi is not the right taxon and we are looking exactly for taxa that meet the criteria you mention (otherwise, this project “solves” a problem that doesn’t exist).
Just checking a couple taxa I have a moderate amount of identification experience with in Ontario:
For both of these groups I have put some effort into identifying them in the past but haven’t had time for a few years, and I don’t think anyone else has really put in any effort either. Hopefully my notes give some context to the level of discernment needed - some can go to species but for the rest it’s a question of which level between order and genus an observation should go to. That’s why they haven’t gotten enough identifier attention - not only are they obscure taxa, but they’re also very tedious and frustrating to pick through to find the identifiable observations.
As others have mentioned there is that rule against machine-generated content so I’m guessing staff wouldn’t allow you to “let the machine loose” on identifying on iNat. If you could do that and let’s say restricted it to a genus of millipedes in Ontario where I’m the only person identifying them, I wouldn’t mind (depending on the accuracy of course) but it’s likely some observers would mind. If you expanded to other provinces or states it’s possible the machine would be working in another identifier’s “turf” and they might mind more if they have more distrust or principled opposition to machine content.
There are a lot of variables involved in identifying on iNat, hence endless discussions on this forum about how the current computer vision works and doesn’t work in collaborating with human identifiers.
You have said this several times above. Can you tell how many of those identifications have come from a mycologist and how many are observers using the iNaturalist computer identification?
FYI regarding forum etiquette: You do not need to reply to each person individually unless there are different ideas you are responding to that warrant separate posts. It makes discussions easier to read if there are not dozens of posts with more-or-less the same content.
It is not at all obvious from your initial post that this project is being developed in collaboration with biologists. Are these biologists familiar with iNat and what do they see as currently unmet possibilities for “organizing/modelling citizen science data” that a new machine learning algorithm could address? I doubt that “reducing the number of Needs ID observations (by relying on automated identifications)” is the top priority for people who want to use iNat data.
“citizen science data” in general or iNat data specifically?
Again, do these people have more than a passing familiarity with iNat and its CV and community validation system, or are they merely researchers who would like to use iNat data but have found that IDs are not reliable or the data does not include information that would be useful for their research? What are their research specialties beyond “biology”?
There are a number of reasons that citizen science data might be felt to be poor quality for certain research purposes, but exactly why and what the solution is will depend a great deal on exactly what questions one is investigating. Some limitations are inherent to the sort of opportunistic data that most iNat observations represent (e.g., it is not collected according to a specific protocol); the reliance on field photos (often taken with non-professional equipment) instead of specimens creates additional limitations for many taxa. User error and lack of training in how to use iNat effectively is another significant source of “poor quality” observations. These will not be solved by machine-learning identifications.
And the fact that there were multiple identifications but identifiers were reluctant to confirm species-level IDs did not lead you to investigate why this might be the case? This would seem to challenge the assumption that the cause of the large number of Needs ID observations is merely a lack of identification capacity, as you suggested in your first post.
Sorry, but it seems like your team needs to do a lot more preliminary research to better understand what the status quo is and to more clearly define what problem you are trying to tackle. It is not the responsibility of forum users to explain to you how iNat works. There are numerous scientific articles on using machine learning/computer vision for biodiversity recording, both in the context of iNat and other projects. Your posts give no evidence that you are familiar with existing work in this area.
And how do you plan to determine whether your model’s predictions are correct or not, or whether they are overly specific? You have chosen a data set that has a large amount of Needs ID observations – that is, it is one where there is a lack of IDers or is difficult to ID from photos. This means that even RG observations are likely to be less well-vetted than for taxa where there are lots of IDers and people are fairly confident with their IDs. In other words, given that one of the criticisms frequently directed at iNat is that observations are not reliably ID’d, how are you going to ensure that your data set includes only those observations that are correctly ID’d? It seems to me that you needs someone reviewing both the data set and your model’s predictions, particularly if your model will focus on a small subset of taxa rather than trying to be broadly applicable for organisms of all kinds.
Again, it is not our job to do your research for you or to decide what you should investigate. One gets the impression here that you have a solution in search of a problem, not a concrete problem (“poor quality of citizen science data” is not a clearly formulated problem) that you want to use machine learning to tackle.
There are lots of possible applications that would no doubt be interesting and avoid duplicating what iNat’s CV already does. For example, I imagine it could be fruitful to train a model to distinguish life stage or sex of certain taxa. Or there are lots of animal constructions that iNat’s model doesn’t learn because they generally can’t be ID’d as specifically as the animal itself but are often distinguishable to family or genus.
If this is the concern, better to explore the output from iNat to GBIF and ways to fine-tune that after export than try and work with the existing system here - for all the many reasons others mention.
The fact you want to insert ML content into the system without individual human control on each ID makes it a total non-starter on the iNat side, so there isn’t much point in discussing iNat interventions further unless you totally alter the starting premise.
Also, are you taking the biologist’s concerns at face value or do you have stats to back up the claim the citizen science data is weaker? There is a lot of prejudice towards citizen science data - but from the conversations I have had with external data users in the UK, the issues are for the most part misplaced imo.