I must admit, this thread has been very enlightening for me. I especially appreciate lothlin’s statement:
”I think the problem, ultimately, comes down to our broader cultural relationship with fungi, especially here in North America - there simply aren’t that many people, in the grand scheme, that give a hoot about truly accurate fungi identifications.”
I’m going to try not being so judgmental about AI’s poorly IDed observations from now on.
Not caring what about a fungus ID is different from making a wrong ID. I don’t really care what the name of a fungus is, I just put them up as “Fungi”. I figure that in the future AI will improve, and ID stuff. Meanwhile, I’m not speaking beyond what I know.
iNaturalist Computer Vision over reaches,badly, on fungus. If the developers could tone down the suggestions, it would help. Having “Fungi, if you aren’t sure” as the first option on all the suggestions would help people know that the computer vision is weak on fungi.
I found that there is a shelf fungus that has problems. If you read the thread, there are people who took notice of this.
I was going to say that Stereum ostrea is just kind of a dump species so it is very difficult to clean up… but there are only 20 observations marked as that in thr US right now. Nowhere near as bad as I expected - someone must have put in effort to correct it
Perhaps you missed what I said:
Which suggests that multiple angles’ worth of photos, notes, etc., are not really what matters, despite the aforementioned journal post suggesting that they are. Lothlin summed it up:
Given that most of us aren’t going to be sequening our observations, most observations will never be considered identifiable by the mushroom cadre of identifiers.
Honestly at this point, I think you should just stop taking fungi observations.
Clearly no one is capable of living up to your standards.
More like: I’m not capable of meeting fungal identifiers’ standards.
We’re overworked, underfunded, extremely tired, and most of us can barely give good IDs on some of the more cryptic species even after spending hours digging through papers and old literature. Most of the most difficult groups are a huge struggle to ID even WITH microscopy - Russula in the eastern US are an absolute DUMPSTER FIRE that desperately needs experts to go through, systematically, section by section and figure it out. (and by this I mean, publish literature on it. Seriously, it’s a big issue.)
The complete lack of grace or understanding that you show to anyone associated with fungi that post on these forums is beyond frustrating and exhausting. Every single time there is a thread related to fungi, you turn it into a personal complaint session about how you aren’t getting satisfactory IDs.
We’re trying our best, seriously, I beg of you to try to understand that.
I want to thank those who put in the work to make sure at least some fungal observations are IDed correctly. It takes a strong will to keep at it.
I’m going to go out on a limb and speculate as to why staff are unlikely to make changes that meaningfully address this problem. I’m going to put it in starker terms than staff likely would, and perhaps starker terms than it merits.
None of this is meant to say that your complaint is incorrect, or that the problem you are pointing to shouldn’t be fixed. I do think that you are right that it is unlikely to be fixed.
The CV is bad because it is being fed garbage in a positive feedback meltdown. Poor photos, no photos of representative characters, and lack of microscopy are significant issues that result in inability of the relatively few experts to offer identifications beyond order/family/genus. And the relatively few experts don’t have time to counter the deluge of over-optimistic non-expert idents that fuel the positive feedback. There are however other issues that need highlighting when it comes to identifying fungi. Perhaps these factors aren’t sufficiently understood by many.
We know that fungi are the second largest kingdom next to the insects (ignoring bacteria). My guess/feeling is that there are currently about 10 professional entomologists for every mycologist, and many professional mycologists are employed to study economically important species and not the charismatic/easily seen species observed on iNat (although the same is true for insects). The result is about 1 million described insects versus 150,000 described fungi. Most fungi have not been described, even in areas well studied for other groups. It is highly likely that any set of iNat observations, from anywhere, will contain many undescribed species. They simply cannot be named to species, not even with sequence data.
From my barcoding work in New Zealand I have data on about 1,000 undescribed species of just mushrooms. Half of them represented by multiple collections and the rest are sequence singletons. Not many of these will get described in my lifetime. We also accumulated data on about 85% of the species that have been described.
The data show some interesting features. Multiple sequenced collections allow us to unambiguously assess morphological and ecological species boundaries and what characters provide reliable signals for identification. Often traditional concepts of infra-species variability in characters (or lack of variability) is not supported. But it isn’t all bad news. Whilst many species are superficially cryptic they can sometimes be reliably identified on subtleties not previously noticed. In addition, we have always known that ecological data is a significant factor - certainly host tree associations for mycorrhizal fungi are critical, and substrate/ecosystem for others, and generally a much more restricted ranges than previously accepted. Sequencing tells us these support data are critical. That kind of support data can sometimes be seen or inferred on iNat - but the CV has no chance of competing with human inference as it stands. If the CV was trained on a high quality subset of observation data, with decent representative photos, and the supporting data, then its quality would improve dramatically, and we might at least flatten the positive feedback curve.
Why is this made out to be a bad thing? If there was a hold put on fungi, or the fungi CV curated. Like species with very high inaccuracy rates removed. Would we still have fungi in the same state it is now?
If something isn’t identifiable from photos, why should anybody accept an image matching algorithim try and identify that thing using photos? I don’t understand how that makes sense. If it is indeed not identifiable by photo, say you need a chemical test, or smell. Letting an algorithm ID by photo seems like a recipe for misidentifications.
While large parts of fungi may be unfixable due to the scale of this issue. There are still other groups accumulating misidentifications that aren’t too late to get a handle on.
I have a feeling that currently the issue of overconfident CV suggestions has gained momentum in the forum community, as recently a wave of threads have been created circling around this topic.
Back in 2020, I made a feature request to tackle this problem, via implementing disagreements on CV suggestions in the learning algorithm:
I still feel this would be an elegant solution, as no curatorial steps would be needed to manually block certain taxa from the CV pool. It would just push the overconfident ID further down the list of suggestions and forcing the algorithm to suggest a higher level.
Another aspect why I am not in favor of forcing to stop the CV to suggest certain taxa is that in the future it might reach a level of acceptable quality again.
In that feature request, I imaginge that the algorithm, by constantly re-evaluating, and the addition of more species/photos to improve the training set, eventually the CV suggestions might reach a quality again, that it can distinguish similar species, and it automatically gets more ‘confident’ again
And lastly, it would immensely increase the value of those hard working identifiers, which currently might feel like a sisyphean task to provide disagreements to the ever increasing pool of mis-IDed observations
That could be made an obligatory observation field, tied to a mycorrhizal fungi ID.
Which tree ?
Yeah that’s a good feature request (you got my vote). Another one I’ve been thinking about suggesting is tying CV confidence scores to the proportion of observations that are RG. That way, relatively easy species would still get high-confidence species-level suggestions, but if only, say, 10% of observations of a given species in an area are RG, then the CV would assign a lower confidence score, and hopefully suggest a broader taxon.
Then again, I don’t really know the inner workings of the CV algorithm, so maybe this already happens? But it would also be a solution that wouldn’t require users/curators to manually decide which species shouldn’t be suggested at species level. Plus, focusing efforts on correcting erroneous or overconfident RG observations feels like a less insurmountable task than trying to ensure that certain taxa don’t meet the threshold for making it into the CV at all.
Except that the chance of the data being correct is probably low on average, particularly if it has to go to species. And what do you do if there are multiple possible host species (ie, it’s on the ground in the middle of a forest with several tree species present)? I can see the importance, I just don’t think that forcing people to provide/make up data they probably mostly don’t have is the answer. (Yes, I admit it - I’m terrible at noticing what tree species are around in my fungi observations. Even if I remembered, the answer is generally likely to be ‘Eucalyptus - but no idea of the species’.)
Having estimated confidence levels attached to individual IDs, as well as the consensus IDs, might help?
The current hard-edged, discrete consensus ID implementation results in situations like “100% it’s species A” vs. “100% it’s species B”
This is operationally convenient, but demonstrably suboptimal, and prone to propagating errors as described by others in this thread.
For many challenging and under-evidenced observations, and particularly in fungi, the reality is probably more like “70% chance it’s genus C, 30% chance it’s species A, 10% chance it’s species B”
Suppose individual and consensus IDs had a confidence assigned to them. These could begin at a modest starting value (0.2?), then increase with additional IDs, according to the strength of evidence that each additional ID is correct. CV training could then experiment with omitting (or down-weighting) low-confidence consensus IDs.
The evidence score–or Bayes-like prior–that each submitted ID contributes to the confidence in the consensus ID could be higher for experienced or professional identifiers with demonstrably low (within-taxon?) error rates–something that was apparently very controversial? last time I mentioned it.
https://forum.inaturalist.org/t/confidence-scores-for-ids/68018
With plants, a big complaint is that the CV does not include hybrids, leading to misidentifications when people observe hybrids. The staff are currently working on this issue (see: https://forum.inaturalist.org/t/make-computer-vision-include-hybrid-taxa-on-an-opt-in-basis/60160/42 ), but they are working using a “slow, methodical approach” which necessarily means it will take some time.
I’m not an expert at the intricacies of computer vision but to me it seemed that adding back plant hybrids would be a fairly straightforward change. The fixes for inaccurate fungi observations would likely be much more complex, and take even more time unfortunately.
My two cents: although I certainly understand the instinct to be cautious with CV changes, I wonder if the staff are actually being too cautious. Keeping with the status quo is a choice in and of itself, so the risks of rolling out a worse CV should be weighed against the existing problems caused by the current CV.
Well. But. If we want better Fungi IDs in future, observers need to learn to provide reasonable info. If the host tree is important.
Don’t look at me. My obs and IDs are carefully at Fungi. Or Lecanaromycetes. Maybe Ramalina vs Usnea.
I understand your point - I just think that the wrong information you’re likely to get if it’s required may easily be worse than no information.
Intended as a reminder (if you want a better ID) to add the info if you have it.
For example - galls - get a polite human request - do you know what the host is? https://www.inaturalist.org/observations/193968447
Follow by an impressive response from that taxon specialist! https://www.inaturalist.org/observations/273763645