While I think it’s possible this can have a benefit and work in some cases. I tentatively agree. This feels like trying to make an inferior guide with extra steps and less control over ones comments. Why should I try to tailor comments for AI summary when I could write identification information on purpose for a system to display it? Or even use this exact same system this demo has and just have the AI summary removed? A list of comments with ID information is nice to have and the highest rated one could be the first shown.
The only advantage this AI demo really has is the ability to attempt and get Identification information summarized for 100k taxa in a short time. In a sense quantity over quality.
Personally, I do not see much purpose in implementing the LLM. Sure, if it produces flawed and incorrect data, then people can correct it. Though, in that case, why not let the community write those summaries and cut out the middleman (the LLM)?
I understand that human-made summaries wouldn’t be perfect either. There will be disagreements too. And yet, editable wikis are still a prevalent host of information; we can do a wiki instead.
I agree that the best course of implementation is just the comment-collecting algorithm. It is a good concept and may help people to add tips to taxa that have no description, similar to the role an “Unknowns” identifier plays for observations. On the other hand, I am not interested in this LLM and do not think it will add much value.
It’s disappointing that iNat staff continues to pursue the use of generative AI to write explanations for species IDs. We don’t want it, and we didn’t ask for it.
As @JeremyHussell pointed out, the AI-generated content in the mockup itself is noticeably flawed. It restates the same thing twice (“the Monarch lacks the bar, and notably, it does not possess the bar”). If the AI-generated summary specifically chosen to be the mockup is this poor, I shudder to think what the typical one looked like. Then again, anyone who has ever used genAI knows that this kind of blind, uncomprehending sentence structure, stated with absolute authority, is very typical of AI-generated text.
We don’t want AI-generated explanations for species IDs on iNat. These should be written by humans. Plain and simple. You have a whole community of passionate experts who are willing to contribute to this.
As many others have already stated, user-written explanations, summaries, and guides will not only be undeniably more accurate, more understandable, and more user-friendly - it’s actually much more sensible, feasible, and economical than to attempt to use genAI for this purpose. To be at all useful, any kind of AI-generated species ID summary will need to be heavily edited by a person, which takes time. That person could’ve spent that time simply writing their own ID summary that would be better in every way. So why use genAI in the first place?
The answer’s pretty obvious - because Google gave a grant to iNat to build a genAI tool.
In addition, iNat cites a paper in their energy estimate spreadsheet that claims that the energy consumption used by Google’s GenAI is substantially lower than many public estimates. The authors of this paper… all work for Google.
I like the wiki idea. Maybe the best use of this language-processing grant would be to make a robust search engine for ID comments/information. Some sort of process that goes:
computer language processor identifies a comment that contains ID tips → sends a automated message to the author of the comment if they’re willing for their comment to show up in the wiki → sends an automated message to the observer if they’re willing for their obs to show up in the wiki → if both say yes, then the text + obs is saved for later and is searchable by other iNat users.
I think the existing filtering methods for observations are fantastic, and if done creatively (& ethically & in a way that makes sense…) we can have a search feature for comments too.
I was cautiously optimistic about this project, though I didn’t have very high hopes that it would be broadly applicable initially. Looking at the demo, I’m disappointed if this is the best it can do for an easy to ID species with a massive number of comments. The demo shows that the AI summary can only really handle one or maybe two ID features, and when there are multiple look-alike species, it only seems to acknowledge the existence of one of them. It also seems to struggle with context, including comments that are only useful in a small, unknown area.
I was hoping for more of a broad synopsis, maybe pointing out the difference in photos (like the example in the original post about this project, where there were two pictures of similar frogs, and a description of how a stripe on each one differed). It also seems to struggle with other life forms, implying that an oblong shape and “diagnostic” ridges is a good way to ID Monarch eggs, when that describes thousands of species of insect eggs. Considering that the CV incorrectly IDing a multitude of different caterpillars as a few distinct species is a major problem, this seems like a step backwards.
More importantly. What are these ID suggestions tied to? If they will only show up for taxa in the CV, wouldn’t that mean if a genus or higher rank above species has identification information compiled like this, but then a lower taxon or species is learned. The model will just forget and not compile ID information for the parent taxa?
If this only applies to leaf taxa in the CV. It’s hard for me not to say this is going down a very bad route. It also brings up the question, why put extra effort into certain comments if those comments can be unlearned and not recommended to people?
Some how the consistent pattern of unlearning parent taxa keeps coming up.
If, and only if that’s how this works. This may be the first “guide” that periodically unlearns perfectly good information by design for no other reason than it only accepts leaf taxa.
I imagine this will work great for things like common birds and flowers, but will be a disaster for the long tail of more obscure species (especially arthropods), where taxonomy is constantly changing, there is less understanding of species boundaries, and identification knowledge is still evolving. In those cases, none of the offered feedback options are suitable. What we need are feedback options for “This statement is overconfident” and “This statement is oversimplified”. Just saying if a statement is true or false isn’t adequate. Often there are known unknowns and LLMs generally suck at incorporating those as they are trained to provide answers, not express uncertainties. Unfortunately, most of our knowledge of the natural world is uncertain and I fear this tool will not properly reflect that, instead reinforcing existing feedback loops.
you just mean the Pattern ID summary?
the Coloration / Shape / Structure summaries seem better
I’m a bit confused about the votes for the Pattern one though, as it does seem weaker yet has resounding support from voters
But yes, as a person with zero familiarity w butterflies I’m also confused as to the exact meaning of text - but thats true of both AI and human comments. A diagram would be helpful in theory, as you say. I think this would be a natural shift by community to counteract this issue tho as stated:
I’ve lost count of how many times the anti-AI lobby has taken upon itself to speak for the entire iNaturalist community. You have expressed your opinion and that of a certain number of iNaturalist users. That is all.
As @nathantaylor states:
This is just a mockup of what the demo might look like, and how votes would be displayed on it. The numbers are arbitrary, just to show how it would look with one digit, two digits, three digits, etc.
yeah, why are iders expected to basically babysit the ai and to work around it? its just disrespectful when theres a better solution right there, which would be easier to implement than a whole damn llm.
You also missed the part where they gone on details on why they are dubious and sceptical of the project. But apparently everyone who are dubious of this project are seemingly part of a lobby….
Anyway on the relevant part, i personally can’t see how this tool is gonna be useful especially on obscure taxa like @zygy said. The suggestions for pictures id often becomes weird for obscures taxa or locations. Plus if we have to tailor and correct the things collectively , we should just start at wiki by this point.
I must say I really like the idea of an AI that can go and find specific comments that are helpful. There are heaps of incredibly useful ID comments spread all over iNat and the difficult part is finding where they are. I sometimes even struggle to find my own ID comments! So if we have an AI that can go and find all of these comments and present them together like in the monarch mock-up, I think that would be fantastic.
But I just don’t see the relevance or benefit of the LLM summary that accompanies them here. It doesn’t present the information in a more concise or understandable way than the comments, and more importantly for me, there’s no guarantee that it will be correct because an LLM cannot understand things, it’s just predicting the next word in the sentence. We already know that the CV is often very wrong, especially with more obscure taxa, and the CV has been slowly improving for years. So I cannot imagine that a completely new AI tool will be any more accurate than the CV. And while I like the idea of having an ID Summary Feedback section to mitigate this a bit, if I’m putting that much effort into correcting an AI-generated result then I may as well just write a new summary myself. Certainly it would be less frustrating, and I imagine it would take about the same amount of effort! So I guess I just do not see the use of having an LLM incorporated here at all. Having an AI that finds useful comments and perhaps highlights certain sections of them would be far more useful for identification purposes.
Thank you for the energy estimates; that is good transparency for the demo. What I am missing is some reflection on what the results say that relate to the overarching goal of a healthier planet and Environmental Responsibility. While the impact of the demo itself is fairly small, what would it look like if we scaled this up to the entire community’s observations?
I prefer to exchange with the person holding the knowledge. But great for the others that you gave in to all the requests for AI comment summaries, I think. I can luckily just not use it.
What level of ID skills are you aiming at? For your example of Monarch butterflies, I am aware that there are 2 or 3 that can be confused - but for my skill set I would stay broadly correct at Lepidoptera. And leave it for those taxon specialists to filter it out. Lepidopterists (whether professional or enthusiast) would understand the terminology - so it seems fair to use the appropriate technical terms.
Random example from today. Won’t link to the obs. It is, unambiguously, a Leucadendron. Observer’s ID Erica coccinea (why - it is not even in the background?). Second ID Leucadendron. My third ID also Leucadendron.
I could explain the difference. But this seems more like a mis-click than actual confusion. And I am wary of leaving a comment for a new to me observer, which might be misinterpreted as unkind.
PS good sleuthing @loarie ;~) and since you have found the obs - why do the IDs not show with their Family? Ericaceae versus Proteaceae - which is useful info missing from the display to my eyes. Those families also give their names to their orders.
I’m glad to see the referenced comments being directly quoted. I do think there’s a place for AI in helping to address the problem of disparate and disorganized ID tips, which is as an advanced search engine that can surface links to relevant comments directly in a list far more thoroughly and effectively than a traditional keyword search. That’s a useful feature I’d be happy to see integrated into the core site.
I do think the question of whether this works equally on branch taxa and leaf taxa is an important one to clear up.
The LLM summary itself on top of the search function is something I’m less enthused by. It’s nice we can correct it, but it feels a bit unnecessary when the comments are right there, and the fact that it can be edited is so conceptually close to being a wiki that it really raises the question of why we don’t just have a wiki. The fact that the comments are there at least means I’m less worried about its trustworthiness, because I can check its sources immediately without a single click, so that’s good.
As others have said, we really need a community wiki for ID tips (which could live on the taxon pages). That seems like the extremely obvious way to help people consolidate communal knowledge so others can learn to ID things. Wikis are a proven technology and structure to effectively aggregate knowledge on the internet; one of the most successful and best internet projects is Wikipedia! If it somehow helps iNat secure grant funding I wouldn’t necessarily be opposed to an ID wiki being initially populated by LLM content (suitably and explicitly flagged, with a request for human contributions!) until humans can get around to replacing it, but the goal should always be fully human-curated content. Fundamentally, the wiki structure is close to perfect for addressing this problem through real, human, community-sourced contributions.
I see there was discussion about this in 2019; was there ever a follow-up? If grants can get a demo like this made, can you maybe try to crowdfund development for a wiki? I’d chip in.
I’m looking forward to seeing how this tool develops.