It’s frustrating that the CV model is no longer making accurate suggestions for certain taxa since the 2025 City Nature Challenge. Here are the taxa I’ve noticed for which CV suggestions are no longer accurate: Erodium, Geranium, Malva, Sidalcea, Trifolium, and especially Rosa. I review observations for these taxa on a daily basis, so I notice when things change.
I know the fix for this situation is to find the erroneous IDs and correct them, but I’ve put in a ton of work “fixing” the data so the CV suggestions are more accurate for these taxa and now I feel that all that work was for naught.
I’m wondering if anyone else has seen this phenomenon?
Observations posted for the above taxa have unusually high misidentification rates, so it takes a lot of work to fix the incorrect IDs and get the data “cleaned up” enough so the CV model gives more accurate suggestions. Here are several analyses I’ve done that give details:
Interesting! I thought iNat ran their CV model monthly, so I wouldn’t expect the latest CNC observations to affect the CV so quickly. Or are you seeing changed suggestions because of the “expected nearby” data?
I would guess that this is due to “expected nearby” IDs. There is some discussion on other threads about issues with this during CNC. The issue seems to arise when
There are bulk bad faith observations (of copyrighted photos mostly) during CNC from other areas/continents.
There are then bad faith confirming/agreeing IDs from friends/sockpuppets that get observations to RG that should not be (which makes it harder to find/correct).
These IDs are then suggested for other nearby observations as expected nearby creating an antivirtuous cycle that leads to more erroneous IDs.
That then degrades data and is very labor intensive to fix
Optional: If these issues aren’t cleaned up manually by IDers/curators by the next time the CV model is retrained or geomodels created, they are enshrined more deeply in the fabric of iNat, making it even more difficult to fix them.
Edit/Addition:
More broadly:
Nat’s structure is not robust to large scale, bad-faith data entry.
Increasingly, CNC’s gamification seems to encourage bad faith data entry at large scales.
In this sense, these two elements are fundamentally mismatched/opposed.
The only long term solutions will be to:
Alter iNat so it is more robust to large scale, bad faith data entry (new curatorial tools that allow more efficient curation, more aggressive curatorial actions [deleting bad faith observations/accounts], etc.)
Change the structure of CNC (reduced gamification [copyrighted/casual observations don’t count, etc.], penalties for bad faith data, increased training for project leaders/participants, increased requirements for project leaders)
A way to suspend for evaluation any profiles making joke or fake IDs … till the CNC cutoff is done. Or till they resolve their bad behaviour.
We can already flag for Copyright.
As we have the new DQA for - not Single Subject - pushing the obs to Casual. (And Casual needs to be split to Broken vs Cultivated/Captive but not ‘broken’)
I would like new DQAs for these 2 ways to inflate (maliciously or innocently) obs numbers. And to reduce unnecessary work for identifiers.
this is an obvious Duplicate, DQA all the copies as Duplicates to Casual (till the observer deletes all but The One for RG)
these multiple obs are obviously many views of the same individual. DQA as Multiples to Casual (till the observer combines as The One for RG)
I agree with cthawley that this phenomenon is caused by the “expected nearby” feature of the CV model. It’s really frustrating the the CV model was once giving very accurate suggestions for the taxa mentioned, but now it is not, thus perpetuating the problem of misidentifications.
Would it be possible to remove all observations from the problematic CNC/2025 projects and place them in temporary quarantine holding tanks where they could be worked over by the project organizers to remove fake, fraudulent and dishonest observations? Then the cleaned-up sets could be returned to iNat. I realize this would be somewhat unfair to the people who made observations in good faith, but allowing spurious observations to contaminate the CV model and iNat as a whole is worse.
At least for the ‘winning’ Project which has been flagged.
The obs remain a problem. Since only 4% of those are RG, the other 96% could be quarantined.
I agree with your assessment. One problem I’ve encountered is mass agreement to Research Grade where I’m not certain whether or not the IDs are accurate, but I’m 100% certain they were based only on the Computer Vision and not observer or identifier expertise. Here is one example with a moss taxon entry after CNC Macao in particular. I guess the thing to do is to add disagreeing IDs on basically all of them, but I don’t want to hamper future identifiers on the off chance that the IDs were “correct by accident”. Is it still best to eliminate the Research Grade records if they seem to be driving a large-scale reconfiguration in the “Expected Nearby” suggestion order?
I want to clarify that I personally don’t think the ID errors I’m seeing are due to mischief or malevolence. I think they were made in good faith by ordinary users of iNat, some of whom are new to the platform.
One thing I’ve noticed is that the ‘Suggestions’ tab within the ‘Identify’ feature gives different suggestions than the ‘Suggest an Identification’ on the ‘Edit Observations’ page.
@tiwane, you asked for examples of where the CV model suggests an incorrect ID. I’m listing 10 examples for each taxa mentioned in my original post. If these aren’t enough examples, I can find lots more.
Are you able to provide a few links that can be clicked on? Trying to look at a URL on a screen and type it in correctly is much more difficult and prone to error than just clicking on a link. Just a few will suffice.
G. molle is the 6th ranked suggestion there, and the second Geranium listed as a suggestion. There are definitely Carduus leaves in there, plus splashes of pink, so I can see why that would be a top suggestion.
All the suggestions are “expected nearby”. I don’t know all of the taxa but most of them seem correct for the area.
No, it’s separate but it’s used to weight/filter CV results by default. More about gemodel here. FWIW here’s G. molle’s current geomodel map.
I understand. I wanted to attach my Excel spreadsheet so you could click on the URLs, but I couldn’t figure out how to do it. Here are a few examples. Please understand that I was using the “Identify” feature to review the observations and the “Suggestions” tab to see what the CV model suggested. Here are a few from each category. Please see the Excel “pictures” I uploaded above for the CV suggestion and the expected ID.
@tiwane, in my opinion, this top ID suggested by the CV model is totally unacceptable. I don’t think it’s reasonable for the model to suggest Italian Thistle for the most likely ID. The fact that Geranium molle is 6th on the list makes it very unlikely that anyone will select it from the list. It seems that something has definitely changed recently because the model used to be much more accurate in its suggestions.
But in the particular observation Tony selected, I am seeing a thistle dead centre in the photo – it’s out of focus, but it’s there, and even has two large flower buds (one in focus on the centre-right side). Carduus pycnocephalus sensu lato is not an unreasonable species suggestion for California either, and seems common in the region. While several thistle suggestions in a row is a bit much, and it’s odd that the Geranium doesn’t place higher given that that plant is what’s in focus, I don’t feel this is really that bad since the thistle is the centred plant. I feel this isn’t a great example for showing the CV breaking down per se…
