I’ve read through this entire thread. I’m not a programmer. I’m just a regular user of iNat and I think you hit the nail on the head right here…for those who are entering observations (that don’t really understand the GeoModel, hexagons, how computers work in the background, etc.) we are relying on iNat suggestions to get us at least into the correct Family or Genus in hopes of getting an ID.
If someone is traveling and does not know the local plants, animals, etc., then they are going to rely on iNat to help them get it going in the right direction. I am seeing enough failures for plants here on the West Coast of the U.S. that I posted a question about it in the forum last month because I thought the “system was broken” due to an upgrade or something. That became my introduction into hexagons and I honestly have been left feeling disappointed about how it works (and at times doesn’t work) overall.
If this is what is happening every day with thousands (or more?) of observations…it is letting down the users who are being told to trust the iNat suggestions (even if just implicitly by “trusting the app”) to provide a basic ID because it has a “suggestion” feature that is often times accurate.
I suggest this IS an issue if iNat is meant for the everyone and not just “scientists.” Why would an average user think to choose an option to see suggestions “not expected nearby” and when we choose that, how do we know that what is in the list is actually possible in that location or not? I suggest that iNat is perhaps making assumptions about what average users will know, understand or expect…and also how closely many users will compare their obs to the top suggestion, especially when working on a small phone screen.
I would never have thought to choose that option had I not asked my question last month (I’ve been using iNat for several years and only just learned that because I came to the forum to ask a question…many users will never come to the forum and many will never read detailed instructions for using the app, so explaining it there is perhaps not the solution…making it work better is, I think, the solution).
Finally, I believe this is a big issue for those making IDs. iNat is trying to get more people involved in making IDs and it has been clear, I think, that one reason people do not help with IDs is they are afraid of making mistakes…if the system is way off, then that is not helping those dipping their toe in the ID pool. Further, for those power IDers, I believe having to correct a lot of IDs adds to our workload and could, potentially, add to the factors that create burnout.
I’m not sure if you’re being serious here, but yes it is sometimes possible to know a change will improve a system without running a series of tests.
Let’s look at the tradeoff that would happen here. Right now the geomodel sometimes does not expect a species in a cell it occurs in. This means basically every time someone photographs that species, it will suggest other organisms instead. So the cost is all those observations, which as you can tell from this and other threads is quite large. Now the tradeoff for implementing this patch is exactly those cases where it causes the CV to suggest the wrong thing AND the user selects that wrong thing AND that user would have selected the right thing were that suggestion not there. Do you seriously, speaking in good faith, believe there is any chance the latter effect outweighs the former? Especially considering the fact that the geomodel already makes baffling extrapolations of the ranges of organisms some thousands of kilometers away from there actual ranges, a probably massively more substantial extension than adding in those few cells where a species is known to occur but not “expected”??
To emphasize how small of an effect this would probably have: the CV currently expects the Eurasian Eagle-Owl, Bubo bubo, in large swaths of the most heavily iNatted parts of the US ( https://www.inaturalist.org/geo_model/20059/explain ). I regularly look through the bird observations in a couple extremely regularly iNatted counties included in that area. I have never once seen an observation with that ID. The only cases I’ve seen of problematic geomodel suggestions have come from it recommending things dozens of miles away, not a couple miles away. Using a hex based geomodel already introduces a good deal of coarseness, and I find it hard to believe you really believe it would be better for the geomodel to exclude cells where an organism occurs than include them.
EDIT: Put even more simply: a cell being included in a species geomodel is not supposed to mean that species occurs in all or most of the cell area spanned by that cell, otherwise species with very specific microhabitats might not be expected anywhere. A cell being included in the geomodel is just supposed to represent any presence at all. And about the best evidence you can get for the presence of an organism in a cell is a bunch of observations of that organism there.
I feel that you’re making this more complex than it needs to be @einsum. You’re correct that the geomodel also guards against excessive suggestions of visually similar matches with taxa from an entirely different continent. But that’s not the issue here.
As I understand it, the output of the geomodel training process is a net or map of hexagons with binary values indicating the species is “expected” or “not expected”. Nothing in that structure is incompatible with running a post-build check to see whether there are RG observations for any empty hexagons and if so adding those to the final geomodel map.
I believe you’re saying “We need better statistics to determine the pros and cons of that tweak”. (I’m paraphrasing, so I apologize if you feel I’m oversimplifying.) But really, we don’t need better statistics to make this fix. The presence of RG observations in a hexagon indicates that, within the level of certainty that iNat has defined as “Research Grade”, the species is actually present in the hexagon, and so the geomodel generation process should always add cells where any RG observations are present. If incorrectly IDed RG observations cause the model to mistakenly include hexagons where there are no reliable observations, the iNat community can easily fix that by adding new IDs to those observations.
No typical iNat user, beginner or experienced, expects the platform to fail to suggest a species known to occur nearby for the reason that the CV algorithm prioritizes an abstract elevation model over the actual occurrence data present within iNat.
I continue to be bothered by the geomodel’s problems as well. The largest municipal park in my city (setting aside one regional park) and one that exists at sea level, happens to sit at the northern edge of a hexagon that otherwise encompasses the mountains north of the city. For this reason, the city’s largest urban park is prone to frequent suggestion-caused errors with very basic taxa, like eastern grey squirrels which exist in the cell by the thousands, because the geomodel excludes things that are not expected in the mountains north of the saltwater inlet that surrounds the park.
As others have noted, from a user perspective (both identifiers and observers), including known-to-occur species within a cell would be a clear and definitive improvement. The purpose of the geomodel isn’t to model biodiversity accurately. It’s to support the user experience of observing and identifying organisms, in this case by making the correct species suggestion as available as possible. As someone who has designed user-facing software for many many years, experience has shown me time and again that elegant, correct, precise, realistic models can often lead to worse user experiences and worse concrete outcomes for a project than systems that may mix logics and involve multiple steps but remain focused on the actual purpose of the tool. That is a different kind of correctness, and one well worth keeping in mind.
Totally 100% with you on this here. The balance is definitely missing. but also on when and where exactly within the same cell they should be injected.
not the “exact same suggestions” for every observation, as the other side is missing in this reasoning.
suppose the true answer is “red bird”, and the current geomodel already says “red bird at top-1”, but by injecting another “presence suggestion - visually similar - and seen in that cell” now as “yellow bird at top-1” for that red bird, we are gonna enter degrading performance for those red bird like predictions (in however faulty notions the model already learned it as red bird before). if we say lets not inject them at top-1 ever and only inject these new ones at top-2 position, the same argument can be extended - what if the true answer is at top-2 earlier.
and then we are so far considering example of single injections, but what if there are more than 3 or 4 that end up under that notion of “100% relevant recall inject” and where should their positions be? and how would these all effect the true answer if it could have already present without inject? …
I totally understand that you mentioned earlier that injecting such suggestions for cells can primarily take away certainity of only suggesting wrong ones, and provide net benefit; but all I am saying is such good injection strategy onto the one side of such observations that benefit from that uncertainity will definitively effect some other observations top-1 and MAP and F1 now (where the same uncertainty may not be even needed as its already learned some other valid geoprior on those) and without metrics we cant quantify whether such injection violating one side faulty geomodel predictions is gonna be superior to whole model or atleast some class of taxon or not.
Ok, this is exactly the one-to-one reasoning issue I am trying to raise everywhere here in this thread. I blame myself mostly for not able to convey what I mean in text, and I hope someone else can convey my ideas better via illustrations or relevant material.
Any classification system should be looked on lens of both “top-1 precision & recall on all taxa observations” (and if we have to take metrics MAP and F-score are also better choices). Just to ascertain these metrics are not arbitrary or not just my demands - the inaturalist research SINR paper itself mentions F1 score - see this line under figure 7: “We convert the predictions to binary values using the threshold that maximizes the F1 score on the S&T data.” and see how the paper uses MAP scores in table 1. and also see how all these three metrics I am talking of precision, recall and f1 are actually monitored by iNat during training and eval in its code too. so just want to highlight that model suggestions are not just optimised for one metric oriented as it may have been implied in their model updates blog page (of top1 accuracy curve) and all these metrics combined are highly relevant in selecting a final model a classification system is designed for and what users see when deployed.
Under that notion, any objective superior patch should also rely on those same metrics. because a seemingly good patch, that looks maximizing recall by patching presence cells into suggestions, as you are mentioning above will have knock on effects on both MAP and F-score as whole on other taxa suggestions - it is fundamental result - precision and recall live onoppositeends of the manifold in our problem setup - Figure 1
notice here, the other side that is missing in your line of reasoning is “a user can also select wrong thing where such new suggestion now added in your way above is actually wrong in first place” :)
aka any argument of recall is now fixed 100% by naive patch argument wont carry onto “objective improvements”, as we havent quantified how these above alternate effects are.
No . the geomodel is designed to learn with elevation (read “Coordinates and environmental features are complementary” section in SINR paper and iirc they removed other bioclimatic variables in current deployed model but still uses elevation as it simplified development things but only at marginal performance loss; also elevation combined with higher resolution regularizes coastal areas better and also provides inductive spatial bias to model of where the coastal areas are and himalayas are … ) in first place instead of learning with just coordinates and presence alone for everything and then there is also well known step in SDM where lower end of already presence observation areas are deliberately thresholded out to improve the robustness as those presences can be noisy data and on other side being acting as control to reduce False positives - think of this question “why is some lower percentile of prediction scores where the cell is already known to be present with that taxa are deliberately thresholded out while setting a static taxon threshold?” and this is the code line.
so the geomodel is actually already learning some taxa habitat well from elevation and coords (note elevation is DSM like data from SRTM radar and not true terrain elevation data like FABDEM and such noisy DSM data also encodes information of habitat or such as the radar signal propagation inturn relies on canopy cover and such) although not perfectly nor exhaustively for all taxa. and the metrics are important to say how much that some are and not are as this thread. and how balancing for latter effects former.
so, I am personally certain that geomodel problem persists unless and until the elevation information is somehow dealiased within that single cell for predictions and correlated with their learnt taxon elevation profile (one can easily prove this can lead to objective improvement via mathematics) as this is the primary reason the entire issue is happening in first place. or even if one wants they can increase the resolution as blunt attack although it wont come with objective improvements as I mentioned there is regularization effect from aggregrating sparse data to coarser resolutions and learning on them than going to finest level and asking model to fill too many gaps. again see Table A1 and how hex 3 decreased over hex 2 in SINR paper.
what is even “quite large” without the reference of how many cases a no-patch is already working fine under different notions. note those working fine observations are not reported just as equally in the very same threads we are reading - this is what I meant by red-bird and yellow bird example earlier in this thread. such red-bird (where the true answer is red-bird, but the cell presence injection answers made the top-1 as yellow bird) issue threads popup afterwards as something like “why is iNat cv(&geo) suddenly regressing now?” :)
and if those injection happened anything beyond top-1 or so, you are left again with same assumption that observers who made mistakes earlier are suddenly gonna pick those non top-1 but some other top-k suggestion which is injected.
so such quantification is highly relevant, as I bet if we ask for all these metrics on model versions 2.21 (grid) vs 2.22/2.23 (sinr) these very same metrics would have already statistically shown some issues had popped up in 2.22/2.23 models without even deploying and waiting for such new community issues and then having to regress the 2.24 back to 2.21 design after weeks.
see the page that iNat reports in good faith for each model version update completely lacks the very metric that we are all trying to talk of here “recall” (and recall at class levels) that can answer “how large systemically is this issue of missing true answers in cv&geo list by current models” (and also MAP and Fscores to compare any real future patch tradeoffs on these quantified metrics)
I dont think anyone is told anywhere to trust iNat suggestions for picking it directly for ID. But sadly everyone can see how the reality ends up on platform :)
(I bet large set of inat users arent even aware of h3 centroid elevation issues as this thread is about, and the lowest hanging fruit will actually be atleast highlighting those problematic cells for now during suggestions, as they are already detectable with real presence vs a prediction miss of that taxa group and true elevations of observation, in some way like “geomodel suggestions maybe unreliable in inhomogeneous elevation cell as this one”)
also it would be far better relatively if those users are encouraged to anchor on selecting common ancestor suggestions over committing to top-1 always especially when they are unaware of those top-k suggestions taxa and any lookalikes that may not be in those top-k (note unlearned taxa is on equal issue as recall missed taxa)
But we just cant uniformly recommend a mantra to those top-1 biased users to settle for picking common ancestor now always over top-1 (As there will always be cases where the same common ancestor breaks too on unlearned lookalikes or ends up as overly conservative suggestion and nobody can say what each user has to decide except the universal mantra “always only pick suggestions or make IDs u can verify and support”)
They are incompatible in another levels when viewed on suggestions lens :)
note the suggestions list are primarily scored and sorted list. aka the geomodel score “always” acts as boost or loss to the suggestions position of any suggestion within our cv&geo list.
i cant find where the code is now, but just note this equation from SINR paper. notice how the final score is boosted on CV by h(f(x)) geomodel value on those list.
and under your scenario, yes we can grab “RG observations for any empty hexagons” but what is their real h(f(x)) ? note if you say lets use their unthresholded score, it is not correct representation as the model already assigned low threshold to that cell in first place; if you say lets assign any other “fixed value”, but just let them get injected into suggestions list, it can have degrading effect on top-1 precision or MAP/F1 as whole as it is gonna break that score-sorted list positions of correct non-injected suggestions of any other observations where such inject wasnt happening earlier.
note the SDM line I mentioned in my last reply. a perfectly valid presences are equally deliberately thresholded in conventional statistical methods too but under different notions than “elevation model mistake” here
and if the model made learning mistake (centroid elevation dependency) it is important to correct that exact mistake rather than introduce another mistake (presence is to be injected) to correct it which will have different effects that cannot be argued without metrics as I highlighted in my last reply too.
its not just present that is not what the suggestions list are. the species is present in that hexagon at some elevation profile. so even if u think the injected new suggestions doesnt need to care of elevation, as they are going into a space of that scores in suggestions list, we cannot get away without firstly correcting for elevation mistake.
i think everyone here is aware of said tradeoff, its just that we believe its clearly a tradeoff that’s worth it, with very little lost in proportion to what is gained.
nothing is missing there. if they were going to select some other incorrect taxon anyways then no harm is done. if they were going to select the correct taxon then that is covered exactly by my case.
all i’m saying is we can see how negative the effect of a single cell missing from a species gemodel is, meanwhile most geomodels include dozens of cells in the geomodel where said species does not actually occur to little negative effect, so i think its clear patching in the rare occasional cell where a species is occurs but is omitted from the geomodel will have positive outcomes that massively outweigh the negatives.
I don’t think position really matters, especially if no confidence score is shown. Yes, people that don’t want to put in any effort to figure out the difference between similar species and aren’t conservative enough to choose the genus or family if offered will likely choose whatever is in the number 1 position, but that is a failing of the people making that choice. If the CV model has a confidence of 90% for red bird and a 8% confidence for yellow bird, that could be shown and help users, assuming those confidence values are somewhat accurate. Same applies if yellow bird was 30%, red bird was 29%, and green bird was 28%. Whether that confidence is shown or not, it can vary from one extreme to another. If people are to choose from a list and the taxon known to be present there is not on the list, their choice will be wrong 100% of the time. If the choice is there and it is at number 2 or 3, many people may still choose number one but many people will also have a solid learning opportunity to figure out what the difference is between the top 2 or 3 that look very similar. That is a huge win. People need to learn that some taxa are very similar and confusing and that they should not trust that the number 1 suggestion is correct. Even if the correct suggestion is at number 3, when the correct species is included, the performance overall has increased infinitely from when the correct species was not on the list at all. Not included = failure. Included = win.
no. only the recall improved. the net precision improvement is uncertain without real metrics. and so what you are calling is performance is only recall oriented while the true performance of classification system is F-score or precision-recall tradeoff.
see this extreme example to illustrate again:
say only one red bird is correct, the system only predicted red bird (relying on elevation) and a suggestion list ended up only as “red”. this is the ideal system state and nothing else is gonna beat on any metrics for this suggestion list.
now you inject equally valid examples of (visually similar - cell present) yellow, green, … anything from cell presence without considering their injected order in the final list and just say users are supposed to check them all. so no harm.
and say the new suggestions ended up as “yellow, red, green, white, …”
even though it looks harmless under the realm of “yes let users check every suggestions, the precision and performance (f-score) of this new suggestion list is absolutely lower than step 1”; and for any model its a real tradeoff (also see red can be below its true position or not if order is ignored in this new list injections).
and if these kind of tradeoffs happen on average more than your recall tradeoff on such iNat new model as whole, the net performance can be statistically said as bad.
and so notice your injection could actually end up be the reason user picked incorrect taxon now for this scenario if they are relying on top-1 earlier and could have happily picked correct red before.
firstly why did they have to always pick one item from that list alone? what if the true answer is not learned by vision itself? this is equal argument to true taxon is not in suggestions no matter whether the model learned such taxon firstly or not.
aka if users are enforcing trust to always pick one suggestion from model list alone without checking beyond such list, they are already making a fundamental mistake of assuming all taxa are learned by iNat in first place and hoping they can now correctly pick true one only among those suggestions.
But we already have tons of data on what happens if you add cells to a species it doesn’t occur in (let alone ones it does occur in!), pick a few species in the geomodel at random and theres a good chance one of them will include dozens of extra cells. This would barely move the needle in the slightest
Just an illustration of what currently happens: I’ve literally had conversations with people who incorrectly identified eastern grey squirrels as douglas squirrels even though they knew it was an eastern grey, because, quoting as closely as memory allows, “I thought it was a grey squirrel, but the app didn’t suggest that, so I guess those don’t live here? So I guess it’s probably a douglas squirrel.”
Convincing someone who knows what an organism is to misidentify it is a truly glaring failure of design, in my estimation.
The latest CV update post is here and these always show estimated accuracy of CV only vs. CV +the geomodel. And, of course, CV + the geomodel is always higher than just CV. It would be interesting to see a three-way comparison of CV vs CV + the current geomodel vs. CV + a geomodel that includes all taxa present in a hexagon to be considered as an option. Sampling might have to be much much higher than 1000 to detect much but it would be great to see some real numbers.
A better approach than how they sampled for that might be to actually do queries that summarize what percent of observations for each species occur in hexagons marked as “expected nearby” vs those that aren’t. The difference might not be significant for many species but be incredibly significant for others. It would be good to see what that range is and what percent of species it actually affects. It would also be good to look at a finer scale. If it works great in one region but terrible in another, we still have a major failure even if it is good 99% of the time overall.
For those that think we need real metrics, we should get them. We have many examples showing that there is a huge problem but we don’t have any metrics showing the scale of the problem. What percentage of taxa is it affecting and how badly? Again, if it works great for 99% of taxa, it still may be a major failure to 1% and 1% is a lot of taxa.
I would assume that CV + no geomodel is equivelant to CV + geomodel that includes all taxa?
The whole point of geomodel is to not include things from the wrong continent, so including all taxa and introducing every possibility is the same as not having that limit?
What was said was not ‘including all taxa’ but ‘including all taxa present in a hexagon’ - in other words, adding any species that have observations even if the geomodel doesn’t think the species is present.
. the geomodel is designed to learn with elevation (read “Coordinates and environmental features are complementary” section in SINR paper and iirc they removed other bioclimatic variables in current deployed model but still uses elevation as it simplified development things but only at marginal performance loss; also elevation combined with higher resolution regularizes coastal areas better and also provides inductive spatial bias to model of where the coastal areas are and himalayas are … ) in first place instead of learning with just coordinates and presence alone for everything and then there is also well known step in 