I think the system should be changed so that users without a good history of making correct identifications on iNat do not have the ability to convert “Needs ID” obs to “Research Grade” obs until an objective criteria is met. I am not talking about a reputation system, but about some method that takes into account the particular person’s history on iNat of making correct v. incorrect IDs.
Probably requested because of the recent Penang Intersecondary School City Nature Challenge.
Earlier discussion in thread in the General category here.
Possibly related/similar feature request: Treat agreements differently than other identifications.
2 Likes
I will be honest, that while misidentifications are a problem on the site, I have strong misgivings about any solution that implies one user of the site is more valuable or important or trustworthy etc than another in terms of their input.
I’d be more comfortable with a solution that retained the current model of ‘one person, one vote’, but simply made it harder to get to research grade. For instance
- change the requirement to a minimum of 3 identifications versus the current 2 to prevent blind agreements promoting to RG.
- what is really a variation of the above, not count the submitting observers vote, so that it still takes 2 votes, but 2 parties other than the submitter.
4 Likes
That makes me really nervous without first recruiting more identifiers somehow. There are already so many things that haven’t even gotten one ID, the practical result of your proposals would be more eyes on things that already have 1 or 2 IDs but that attention taken away from things with no second ID yet at all. So i doubt the result would be better quality data? I think there are two root issues here: not enough identifiers, and too many ‘duress users’ or the apparently similar ‘contest users’ who dump mass amounts of stuff onto the site to win a contest but don’t bother to curate the data. I’d rather have more structural ways to deal with those. To be honest, i am starting to think classroom and contest version of Seek should be developed and used for a lot of this stuff, and iNat saved for people who want to be here to learn about nature and collect biodiversity data. The issue is not experience or training, it’s motivation and interest level. I don’t think these contests or class assignments generate enough good data to be worth adding to the main site if we can find an alternative. And we can always invite those intrigued and excited individuals to move from seek to here.
Maybe that’s kind of a controversial opinion here but… i personally believe if data quality gets bad enough, experts and power users leave or go elsewhere… and then the whole system crashes. This seems to have happened with other sites, such as Project Noah, and is probably one of the biggest pitfalls facing iNat. (imho)
6 Likes
I think there are 2 separate issues there Charlie. Yes, we need a larger pool of identifiers, but I don’t think in general that the person who could have identified that obscure sedge or fungi or whatever is the same person who could be the 2nd identifier of a Canada Goose.
If you changed to a 3 vote (or 2 vote excluding the submitter), most things that currently get to RG would still do so, they only currently get 1 additional ID because as soon as that is entered, they go away from the identify page. I personally think that most things would just be on the ID page for a few minutes more at worst.
The things that are the hard ID’s would still be the hard Id’s
5 Likes
My half-formed thought is that the binary yes/no Research Grade might be improved with more of a focus on an “ID Confidence Score” where each additional ID increases the score, and maybe you have a visual indicator that changes color as the number of IDs increases.
“Research Grade” under this system would be more under the control of the data end user. For some cases the user might want to see every observation with a given ID. In other cases one might only want to see those that have higher confidence (2 or 5 or 10 or whatever).
In the simplest case we could just keep the one person one vote model. If we wanted to get more complicated we could implement some sort of reputation system where more experienced users carry more weight. And if we want to go really crazy we could have a system where each observer scores an observation based on how confident they are in the ID: maybe a scale from 0 (best guess/AI suggestion) to 3 (absolutely sure).
8 Likes
i guess to distill my point ( i don’t want to belabor here)… requiring 3 IDs for research grade would more or less require 50% more IDs to be added to the site, in a situation where we already don’t have enough IDers. Granted some of them are the quick ones where you just hit ‘A’ but this would not be in any way an insignificant burden! It would be huge, i think.
I think a more neat and exciting approach albeit maybe not possible now… would be to change how the ID help algorithm works. Instead of just filtering for research grade, set it up so that maybe 10% of what you see is already research grade, mixed in there, to do a sort of audit on existing observations. Then maybe if you were really clever you could set it up to track where there were more ID disagreements, and with those taxa, maybe bump up the rate that they are offered to those doing ID. The main issue with tryign to have scores by number of ID is that at least for plants, once you get two IDs it’s rare to get a third because people who filter for needs ID don’t see it. It’s a big and a broad issue and one that is by no means easy, but i guess at this point i would personally vote ‘no’ on requiring 3 IDs. I’m not gonna fill with rage if they change the site so it does this, but i think there’s a real risk of diluting the ID help.
6 Likes
I don’t think it would be a huge increase in the ID’s needed or entered, certainly not 50%. The average observation her already gets just under 3.0 identifications (as I write this there are over 46,000,000 identifications that have been done on 16,500,000 observations).
To me the larger issue, and this too may be sematics, is not that we need a major increase in the number of identifiers (it would be nice, but is not the end all and be all), but rather we need to change the distribution of identifiers within the familiies. Specifically we have too few capable plant and fungi identifiers on the site.
4 Likes
That may be part of my concerns. I entirely do plant Id which skews my viewpoint some. I do notice any time I put in even a blurry bird picture I get several ids. So my comments and concerns are based on that. And maybe there’s some creative way to account for that.
4 Likes
To me, a low-bar entry that applies to everyone wouldn’t imply one user is more valuable than another–just as the trust-level system on this forum does not imply that.
3 Likes
I am unsure if I have improperly interpreted the suggestion, if so, my apologies, but to me any system that removes or devalues a vote based on length of time on the site, or number of identifications made, or any other criteria implies their input is less valued.
1 Like
I can see your point, but I’m not sure how much it will help - I’ve been finding quite a few cases where a blatantly wrong ID is backed up by 5 or 6 different users. Usually it looks like a bunch of people who went on the same outing all blindly agreeing with each other’s IDs. It seems counterproductive to have those observations have a higher certainty than an obscure species that might only ever get a single agreement.
7 Likes
i wonder if linked joint observations would help with that. So long as the IDs didn’t count towards RG if they were together. Which i guess is kinda weird too
3 Likes
This is probably the case in regions with lots of iNat users like the United States, New Zealand (?), and now South Africa, but my experience in South Korea is much different.
Observations of waterfowl and passerines tend to receive identifications quickly but when it comes to observations like arthropods things are quite different. Until @wongun joined the majority of my Hemiptera observations were at Needs ID and the same was true in other areas as well (Orthoptera/@taewoo, Zygoptera&Salticidae/@bio96, etc.) If my identifications/votes no longer counted on my own observations I would imagine more than 70% of my Research Grade arthropod observations would return to Needs ID. This isn’t a case of the observations leaving the pool after getting one ID and thus not being seen any more – those were observations that were in the pool for more than a year without getting IDs.
Another issue about votes and the majority of iNat users living in North America is that I occasionally end up with users trying to be helpful but adding incorrect identifications. For example, adding an ID of a North American species (no previous observations) instead of the similar-looking Asian species (previous observations). When asked about their reasons for the ID those users have often told me they didn’t bother looking at the range map and just went by appearance; however, in some cases the user never writes back and the ID stays there. I worry that removing the vote from the original submitter would mean that more of my observations get misidentified and it would take more work on my part to prevent that from happening. (e.g., I identify something as ‘European Beak’ but two other people say it’s ‘American Beak’. My ID counts for 0 so now it’s a Research Grade ‘American Beak’ observation until I go to DQA and check “Can community ID still be improved?”.)
I agree that there are problems with how easy it is for users to blindly agree with the IDs made by others but I do hope that any solution takes into account that there are areas in iNaturalist that don’t see the same attention from identifiers and would thus suffer from making the threshold too high.
8 Likes
I do like the idea of linked observations, and that is a whole other discussion.
Multiple people adding their uncertainty together to make an observation look certain would definitely be a problem; I suspect there a quite a few such issues that I have not considered.
I have been having a hard time formulating a reply because I am not clear in my own mind what I am trying to accomplish. Maybe it is just that I am trying to chip away at the current idea of what constitutes “Research Grade”. There is a very low bar to reach that status, and once achieved the impression is often that the work is done and further review is not needed, or at least has a low priority.
This impression is no doubt heavily influenced by my experience in my own main area of interest (plants) where there are a relatively few dedicated IDers trying to keep up with a flood of observations. (Thanks IDers!)
2 Likes
Observations have value whether they are RG or not. There are rubbish RG obs, and there are tremndously useful casual grade obs. We need to de-emphasise the “requirement” to get obs to RG. RG is simply a state an observation reaches as a consequence of CID, and if we educate people on the correct application of CID (which I believe means “without regard for RG status”) then the problem will fix itself. I myself have become much more relaxed about many of my observations still sitting at “Needs ID” or Casual. If they are going to get an ID from an expert, then that will happen when the expert is ready to do it! There have been situations where I have needed an ID with some urgency, usually because I have wanted to use the observation as part of a presentation… but 99% of my obs aren’t really that critical!
I still think the biggest impact for reducing the blind confirmations will be a probationary period where the IDs made by new users don’t count toward CID. If adding to this the idea of mentor accounts that can “release” IDs from their probationary state, then ensuring that users that are going to become mentors should have a proven history of “sticking around long enough to fix things”.
5 Likes
A lot of this discussion is just crying for an honest reputation system. After all if the world’s leading taxonomist in a group joins the site and makes IDs for some family of obscure bugs, surely his ID is worth far more than half a dozen school kids. In that case a second agreeing ID is entirely superfluous (or realistically would be impossible for 99.9% of iNat users). On the other hand, if some schoolkids are having fun, then 10 agreements are virtually worthless.
The question is simply one of how one creates and managers the reputation system. Like it or not, the iNaturalist reputation system at present is that everyone is a novice, and you can never progress from being a novice, no matter what your outside expertise, years of accumulated expertise working with and in that group, or your contribution to the identifications on the site. I dont see how any internal reputation system can detect or score an expert in an obscure group at all: it has to come from outside. For well-known groups with many identifiers and much agreements, an internal reputation system might work very well indeed.
And yes, any system can be gamed. Including one in which all users are perpetually novices and equal.
Personally, I like the concept of “Expert Equivalents”, where a score of 1.0 approximates the confidence one would achieve if one took a specimen to your local herbarium or museum and asked them to have the ID made by the nearest expert in the group. That would in my books be equivalent to “Research Grade”. (Note that the scale does not stop at one: if there are 5 experts and they all agree, that would be 5EE).
How one builds this into a reputation system for novices, non-experts and highly-competent amateurs is probably one of iNaturalist’s biggest challenges.
But statements like “There are rubbish RG obs, and there are tremendously useful casual grade obs.” are utterly meaningless. Who knows which are which and how does one tell? One might as well just randomly assign identifications with a half trained AI.
3 Likes
Tony, now that we have this new message board, there’s a more formalized way to make feature requests. I suggest you write up your desired reputation system as its own ‘feature requests’ topic so people can vote on it and we can keep the comments about it in one place. I know others want it too, so it would be good to have it written up.
And for that reason i will keep my varied opinions about Tony’s preferred reputation system out of this thread… it would sidetrack from the original thread, though admittedly it is at least somewhat relevant to it.
3 Likes
I created this Feature Request because other topics were being raised in the “Question” thread I started on this topic. I think it is really important for everyone to make sure that when they add a comment it relates to the title, because this forum gives us the opportunity to selectively track threads, and that just will not work right if the subjects of comments do not relate to the title. Maybe an “I have an alternative idea at [link to other thread]” would do the trick. Maybe I didn’t have the greatest title on this, though.
2 Likes
I agree. These boards are just starting out, and work differently, so there is an adjustment period and i don’t mean to jump on anyone in particular. However we should keep these requests on topic and use general discussion topics for other things. I need to remind myself to stay on track too (and indeed this post is also off track…)
2 Likes