Issue with users automatically agreeing to an identification

Yeah if it were removed from the observation page I’d do less IDs and I doubt it would solve the problem. I’m ok with it being removed from those thumbnails

2 Likes

Maybe a middle ground is to only show that Agree button if account has more than 500 IDs made (or maybe 50). That way power IDers wouldn’t be hindered, and it gives a chance for new users to get to grips with things first before they get access to the Agree button…

5 Likes

Is it new users who are agreeing too readily ?

I think so. As previously mentioned, the agree button is almost treated like a “Like” or a “thank you” for the id, and it also makes the obs RG which kinda reinforces the “goodness of agreeing, even if you don’t have a clue yourself”

1 Like

10% of IDs incorrect is 10%, regardless of how large the dataset is. Just using more data won’t help if that additional data has the same error rate. You will just end up with 10,000 wrong IDs in your data instead of 100.

1 Like

Maybe, though, crucially, misidentifications can be corrected. This is probably the only real solution. Also, your statistical evaluation may be misleading. If I have 10 data points, 9 of which agree plus 1 outlier, vs. 100000 points, 90000 of which agree, I’d say that 90000 agreements cannot be overturned by 10% outliers, but 9 agreements could be overturned by 10% outliers.

I think that the tendency for users to agree with identifications without independently verifying them (this topic and many others) and the tendency for users to choose the top computer vision suggestion without independently verifying it (this topic and many others) results in a feedback loop that produces many research grade observations of obscure, difficult (or impossible) to identify species that do not have identifiers specifically monitoring them for issues like this to pull the observation back to a higher level. I’ve come across this situation in multiple species, and because they are into the computer vision suggestions the problem never really goes away with these species.

The plans to have the computer vision work with taxa levels other than just species should help with the computer vision problem, but I think something else needs to be done in addition to help with the “automatic agreement” problem.

4 Likes

I think it is too easy to Agree with and ‘expert’ when you have little knowledge. After all, who would disagree with the expert? When I provide an id for someone who I can tell has no clue, I hate it when they agree immediately and send it to research grade (even when I am sure).

Suggestion. Have a Thank you button in addition to the Agree on the observation page. If you made the observation, disable the agree button, but allow an identification to be made.

11 Likes

I’m not sure population size is the best predictor of how likely something is to achieve research grade status, there is no evidence to suggest that smaller areas get less rates of successful ID’s. I’d think relative level of engagement within the population is going to better track this.

I took a look at the ‘success’ rate of getting verifiable records to research grade for 11 significant contributing nations to the site. New Zealand actually ranks high using this metric.

What I got (sorry I wont try to remember the html to build a table)

  • Russia - 82% of possible records have achieved Research Grade
  • Germany - 76%
  • Canada - 72%
  • New Zealand - 70%
  • United Kingdom - 64%
  • Mexico - 63%
  • US - 61%
  • Australia - 61%
  • South Africa - 59%
  • India - 49%
  • Brazil - 40%
2 Likes

The following post has now been double flagged as “off-topic”, which it clearly IS NOT! If anyone fails too see the relevance of it to the topic, then they can either ignore it or reply to it asking me to explain its relevance.

My argument is that if 3 “independent” identifications were required for RG, then those current statistics could change dramatically to the disadvantage of N.Z. and some other countries with low populations and/or poorly documented fauna/flora. Simply showing the stats as they are doesn’t suggest anything.

2 Likes

I don’t see any flag, maybe only forum admins can see flags…

When flagged, a post gets hidden, which is why you can’t see anything! I have simply copied my hidden post to a new post to make it visible again. If that gets flagged, then it will disappear also.

@stephenthorpeI think just adding a sentence or two couching in context of the topic would have fixed it, something like “I think the AI contributes to some of these feedback loops, so I wouldn’t object…”

OK, back on topic. I’m intrigued by the idea of a thank you button or some other way to acknowledge the help without saying you agree with it. But I’m worried about making the interface more cluttered. I would also like “Agree” to still be available on Identify, where it’s more likely to be used appropriately.

1 Like

EDIT: @tiwane This is getting very stupid. My reply (below) to your post has now been flagged as “off-topic”, even though it quotes your post. Please get this nonsense stopped!

Yes, but on the subject of context, you (@tiwane) have already sent me a message yesterday accusing me of having 13 flagged posts on this forum and indicating that there may be consequences if this persists. You lumped together “inappropriate” with “offensive” and indicated that most of my flags were for “inappropriate/offensive language”. Actually, most were for “off-topic” and none at all were for offensive language. Now you are saying that “just adding a sentence or two couching in context of the topic” would have prevented flag #14. Well sorry, but if people are going to nitpick my wording to that extent, and continue to flag my posts if they fail to measure up, then that is not a workable way to proceed.EDIT

I agree, this is a problem. I frequently see users agree immediately and without question to identifications, which is probably reinforced by their observations moving to research grade.

I would be ok with the agree button being removed from the thumbnails on the identify page. I almost never use this agree button, opting to take a closer look at the photos before making an ID.

Yes, “Agree” is too weak of a verb to convey what the user is actually doing by clicking the button. It has too many meanings and interpretations. I do not think this button should be removed completely, but I think other words or phrases could better convey what this button actually does.

I really like “Confirm ID” or something along these lines as a replacement for “Agree”. Great suggestion!

3 Likes

That’s going to be just as bad, because new users will think they are being asked to confirm it! Who wouldn’t want to be helpful to an identifier “asking you to confirm” their ID…

I still reckon nix ALL Agree buttons for new users, and maybe a pop up message when they qualify at 100 obs/IDs to explain clearly that the agree button they are getting access to is ONLY for when they do actually want to ID the same. Entering in an id is not a difficult step for a new user to have to do.

3 Likes

It may be a problem, but is it a big enough problem to warrant making the system more complicated? I think not. One key issue for me is that RG isn’t irreversible, so we can reverse it in cases which involve misidentification. If it doesn’t involve a misid., then worrying too much about the fact that it shouldn’t really be RG at this stage seems a bit pointless to me. The other key issue is whether a significant number (proportion?) of misidentified RG observations are being generated under the present system? My feeling is that the number is tolerably low (and any that are discovered can be reversed). No system is going to be perfect, but the current system seems to me to work well enough.

1 Like

It is not only new users who sometimes (more or less) uncritically agree with existing IDs. I posted an example somewhere, but no time to find it again now. There is a continuum of cases from 100% uncritical to 100% independent. In between, you can do a quick Google image search to check the plausibility of an ID, or you can trust the reliability of the previous identifier, etc. Ultimately, you have to trust something and make some assumptions. Given this continuum, I suggest that there is no reason to restrict agreement options unless it can be demonstrated that there is a real problem to be solved by such a move, rather than just a “niggle”.

2 Likes

What about preventing someone who enters an initial high-level ID for one of their own observations from taking any further part in the ID of that particular observation? I have no idea how technically feasible it would be to set this up, but I suggest they should not be able to use the Agree button and should not be able to change their ID by writing in the name provided by a subsequent identifier. Their initial high-level ID for that one obs is effectively ‘frozen’, but I see no reason why they couldn’t add comments, just not further IDs. That may sound a little draconian, but I suspect in most cases people who enter initial high-level IDs aren’t really qualified to agree at lower levels anyway.

Personally, I wouldn’t mind that restriction. As a test, I’ve been practising what I propose; I know little about plants, invertebrates, or fungi, so I commonly ID them at higher levels and then deliberately don’t take any further part in the ID process. It hasn’t stopped most of my obs being IDed to species level (most within hours or days, a few within weeks), even here in small-population NZ. If it’s possible to set this up, there seem to be several advantages. First, it should stop many of the ‘auto-agree’ IDs by observers who are probably not qualified, and so make RG a little more robust. Second, there’s no need to go to three IDs for all obs to achieve RG. And third, the Agree button would still be there for others to use, so identifiers aren’t hindered. And it also doesn’t discriminate against new users.

1 Like