DQA Discussion --- an observation cannot be “Research Grade” and “Needs ID” at the same time

[cs16-levi]

This post is intended to continue a conversation started in a separate, now-solved thread regarding the Data Quality Assessment (DQA) question: “Can the Community Taxon be improved?” and titled users are (understandably) saying they “disagree”.

This was solved by this comment; I recommend reading it before continuing so you know what has already been discussed.

Because this is a complex, long-standing system challenge with many moving parts, moving the discussion here allows us to freely brainstorm and examine potential improvements without cluttering specific observation inquiries.

During our initial discussion, a few distinct but overlapping challenges and ideas were brought up regarding how the site processes high-level Research Grade (RG) observations and subsequent identifications:

• Conflation of Observation and Community Taxon: There is an inherent structural challenge in how iNat displays and handles the relationship between individual observation taxa and the broader community taxon consensus.
• The Imbalance of “Resetting” DQA Votes: When a refining ID is added to a high-level RG observation, it can strip away the previous “As good as it can be” DQA votes. This effectively erases a specialist’s intentional assessment and drops the observation to Casual status, which is rarely the user’s intent.
• Decoupling RG Status from Search/Needs ID Filters: One idea discussed was whether decoupling the “Needs ID” search logic from RG status could help. For example, allowing an observation to retain its RG status (to protect its data quality tier) while still triggering a flag to appear in “Needs ID” queries when a new, refining ID is introduced. (This was the imperfect form of the idea, see linked comment earlier for details)
• Automating DQA Interactions: Alternatively, could adding a refining ID to a high-level RG observation automatically toggle a system flag (like an implicit “Yes, it can be improved” state) to alert the community without throwing away previous history or forcing the observation out of Research Grade improperly?

This is a draft framework for discussion, and I don’t claim to have an overarching solution. Given that tweaks to this system carry a burden over millions of observations, I’m eager to hear more perspectives from those who similarly deal with high-volume notifications or those whose specific taxonomic groups result in deadlocks most often.

Hoping to hear your thoughts!

---

[spiphany]

@cs16-levi

• When a refining ID is added to a high-level RG observation, it can strip away the previous “As good as it can be” DQA votes. This effectively erases a specialist’s intentional assessment and drops the observation to Casual status, which is rarely the user’s intent.

It does not currently erase the DQA votes in such a case. The DQA votes are erased if the community ID changes to a higher taxon. The observation becomes casual precisely because the DQA remains active. This is because, as the DQA is currently defined, it is meant to only apply to cases where the community taxon and the observation taxon are the same, so when observations violate this rule they are treated as defective. It is not possible to add DQA votes on observations where there is no community taxon or where the observation taxon and community taxon are different, but if there are existing DQA votes it is possible to add refining IDs that would create such a conflict.

[paul_dennehy]

@cs16-levi

• The Imbalance of “Resetting” DQA Votes: When a refining ID is added to a high-level RG observation, it can strip away the previous “As good as it can be” DQA votes. This effectively erases a specialist’s intentional assessment and drops the observation to Casual status, which is rarely the user’s intent.

I don’t think this us currently the case, and I hope no one is proposing it. No ID will “strip away the previous ‘As good as it can be’ DQA votes”, as far as I know. Observations become casual in these instances specifically because a user added a refining ID failed to counter the “as good as it can be” vote with one of their own. If it’s marked “as good as it can be”, and you disagree with that assessment, you have to vote on that DQA before adding your refining ID, or you send the observation to Casual.

@cs16-levi

• One idea discussed was whether decoupling the “Needs ID” search logic from RG status could help. For example, allowing an observation to retain its RG status (to protect its data quality tier) while still triggering a flag to appear in “Needs ID” queries when a new, refining ID is introduced.

The issue is that “Needs ID” is a data quality tier. You can’t protect the data quality tier and also make the observation Needs ID. I think that a major issue here is the terminology used for the data quality tiers. “Research Grade” doesn’t literally mean “ready to use in research”, nor does “Needs ID” mean “these are the only things that still need ID attention”, any more than “Casual” means the observations are wearing nice jeans and a graphic T shirt. Specialists routinely look through all three of the tiers to find things that need IDs. Researchers frequently look through all the tiers for datapoints that may be relevant to them.

I think in a lot of these forum discussions, too much emphasis is put on the importance of which data quality tier observations are in. The only major value of “RG” is it’s what gets exported to GBIF. And I wouldn’t want something getting exported when someone is claiming its ID can still be improved. So “allowing an observation to retain its RG status” when there’s still disagreement about whether the ID is finalized doesn’t seem desirable to me.

[cs16-levi]

@paul_dennehy

I wouldn’t want something getting exported when someone is claiming its ID can still be improved

Interesting point of view! Species that are later moved to the subtaxon level fits in interestingly.

@spiphany

if there are existing DQA votes it is possible to add refining IDs that would create such a conflict.

Yes, I didn’t mean it literally “erases” them.

---

Regarding the last comment in the other thread concerning arguing because the suggestions I made weren’t perfect, I’m glad you were speaking up about them! I didn’t mean for them to be anything set-in-stone you had to worry about, just briefly formulated drafts for discussion and the bringing forth of fruitful ideas. Seems like it worked.

[meteorquake]

I also noticed that same point - a researcher likely is only going to use RG to find candidates, but actually assess them directly themselves, as well as looking in non-RG, up the tree, and in any species that might look similar… it’s a greater issue to have RG plants that are not, getting onto automatic maps that less in-depth communicators then use.
Still it makes sense to resolve it to me, because the different facets shouldn’t be amalgamated, and it’s the sort of thing that’s usually somewhat easy to do, but not necessarily a priority.

[roadside_rambler]

I think it might be better if a “No” vote on that DQA would function as a sort of padlock-that-everyone-has-the-key-to. Meaning it would be impossible to add a refining ID without acknowledging the “No” and countervoting “Yes, can still be improved.”

Attempting to refine the ID could result in a popup message along the lines of:

“So-and-so believes this cannot be refined beyond genus (or species-complex, etc.) based on the evidence presented. Are you still confident a refining ID is justified?”

Yes (counters their “No” vote and adds ID, sending it back into needs ID)

No (does not allow ID to be added, leaving it RG at current level.)

I think most of the issues with these observations ending up in casual result from overconfident IDs by people who just see something they think they recognize as the one species they know, and overlook the fact that someone specifically said it was impossible to ID the observation to species.

I’ve personally marked species complex “as good as can be” and then had someone identify to species using only the CV. Both times they withdrew when I challenged it and explained, but I think it showed a lack of awareness that when a species complex is at RG, it’s because someone voted it as good as can be.

[yongestation]

@paul_dennehy

I don’t think this us currently the case, and I hope no one is proposing it. No ID will “strip away the previous ‘As good as it can be’ DQA votes”, as far as I know.

https://www.inaturalist.org/blog/120950-revisiting-based-on-the-evidence-can-the-community-taxon-be-improved

To be clear, I think this change is a necessary evil of sorts, as the meaning of a DQA yes/no vote changes entirely if the CID changes. If the CID for a bumblebee was at Genus bombus, and then someone adds a bee fly ID, then suddenly all the “no” votes would instead be saying “no, this can’t be improved past subclass Pterygota”, which was not the voters original intent at all. It also led to a lot of observations being rendered casual unexpectedly, because of the (newish) rule that observations can’t be RG if their CID doesn’t match their observation ID.

I still think by far the best solution is Pisum’s proposal to remove this part of the DQA entirely, and instead have “cannot be improved” tied to your identification (so rather than identifying to Empidonax and checking the DQA box, you could add a single “genus Empidonax + cannot be improved past that” ID). This way the meaning of “no it cannot be improved” doesn’t suddenly change against the identifiers original intent no matter what happens down the line, and it directly addresses most of the weird DQA problems. I was always surprised that proposal never gained much traction, I think partly because the DQA system is pretty esoteric so it’s hard to parse why such a system would be so much better than the current one.

[afdexter]

@paul_dennehy

Specialists routinely look through all three of the tiers to find things that need IDs.

It frustrates me when people argue that observations at Research Grade above species level are “hidden”. This is no more the case for an RG observation at genus level than for one at species level. It only takes ticking one box to include these observations in your search results. You can even search specifically for observations that are RG at a higher taxon level.

[spiphany]

@cs16-levi

Yes, I didn’t mean it literally “erases” them.

Your edit to your original post (“strip away” instead of “erase”) is not any more accurate. Adding a more specific ID that changes the observation taxon has NO effect on existing DQA votes; it does, however, change the status of the observation (from RG to casual).

The resetting of DQA votes only happens when the community taxon is changed to a higher level.

@cs16-levi

Regarding the last comment in the other thread concerning arguing because the suggestions I made weren’t perfect,

You seem to miss my point. As I said, my skepticism is not because the suggestions aren’t perfect, but because they seem to be based on an insufficient understanding of the DQA and how it works and how people use it in practice, and because the situation is not clearly described in any of your posts. This creates a risk that the conversation becomes incomprehensible at best and creates misunderstandings at worst. Once in place, such misunderstandings are often difficult to sort out. The DQA in question is highly complex, as are the ways that it responds to various actions. Unless the way it functions is clear to all discussants, it seems rather useless to brainstorm ideas that might not even be applicable.

Have you read any of the threads or blog posts that were linked?

[spiphany]

@paul_dennehy

Specialists routinely look through all three of the tiers to find things that need IDs. Researchers frequently look through all the tiers for datapoints that may be relevant to them.

I think in a lot of these forum discussions, too much emphasis is put on the importance of which data quality tier observations are in. The only major value of “RG” is it’s what gets exported to GBIF.

One other way that RG (particularly RG at levels above species) is useful is for managing IDer workflows. There is currently no way to search for observations that have only a single ID vs observations that have multiple IDs, meaning that for taxa that often can’t or shouldn’t be identified to species, it is difficult to determine which observations have been looked at by someone and which ones have not.

Using the DQA – if a community of IDers does so consistently – can help separate observations that are unlikely to get a better ID from those that still need to be reviewed, and thus which ones it is most productive to spend one’s time on if the number of observations is too great too look at all of them oneself.

I also find that using the DQA tends to have a discouraging effect – users seem to be somewhat less likely to suggest a finer ID for observations that are already RG, though this strategy is not foolproof (a certain portion of users will not notice the RG status or will add species IDs anyway). So for taxa where people have a tendency to suggest and confirm species IDs even when such an ID is not warranted, it somewhat reduces the amount of effort that IDers have to engage in.

@paul_dennehy

So “allowing an observation to retain its RG status” when there’s still disagreement about whether the ID is finalized doesn’t seem desirable to me.

There is a sense in which it is already the case that it is possible to make an observation RG while there not yet a consensus about how it should be ID’d – namely, if the most recent ID was a disagreement. If an observation has an ID of Genus species, I can add a disagreeing ID of Genus and make it RG by voting on the DQA. This does not mean that all persons who have provided IDs for the observation agree that it cannot be ID’d more specifically. It merely means that the common denominator of both IDs is Genus and the observation ID happens to be the same as this community ID. So not really much different in terms of the individual intentions than if the IDs had been “Genus” followed by “Genus species”, except that the person who ID’d as Genus was able to enter their ID as a disagreement.

It is also possible, for instance, to make an observation RG with two conflicting IDs – for example, an ID of species A (in subgenus cc) and a second ID of subgenus cd. I have seen it happen that the second IDer uses the DQA at this point, making the observation RG at genus. Not the intent of either the DQA or the IDer, of course, but it is the sort of mix-up that can easily happen.

@yongestation

I still think by far the best solution is Pisum’s proposal to remove this part of the DQA entirely, and instead have “cannot be improved” tied to your identification (so rather than identifying to Empidonax and checking the DQA box, you could add a single “genus Empidonax + cannot be improved past that” ID). This way the meaning of “no it cannot be improved” doesn’t suddenly change against the identifiers original intent no matter what happens down the line, and it directly addresses most of the weird DQA problems.

I agree; I think this proposal diagnoses one of the main flaws of the DQA, which is that it is applied to the community taxon even though it is really about questions of what level of ID is felt to be appropriate. I don’t know if it would solve all the problems, but it seems like a more solid starting point than the current system, which performs poorly precisely because of the underlying logical inconsistencies.

[roadside_rambler]

@yongestation

I still think by far the best solution is Pisum’s proposal to remove this part of the DQA entirely, and instead have “cannot be improved” tied to your identification (so rather than identifying to Empidonax and checking the DQA box, you could add a single “genus Empidonax + cannot be improved past that” ID).

Interesting. I like that a lot at least for the “No” vote, especially because it seems like it would allow an expert to make the vote even before the community taxon catches up. If you’re the first one to ID the observation to genus and are sure it cannot go beyond that, a “No, cannot be improved past the level of my ID” vote would mean that as soon as someone confirms genus, it moves to Research Grade.

I’m still trying to wrap my head around whether a “Yes” vote would still be needed if the no vote were attached to a particular ID, and if so, where it would go. It doesn’t make much sense to specifically vote it can be refined beyond your own ID, because if you can’t do so yourself, how would you be sure others can? If a yes vote is still needed, I guess it could pop up next to the ID the “No” vote was placed on, maybe? If I say “genus X, and cannot be improved” and the next identifier thinks it can so be improved, they could have a way of countering the vote next to my ID (similar to the way DQA votes currently cancel each other out while preserving the record) prior to adding their ID of “genus X species Y.”

[rcavasin]

@paul_dennehy

If it’s marked “as good as it can be”, and you disagree with that assessment, you have to vote on that DQA before adding your refining ID, or you send the observation to Casual.

I think this is a major problem. Many people looking through RG observations probably aren’t going to notice the DQA - I don’t think I would. It’s effectively a booby trap. In the past, I rarely used the “as good as it gets” DQA, and I don’t think many others were using it for the observations I’ve worked on. I’ve had many people recommend I use it to resolve various issues I’ve had with problem taxa, but I was hesitant to use it for assorted reasons. As part of a retrospective review of one particular problem group, I’ve been using it extensively. My intent was to avoid the creation of a lot of needs_ID observations that would probably be tempting targets for non-experts who are just trying to eliminate Needs_ID observations. If I’m understanding this correctly, what I’ve done is created hundreds of instances where someone attempting to refine an ID (even if they are agreeing with the ID I’ve applied) will just make the observation casual. I’m going to have to find a few examples and have a “buddy” refine the IDs to see what happens.

****update: the described behavior doesn’t seem to occur. As described in a subsequent post, I had a helper apply refining IDs to several DQA’d observations and none went to casual grade. No other undesirable effects were noted.

[roadside_rambler]

I think you’re using the DQA properly, and should not undo any of your work just because others might mistakenly send some to casual.

The fact that that’s so easy to do is something staff need to fix, but in the meantime, using the DQA is still better than leaving those observations sitting in the needs ID category, wasting the time of every knowledgeable identifier who looks at them, until someone less knowledgeable eventually gives an overconfident ID that sends them to RG.

Hopefully, when a fix finally comes, it will work retroactively and rescue any observations that were sent to casual by the ridiculous current system.

[rcavasin]

@roadside_rambler

wasting the time of every knowledgeable identifier who looks at them

Thanks, but that’s extremely unlikely, and part of the reason I’m reviewing these observations. Many are misidentified (even without factoring in recent taxon changes that we’ve decided to recognize), and have been that way for years now (I’m currently working on observations submitted in 2019, finished all previous years). I don’t think anybody with any expertise has looked at them - some were still needs_id. It’s the whole “fools rush in where angels fear to tread” scenario that I was worried about.

But it looks like I have nothing to worry about, because my experiment doesn’t match up with the expected behaviour previously described. Maybe I’m not understanding, but in each case, I had set the DQA to make an observation RG at a level higher than species. In each case, my helper went and applied an ID that is lower than the RG community ID. I was expecting them to all go “casual”, but none did.

I got a helper to apply specified IDs to a set of 3 observations that I have previously ID’d and then set the DQA to “as good as it gets”.

Here are the results:

Case 1
2 IDs for species A
my ID for species B
community ID at genus level, DQA set to make it RG

helper submits species C (which is in a complex with A)
ID goes to A/C complex in needs ID state. DQA is reset

Case 2
2 IDs for species A
my ID for species C (which is in a complex with A)
community ID at A/C complex level, DQA set to make it RG

helper submits species C (which is in a complex with A)
ID remains at A/C complex in RG state. DQA is unchanged

Case 3
2 IDs for species A
my ID at genus level
community ID at genus level, DQA set to make it RG

helper submits species A
ID goes to Species A in RG state. DQA is reset

Nothing “undesireable” happened in any of these cases.

Then we tried a few other things with Case 2. I’ve added the subsequent steps to the original scenario:

Case 2
2 IDs for species A
my ID for species C (which is in a complex with A)
community ID at A/C complex level, DQA set to make it RG

helper submits species C (which is in a complex with A)
ID remains at A/C complex in RG state. DQA is unchanged

helper submits species B - same genus - no change on community ID or DQA (their ID is marked “maverick”)

I withdraw my ID. Community ID goes to Genus level in needs ID state. DQA is reset (as per expected behaviour described earlier in the thread)

[rcavasin]

@spiphany

The resetting of DQA votes only happens when the community taxon is changed to a higher level.

That’s not the behaviour I observed a few minutes ago. See my post above.

[meteorquake]

My vote would be to ditch the whole flag entirely and just let people add the agreeing/refining/demoting IDs they want and tag experienced people when needed. (A refining ID is anyway equivalent to the flag but the system has an awareness of where it happens.) If someone feels it can be refined but isn’t confident to do it themself they could tag someone.

In doing so, the system should still calculate RG & needs-RG in an appropriate way.

I would though appreciate a nuclear vote to non-RG something (that can be countermanded). Where expertise is only in 1 or 2 people and non-experts are a-plenty, a lot of things go to RG state that shouldn’t be.

[roadside_rambler]

@meteorquake

My vote would be to ditch the whole flag entirely and just let people add the agreeing/refining/demoting IDs they want and tag experienced people when needed.

The “No, it’s as good as it can be” vote would still be needed. Otherwise something that experts know can’t be refined to species would sit in “needs ID” forever. A hundred experienced people could agree it’s genus X or species-complex Y, and the system would still think it was waiting for that hundred-and-first to take a look at it.

[GeolTel]

Too few taxon specialists already overwhelmed + subpar notifications system + refusal to educate users in order to deter proxy-/blind-agreeing + many ephemeral or irresponsive observers = one can’t just rely on identifiers/observers/specialists’ strained goodwill to (1) prevent bad RG data from spilling elsewhere, and (2) have bad RG data re-IDed correctly.

In my experience, a careful use of technical means at hand (namely the DQA checkbox) is able to solve both issues peacefully and quietly, in most cases. :)

[meteorquake]

You’ll have to elaborate perhaps with a hypothetical example to make it clear, as I can’t see where your conclusion comes from. If there are only submitted IDs, no flag for can be refined, nor its converse as good as can be, then any “needs ID” is handled by submitting an id, combined with an appropriate calculation for needs id (distinct from RG).

[meteorquake]

You’ll need to explain how the checkbox helps with bad RG data, since it seems apparent that knocking an observation out of RG state so it doesn’t go to external systems isn’t what the checkbox is meant to do (saying an RG species can be refined to sublevel shouldn’t knock it out of RG as far as data broadcast is concerned).