Add a way to search for IDs which are both improving and explicit disagreements

Here’s another example where there is an active disagreement but the disagreeing ID is not labelled as such:
Screen Shot 2021-05-31 at 1.52.25 PM

Were the ID to become improving, I think this would fit the desired criteria but wouldn’t come up with how you’re looking for things.

1 Like

Surprising indeed. Do you know how that happens ? It would seem that the disagreement is not recalculated depending on the active IDs; or maybe because there were two distinct active IDs at that moment ?

I think something like both of those are the case, although I don’t really know how all that works on a technical level. Seems like it might be related to the issue I linked to above. Might not technically be bugs and need a feature request.

I found a similar example yesterday :

which then became like this (my ID withdrawn) :

and is like this today (now only the upper, relevant part of the ID sequence) :

I hope this helps.
Yet I am not sure these examples are a problem for the parameter. This indicates reduced sensitivity (some disagreeing & improving IDs will be missed), but they should not reduce specificity (all detected disagreeing & improving ID will be such !).

Maybe it is complex disagreements that are missed, but in a way, like in this last example, who could truly claim for “priority” ? I had corrected the suborder level down to the genus after all, and even warned about the species-level issue asking for advice, and then others corrected the species level.
I could say that the disagreement was not the work of a single person… so why not splitting disagreement vs improving between two identifiers, as it happened…

[And in fact three main roles are exemplified here (in chronological order) : 1) being faster or more dedicated, yet taking some controlled risks, and definitely not the best expert (me); 2) being a better expert (the “improver” as defined by the ID category); and 3) being a better expert and a kinder/more available person (giving the fully correct ID and an explanation in a comment). Note that the second role (in that simply chronological order) is the only one that is recognized by iNaturalist here ; it is alreday a good step forward, yet not completely satisfying… ].

Disagreements are calculated against the observation ID at the time the new ID is added. They are never calculated against a particular user’s ID, it just seems that way when there is only one existing ID. They are also never re-calculated. In the screenshots above, the system is performing as intended.

For the one with the Mya truncata ID, Genus Panomya is marked as a disagreement with M. truncata because M. truncata was the observation ID when Panomya was added. After Panomya was added, the observation ID became the common ancestor: Superorder Imparidentia. Panomya norvegica doesn’t disagree with Imparidentia, so it’s not marked as a disagreement.

For the one with Cordulia aenea, Calopteryx virgo disagrees with the observation ID (C. aenea) and is marked as such. After C. virgo is added, the observation ID is Odonata. Calopteryx splendens does not disagree with Odonata, so it’s not marked as a disagreement.

Related discussions:
https://forum.inaturalist.org/t/can-someone-explain-this-quirk-of-the-community-id-formula/9186
https://forum.inaturalist.org/t/change-wording-used-by-the-system-when-downgrading-an-observation-to-an-higher-level-taxa/3862/60

3 Likes

@jwidness Thank you for the extremely clear explanation, highly appreciated.

In other terms, the disagreement label identifies a “local maximum” (in the sequence of IDs), the locally highest taxon-level disagreement.

This will perfectly fit with the feature request proposed here, and also reinforces the requirement that relevant improving & disagreeing IDs be down at the finest taxon level (ideally species or below, but often coarser depending on taxa and regions, as mentioned above).

Indeed it would have been easy for me to choose Calopteryx or Calopterygidae, which would have “got me” an improving label, without reflecting any additional expertise.

This means to me that the “improving” category measures a mix of expertise and cautiousness, and increasingly more the latter at coarser taxon levels.

And is this intended ? How can it be explained ? I have placed my interpretation of the sequence of events on left the margin. The second “disagrees this is” seems unjustified to me. User A is the observer.

I think the most parsimonious explanation is that there was another P. pennipes ID that got deleted. Since there is no record of deleted IDs, you can never be 100% sure what the observation ID was when an ID was added.

2 Likes

Using the correct URLs as above, I think I have made some progress to see where this feature request can be implemented and how it can be used.

Of note, here I will use these URLs to create leaderboards (user names + their ID counts), using the “….identifications_identifiers.html” type of URL. Another possibility is to look for the individual identifications with the “…identifications.html” URL , all else being just the same (except that for the latter, it may be useful for instance to add the user_id parameter in order to restrict the results to a single user).

Aims and rationale : “ID accuracy and expert diversity”

First I would try to say something more about the background of this request. Asking for expert advice on taxon identifications can be quite inefficient when one enters a new field : no response for many days, or in fact for ever ; inconclusive comments ; surprisingly coarse IDs ; difficult suggestions like « here is a bibliographic reference, do it yourself » … One ends up asking a second, then a third expert, and quickly just feels embarassed. This happened to me the few times I tried and to people I tried to help outside of my area of expertise.

It would be nice if a simple informatics tool helped avoid these situations, by pointing to a small number of top candidate experts, so that one can ask 3 or 4 experts with complementary “qualities”, all at once, thus maybe increasing the chances of drawing anyone’s interest, which may be enough to help you, and may be the start of a useful discussion etc.

This is not my first attempt to address the “expert-detection” issue, so let’s see. Let me remind that as discussed in other topics, other parameters not dealt with here would be very helpful and relevant to this issue, but they are currently not implemented to any useful extent (such as comments-based searches or leaderboards). And obviously any leaderboard should be used with an open mind and cum grano salis.

Here is the proposal :

  1. run two URLs that produce two leaderboards, which we will call « disagreers » (counting disagreeing IDs that are confirmed by others) and « all IDs » (equivalent to the current ID leaderboard associated with the observations page),

  2. divide the « disagreers » by the « all IDs » counts, thereby obtaining a « disagreer ratios » leaderboard, and

  3. plot the « disagreer ratios » back against the « all IDs » counts.

The rationale is that it takes time to disagree and be right, and this will create a tradeoff between disagreeing and correcting difficult IDs vs increasing the number of « simpler » IDs. Hence some experts will be top disagreers, yet with relatively few IDs. Hence the % disagreeing ID is relevant.

Method

One of the two URL templates provides the numerator of the « disagreer ratio » leaderboard that is the main subject of this topic :

https://jumear.github.io/stirfry/iNatAPIv1_identifications_identifiers.html?taxon_id=…&category=improving&disagreement=true&place_id=…&per_page=100&hrank=species

The second URL template serves both as the « all IDs » leaderboard per se, and as the denominator of the « disagreer ratio » leaderboard :

https://jumear.github.io/stirfry/iNatAPIv1_identifications_identifiers.html?taxon_id=…&place_id=…&per_page=100&hrank=species&own_observation=false

These URL “templates” lack two pieces of information : taxon_id and place_id, which need to be added for specific searches.

Example . Magnoliopsida from S. Africa

This plot shows real results, but the user name letters have been scrambled. The red arrows point to some suggested expert first choices, which are chosen around the ellipses, trying to avoid the area below the Y-axis (disagreer ratio) median, as a very low disagreeing activity may have multiple interpretations.

The ellipses define the areas expected to contain 95% of the data (top 100 ranking users for each leaderboard, as provided by the URLs), either calculated on the distribution of all the data (blue) or after excluding likely outliers by standard statistics (red).

Conclusions

The “disagreer ratio” leaderboard does add new information that is both relevant to ID accuracy and independent from the standard ID leaderboard (no correlation between the X and Y axes).

When facing a difficult observation requiring expert naturalist advice well beyond my own area of expertise, I now tend to ask three or four top experts chosen from such plots (exemplified in the plot).

I would emphasize that the aim is to sample different areas of the plot outside of one or the other ellipse, as long as we believe that these areas and possibly visual clusters correspond to different types of expert.

Back to the feature request

The URL-based data retrieval and plot representation are simple, but they require quite some time manually.

The “disagreer ratio” leaderboard could be integrated as a checkbox in the identifiers leaderboard of the Explore/Observations page. Following the proposal from that post, the (optional) “disagreer ratio” leaderboard could be shown next to the standard ID leaderboard in the identifications tab of the observations page. Then one can copy the two columns and paste them into excel or some equivalent software to obtain the above type of plot, or for any other further analysis.

1 Like