Community taxon algorithm tweaks

[sbushes]

At present, in situations like this one, the community taxon algorithm balances an initial incorrect ID against the finest expert level ID and disregards coarser input from the community which supports the expert ID.

e.g.
See the breakdown here:

So at genus level we have 1/(1+1+0=2) = 0.5
Thats …1 x expert ID / 1 x expert ID + 1 x expert ID + 1 x incorrect original ID

This completely disregards the weight of the other 3 x identifiers supporting the fact it is a sawfly, not even a true fly. If the initial incorrect ID still forms part of the equation at genus level sawfly, shouldn’t the 3 x coarse sawfly IDs supporting it be taken into account here somehow as well?

Why aren’t coarse IDs taken into account in this equation?

If they were, what might that algorithm look like?

What would be the downsides of implementing change connected to this?

At present to have the community taxon go to genus would require two other sawfly identifiers - that would be 6 x IDs in total to overcome the initial incorrect ID at order and place it correctly to genus. In comparison with an observation which was initially placed at Hymenoptera, where 1 x ID would suffice to place in correct genus. To my mind, the 3 coarse IDs should essentially overrule the initial ID at level of order. At the very least, they should enter into the final equation.

For me personally, this aspect of the existing algorithm is a source of frustration because, as someone who is active at coarser levels, a good portion of my identification time is spent overcoming this imbalance - either adding weight to others or tagging people in to try and help resolve an observation which has been trapped at the level of order/suborder (often for years). This might seem like a minor annoyance, but it adds up - I’d guess 30-50% of my IDs at order/suborder relate to this issue. For me, that’s equivalent to time spent on about 15000-25000 IDs… time which to my mind, could be far better utilised on other IDs.

[cthawley]

It’s a good point that the one incorrect ID is having an “outsized” influence on this observation.

I’m not sure what an algorithm for Community ID that tried to improve this would look like, but the devil is likely in the details (as usual).

I suppose you could make an algorithm whereby if an ID was disagreed with at a greater than 2/1 ratio at any level, it wouldn’t “count” as a disagreement at any lower (more specific) taxonomic levels. That >2/1 ratio is what is used elsewhere on iNat so that would be clear enough at least.

So in this observation, that algorithm would show that at the level of Typical Sawflies superfamily, there are 4/5 IDs for, leading to a score of 0.8. Therefore, the disagreeing ID (Diptera) would not count as a disagreement for any levels below Tenthredinoidea.

I have no idea what other implications this might have for IDing (unintended negative consequences) - probably some. But maybe a place to start thinking about this?

You’d need to also come up with a rule for how to apply this philosophy in situations where there are multiple disagreeing IDs (either the same as each other or different), though those situations are likely less common.

The example you’ve put here is probably one of the most common disagreement scenarios, and I do think that addressing it better would improve ID speed/efficiency overall.

[pisum]

@sbushes

Why aren’t coarse IDs taken into account in this equation?

probably simplicity.

@sbushes

If they were, what might that algorithm look like?

to do what you’re suggesting, i think you’d have to basically split disagreements and agreements into ancestor and same taxon components. in this way, you could keep track of ancestor disagreement vs agreement separately. any net* ancestor agreements would not carry to the next level down, but any net* ancestor disagreements would (and would be added to disagreements at that level). (*in this context, “net” would mean >2/3. maybe a better term would be cumulative?)

@sbushes

What would be the downsides of implementing change connected to this?

complexity. (good luck to anyone trying to understand a new algorithm that would handle this case.) also, it would need to be reapplied to every existing observation. so you might all of a sudden get observations pushed to research grade suddenly. (i’m not sure if any would get pushed out.)

[sbushes]

@pisum

good luck to anyone trying to understand a new algorithm that would handle this case

you mean to observers and identifiers trying to resolve an observation by looking at the equation?
I’d say by far the vast majority of users don’t even know the “whats this” section exists, let alone details such as the one I’m raising here.

…or do you just mean good luck for anyone attempting to form an algorithm which would resolve this issue?

@pisum

you might all of a sudden get observations pushed to research grade suddenly

In what use-case would this be a bad thing?
In the situation above at least, if there were two species level IDs and the three coarse sawfly IDs cancelled out the incorrect true fly ID, it would indeed take it to RG. But to me that would just seem like the correct outcome in this sort of scenario. It would basically give the correct weighting to expert IDs, rather than balancing them against naive IDs/errors which have not been withdrawn.

[alloyant]

Yeah, I also like doing coarse IDs and have noticed this problem as well. I think the easiest solution would be to simply disregard IDs marked as maverick as a disagreement entirely. Of course there are cases where the maverick ID is in fact the correct one, but probably 19 times out of 20 it’s simply an incorrect coarse ID from a casual user.

[sbushes]

Hm, yes - this is similar to how I imagined it (and how @cthawley described it too).

It’s helpful to think of it in terms of mavericks as you say - this is already a recognised state which is triggered in the system, it’s like it just needs to be factored into the equation somehow.
Could one not just make the “Disagreement Count” a “Non-maverick Disagreement Count” instead?

Thinking of the flipside, if I see 3 x people have incorrectly identified a fly as Lucilia sericata and I add the ID Neomyia cornicina, my ID will anyway be maverick and powerless until a 2nd ID comes along which supports my POV, or one of the other identifiers withdraws.

[alloyant]

Right, it would become a drawback in cases where there have been several incorrect IDs, but I just feel like that happens less often. I don’t really know how to fix both at once. All I can think is perhaps giving certain reliable users’ IDs additional weight or permissions to kick the community ID up to a higher taxon, but that just seems sort of against the spirit of iNat and also sounds quite impractical.

[sbushes]

@alloyant

Right, it would become a drawback in cases where there have been several incorrect IDs,

Would it? In the flipside I posted, I’m not sure it would make any difference tbh?
… I feel like I am most probably missing something though!
It is definitely a little confusing to try and map the different implications …

[pisum]

@sbushes

you mean to observers and identifiers trying to resolve an observation by looking at the equation?

yes. i bet, as it is, most folks who look at it probably barely understand it. but add the extra layer of complexity that i was talking about, and even fewer people would understand it.

@sbushes

In what use-case would this be a bad thing?

i was mostly just noting that as a minor consequence. (it might confuse some people why this happened.) i think the bigger (technical) issue is that you’d have to recalculate every observation probably. (or maybe you could just assume that research grade wouldn’t need to be recalculated? i’m not sure.)

[sbushes]

gotcha

am I right in understanding there was a previous algorithm to this one which worked differently?
I wonder what the impacts were following that change (if there was one)

[pisum]

@sbushes

there was a previous algorithm to this one which worked differently?

i haven’t been around long enough to remember seeing another algorithm. i assume that their existing definition of “ancestor disagreements” refers to the “branch disagreement” functionality, and i believe that was added later – and if so, that would have represented a change. that kind of thing would be more of a tack on of entirely new functionality though, and would not have required going back and recalculating everything.

[alloyant]

Well, right now for instance if there are 2 incorrect IDs and someone puts a correct ID, it bumps the obs back up to the lowest common taxon, right? If changed as I suggested, the 3rd ID would be disregarded and the incorrect ID would remain. Again I am not sure this exact scenario happens too often though as it shouldn’t make a difference except in multiples of threes…?

[fffffffff]

Yes, but also from what I saw on old observations every higher taxon id was a disagreement at first, so totally an improvement. The current one appeared in 2019, but I don’t know if there was something in between.

[sbushes]

@alloyant

right now for instance if there are 2 incorrect IDs and someone puts a correct ID, it bumps the obs back up to the lowest common taxon, right? If changed as I suggested, the 3rd ID would be disregarded and the incorrect ID would remain.

2 incorrect IDs and 1 correct ID wouldn’t create a maverick ID.
I think! (unless my memory is failing me )
so your description (“to simply disregard IDs marked as maverick as a disagreement entirely”) wouldn’t be impacted in this scenario.

It would take 3 incorrect IDs and 1 correct ID to create a maverick.
In which instance the correct ID already has no power - until someone else seconds it.

However, in the exact inverse of the original post we would see some loss in power.
e.g. at present if we have 3 x incorrect superfamily IDs and 1 x incorrect species level IDs and someone adds a correct ID which is from a different superfamily or order then it bumps it back to superfamily. If we disregard mavericks as you describe, then it wouldn’t bump the ID back to superfamily as it does now - until it was seconded by someone else.

I think if this is the only trade off to this aspect of the algorithm though, then the optimal solution lies heavily in favour of change. This use-case is relatively rare and low impact in comparison with what is happening at the moment, which is less logical, far more impacting / a regular occurence.

[sbushes]

@fffffffff

on old observations every higher taxon id was a disagreement at first

I’m not sure I understand what this looked like. I tried searching forum posts for discussion of change in algorithm but couldn’t find anything. Wouldn’t this change have involved a recalculation of all observations site-wide as @pisum describes?

[sbushes]

@pisum

yes. i bet, as it is, most folks who look at it probably barely understand it. but add the extra layer of complexity that i was talking about, and even fewer people would understand it.

Thinking further on this, I just don’t think optimisation of the algorithm should ever be subservient to general user-understanding.

I think most users expect there to be an inherent logic to the algorithm and act in line with this expectation more than the actual detail of the algorithm provided in the “whats this” section. Most users simply wouldn’t expect a fly ID to hold sway in the instance originally posted, as it’s illogical - so (arguably), fixing this would simply conform to existing user expectation and be less confusing for most.

@pisum

i think the bigger (technical) issue is that you’d have to recalculate every observation probably. (or maybe you could just assume that research grade wouldn’t need to be recalculated? i’m not sure.)

It would only require recalculating for observations with an existing maverick ID wouldn’t it?
(which would be about a million observations, but most would require no actual community taxon change)

I went through the first page of most recent mavericks to take a look at potential impact.
For 47 of the 50, disregarding mavericks completely would have zero impact to the community taxon - just a digit shift in some of the “whats this” disagreement count numbers.

For two of the three it would have the intended impact I originally posted about :
https://www.inaturalist.org/observations/103520727
https://www.inaturalist.org/observations/103485365

For one it may have a less positive consequence - similar to that discussed above - where it would no longer bump it back to a higher taxon :
https://www.inaturalist.org/observations/21365663
In this instance it actually returns the observation to RG.

These are recent mavericks though. I expect it to be different for older obs.
I would expect the number of positive impacts to far outweigh the negatives across a larger/older dataset. I tried to check the oldest obs to see how they compare, but couldn’t get the URL to work.

In terms of numbers though, if it actually shifts community taxon in 3 of every 50 out of a million, that would be about 60000 observations in total with a visible impact.

[fluffyinca]

@sbushes

if there were two species level IDs and the three coarse sawfly IDs cancelled out the incorrect true fly ID, it would indeed take it to RG. But to me that would just seem like the correct outcome in this sort of scenario.

I agree with this. I think at the very least, something should be changed so that if there is one fine ID, several agreeing coarse IDs, and one disagreeing coarse ID, as soon as the Community Taxon becomes the agreeing coarse ID it should automatically be bumped down to the finer ID. E.g 1 species or genus spider ID, 2 “Spider” IDs and 1 “Insect” ID should put the Community Taxon at the level of the species or genus ID rather than “Spiders”. However, the possible downside is more confusion around disagreements. It would not be possible with the current system to disagree with the species while adding a coarse ID, because the Community Taxon would be at Arthropoda. But as soon as enough spider IDs were added, the Community Taxon would be the species ID and disagreeing with it would require adding a new ID. We already have problems with this sort of thing, so I’m not sure if adding more is a good idea unless the explicit disagreement system can also be reworked.

[fffffffff]

No, this particular change wasn’t retroactive. It looked like you ided it as crow, I ided it as birds and it was a disagreement with no other options.

[pisum]

below, i tried to modify your original example a bit to make it more complex and then lay it out like the algorithm summary does. in terms of modifications, i added a lot more IDs at various levels, including some new disagreements (beetles, narrow-waisted wasps), as well as a branch disagreement (at typical sawflies):

i think the adjusted figures in columns G and I are the hardest parts to do technically, and there are probably a few different ways ways to implement the calculation of G in particular. (the numbers i used above are based on a more complex potential implementation.)

(i’m not trying to demonstrate any particular algorithm here. i’m just showing how many more components i think you would need to keep track of in this kind of calculation.)

@sbushes

It would only require recalculating for observations with an existing maverick ID wouldn’t it?

relying on the existing maverick calculation is one way to go about getting the adjusted value in column G, but you’d still need a way to get the adjusted value in column I, i think. so i don’t think you can just recalculate needs ID observations with a maverick ID, although maybe you could recalculate based on a set of needs ID with any disagreement.

@sbushes

Thinking further on this, I just don’t think optimisation of the algorithm should ever be subservient to general user-understanding.

i think “optimization” can mean many different things, depending on your priorities. i tend to think the existing algorithm is fairly well-optimized for speed (technically) and for user understanding (few parts to keep track of).

@sbushes

Most users simply wouldn’t expect a fly ID to hold sway in the instance originally posted, as it’s illogical

i think when you’re dealing with algorithms like this, i think “logical”, like “optimization”, again just depends on priorities. because your example is relatively complex, honestly, i don’t think all users would have a set expectation for what the end result should be. i bet many would just follow the existing logic of the current algorithm to decide what the output should be.

…

anyway, as i noted before, i think, given your priorities, there is some merit to thinking about an alternative algorithm like you’re suggesting. that said, from my perspective, it seems to me like your use case is valid but relatively niche. so i personally don’t see a clear case that the existing algorithm needs to be changed.

[cthawley]

@sbushes I think that there’s been some really interesting commentary from the community around your idea. A lot (though not all) of commenters think that there might be some improvement to be made in the algorithm for the cases you identified, though there are open questions about whether the additional complexity would be worth it in terms of effort/time as well as potential confusion. I think feedback from staff might be worthwhile here (they may be able to say that there’s no way to make this happen or have some creative ideas) - so you might consider making your initial post into a formal feature request and link to this thread?