Apply community consensus logic to annotations in the same way as for IDs

The current approach for annotations (Adult, Alive, Flowering, etc.) is that any user can apply an annotation and other users can then vote that up or down, but subsequent annotation choices are limited by what other users have already chosen. I’m proposing a system that more closely matches the way we reach consensus on IDs. I believe it would improve the accuracy of annotations on iNat and be easier for users to understand.

Let me outline the issue: Let’s say I come across another user’s observation that is actually of a juvenile male bird, maybe a Ruby-throated Hummingbird. Maybe, being over-confident about my knowledge of the species, I’m sure it’s an adult female and I choose the “Adult” and “Female” annotations. Any other iNat user who then views that observation cannot contribute their opinion that it’s really a juvenile male. All they can do is down-vote (or up-vote) the “Adult” and “Female” annotations. That’s because for most annotations selecting one option causes the others to disappear.

This appears to be happening because reasonable limitations on one person’s actions are mistakenly being applied across all iNat users. It’s good that iNat won’t allow me to say that a single bird is both “Adult” and “Juvenile”, but it’s not good that my identifying it as an adult prevents your identifying it as a juvenile.

I accept that this is an infrequent issue, but given the vast numbers of iNat observations it’s one that does have some impacts:

  • Some observations will be “stuck” with incorrect annotations unless the user who originally applied them can be persuaded to remove them. Contrast this with IDs, where the community can overrule an initial bad ID.

  • If the user who applied the incorrect annotation leaves iNat, there is no way to fix the incorrect annotation. Again, that doesn’t happen with incorrect IDs, which can be outvoted by correct IDs even if the first identifier is AWOL.

  • The value of iNat’s data for research use is diminished either because annotations are known to include unfixable annotation errors, or because observations need to be excluded where there’s any dispute over annotation accuracy.

I propose that iNat applies to annotations a slightly simplified version of the logic that is used for IDs. In the description below I used the term “consensus” to describe an annotation value that has sufficient support for in iNat to return it as a search result or supply it as part of an RG data feed.

  1. iNat modifies its schema to store annotation values tagged with the ID of the user applying them.
  2. In the iNat UI, conditional logic (e.g. “hide life stage choices if a life stage is already selected”) is applied on a user-specific basis.
  3. If there are no disputes, then a single “vote” is sufficient for an annotation value to be regarded as the consensus for an RG entry (i.e. one vote for “Adult” is sufficient for iNat to say the organism is an adult)
  4. If any value for a particular annotation is disputed, then >2/3 support is required for a value to have consensus support. (I say “Adult” and “Female”, you say “Juvenile” and “Male”. Neither annotation has consensus support. When two more users vote for “Female” that gains consensus support.)
  5. For incompatible annotations (“Adult” and “Juvenile”, “Flowering” and “No Evidence of Flowering”) there’s an edge case where there could be > 2/3 support for more than one option. The result should be no consensus for any option.
  6. Where multiple annotation values may be true simultaneously the above restriction should not apply (e.g. a plant can be both “Flowering” and “Fruiting” so long as there’s >2/3 support for those annotation values and <2/3 support for “No Evidence of Flowering”.)

Other logic may work better than my proposal above. The core point is to apply a consensus process to annotations that’s similar to what we have for IDs.

This proposal came out of a discussion here about the new “No Evidence of Flowering” annotation value.

I’m up to make wong annotations be removable by voting, though with observations with multiple specimens different people can have different view on life stage and sex, and we already have some troubles with it, and it can end with voting against the specimen the author chose. Also observers can delete wrong annotations done by others and it will be against this system, I guess? While now it’s easier to deal with others’ wrong annotations.

2 Likes

Hi @melodi_96. I agree that observations where the photos contain multiple individuals pose a problem for ID and for all annotations.

I think I don’t understand your last two sentences:

Can you give some examples?

When one observations havemultiple specimens and you know which one is there, but don’t add annotation right when you upload or with current situation half of annotations don’t safe sometimes someone else comes, looks at the ob, and choose the wrong annottion because it’s a different person with different preferanes. Or you upload one series of pictures, one with female and one with male, and on one you add annotations and something happens with annotations on the 2nd one, someone else comes and chooses not what you decided, thn you have 2 males, for example. Not the most common situation, but happend with me with obs that had both larvae and adults.

2 Likes

Thanks. I agree that those issues exist with an observations that includes multiple individuals. It doesn’t seem to me that the current annotation logic deals with them better than the consensus approach I proposed. In both cases the observer can add a comment to say “This observation is for the larvae” and other users can add comments such as “Your photos include both females and males”. Supporting multiple individuals within one observation would be a subject for a different feature request I think.

2 Likes

I would love community consensus logic to annotations.

I would also love to be notified when someone disagrees with the annotation I’ve made so I can go back and re-asses my understanding of the observation.

11 Likes

@rupertclayton made a very compelling point and has really thought it through. I fully support a solution as proposed, although it might be technically complicated to integrate?

Possibly, it could also be an opportunity to deal with observations concerning multiple indviduals. Typically, sightings of a bird incubating or feeding its youngs, or of an insect laying its eggs would beneficiate from the possibility to select multiple annotations for a single picture, while it would probably be silly to duplicate the observation.

2 Likes

It won’t happen, observation is for 1 individual only, nothiing bad in duplicating.

Thanks for the kind words, @tommy_andriollo. To me the idea of a consensus process for annotations is separate from the issue of supporting multiple individuals in an observation, although there are some connections between the two.

I do see some value in being able to note more than one individual in an observation (e.g. “Adult females: 1; Juveniles, sex not determined: 4”), but I suspect the added value may be outweighed by the extra complexity. My understanding is that annotations are surfaced in three places:

  1. In the seasonality graphs on taxon pages, so that you can see peak flowering season, breeding periods, etc.
  2. In the photo search page to filter observation photos to show just fruiting individuals, larvae, eggs, etc.
  3. In data exports that might be used by researchers, etc.

Let’s consider an observation of a female Mallard and four ducklings. Let’s say the user took three photos: one of the whole group, one of the mother, and a close-up of two ducklings. With the current focus that there is a primary individual in each observation, then the observer or an identifier has to choose between annotation options and goes with “Adult”, “Female”. The result is that the seasonality graphs include the mother duck, but not the ducklings, and the image search returns a photo of two ducklings when the user chooses “Adult”.

If iNat observations supported multiple individuals, we get a different result but maybe not a better one. Now the observation is tagged “Adult females: 1; Juveniles, sex not determined: 4”. The seasonality graph for Juveniles would now be more accurate. But the photo search would be a little worse, with the mother duck showing up in a search for Juveniles as well as the two ducklings showing up in a search for Adult.

The only real fix to this issue would be to associate annotations with specific images. Image #2 is “Adult females: 1”; image #3 is “Juveniles, sex not determined: 2”. But that makes the data structure and the interface massively more complex. Even if iNat were redesigned to apply annotations (and IDs?) on a per-image basis, do we really think that observers or identifiers are going to add all that data? Reworking the existing 30m+ observations would also be problematic.

To me this is the point where it’s clear that sticking with the constraint of one target individual per observation is preferable to the “ideal” solution that would sap huge amounts of development time and possibly reduce usability.

5 Likes

Actually, according to this comment by @tiwane, it does seem like per-photo annotations could be on iNat’s roadmap. If that thorny issue can be solved, then it seems to me the independent issue of community consensus for annotations should definitely be fixable.

4 Likes

Although I totally agree in principle, before I’d prioritize this over other feature requests and bug reports, I’d like to better understand the value of annotations to the iNat data set. Are researchers using the annotations? When iNat data goes to GBIF or anywhere else, are the annotations going along in a useful way? I certainly find annotations useful for filtering iNat data when I’m identifying, but wrong annotations have been very rare in my experience.

3 Likes

Maybe?
I’m revinded of the blog post about a researcher using iNat to analyze the effects of temperature on male dragonfly wing coloration, but it doesn’t say whether he filtered by the male annotation, or looked at everything and sorted out the male observations on his own.

2 Likes

I’d go as far as saying notifications for this are essential. Perhaps this could be considered in the notification overhaul.

5 Likes