The current approach for annotations (Adult, Alive, Flowering, etc.) is that any user can apply an annotation and other users can then vote that up or down, but subsequent annotation choices are limited by what other users have already chosen. I’m proposing a system that more closely matches the way we reach consensus on IDs. I believe it would improve the accuracy of annotations on iNat and be easier for users to understand.
Let me outline the issue: Let’s say I come across another user’s observation that is actually of a juvenile male bird, maybe a Ruby-throated Hummingbird. Maybe, being over-confident about my knowledge of the species, I’m sure it’s an adult female and I choose the “Adult” and “Female” annotations. Any other iNat user who then views that observation cannot contribute their opinion that it’s really a juvenile male. All they can do is down-vote (or up-vote) the “Adult” and “Female” annotations. That’s because for most annotations selecting one option causes the others to disappear.
This appears to be happening because reasonable limitations on one person’s actions are mistakenly being applied across all iNat users. It’s good that iNat won’t allow me to say that a single bird is both “Adult” and “Juvenile”, but it’s not good that my identifying it as an adult prevents your identifying it as a juvenile.
I accept that this is an infrequent issue, but given the vast numbers of iNat observations it’s one that does have some impacts:
Some observations will be “stuck” with incorrect annotations unless the user who originally applied them can be persuaded to remove them. Contrast this with IDs, where the community can overrule an initial bad ID.
If the user who applied the incorrect annotation leaves iNat, there is no way to fix the incorrect annotation. Again, that doesn’t happen with incorrect IDs, which can be outvoted by correct IDs even if the first identifier is AWOL.
The value of iNat’s data for research use is diminished either because annotations are known to include unfixable annotation errors, or because observations need to be excluded where there’s any dispute over annotation accuracy.
I propose that iNat applies to annotations a slightly simplified version of the logic that is used for IDs. In the description below I used the term “consensus” to describe an annotation value that has sufficient support for in iNat to return it as a search result or supply it as part of an RG data feed.
iNat modifies its schema to store annotation values tagged with the ID of the user applying them.
In the iNat UI, conditional logic (e.g. “hide life stage choices if a life stage is already selected”) is applied on a user-specific basis.
If there are no disputes, then a single “vote” is sufficient for an annotation value to be regarded as the consensus for an RG entry (i.e. one vote for “Adult” is sufficient for iNat to say the organism is an adult)
If any value for a particular annotation is disputed, then >2/3 support is required for a value to have consensus support. (I say “Adult” and “Female”, you say “Juvenile” and “Male”. Neither annotation has consensus support. When two more users vote for “Female” that gains consensus support.)
For incompatible annotations (“Adult” and “Juvenile”, “Flowering” and “No Evidence of Flowering”) there’s an edge case where there could be > 2/3 support for more than one option. The result should be no consensus for any option.
Where multiple annotation values may be true simultaneously the above restriction should not apply (e.g. a plant can be both “Flowering” and “Fruiting” so long as there’s >2/3 support for those annotation values and <2/3 support for “No Evidence of Flowering”.)
Other logic may work better than my proposal above. The core point is to apply a consensus process to annotations that’s similar to what we have for IDs.
This proposal came out of a discussion here about the new “No Evidence of Flowering” annotation value.
I’m up to make wong annotations be removable by voting, though with observations with multiple specimens different people can have different view on life stage and sex, and we already have some troubles with it, and it can end with voting against the specimen the author chose. Also observers can delete wrong annotations done by others and it will be against this system, I guess? While now it’s easier to deal with others’ wrong annotations.
When one observations havemultiple specimens and you know which one is there, but don’t add annotation right when you upload or with current situation half of annotations don’t safe sometimes someone else comes, looks at the ob, and choose the wrong annottion because it’s a different person with different preferanes. Or you upload one series of pictures, one with female and one with male, and on one you add annotations and something happens with annotations on the 2nd one, someone else comes and chooses not what you decided, thn you have 2 males, for example. Not the most common situation, but happend with me with obs that had both larvae and adults.
Thanks. I agree that those issues exist with an observations that includes multiple individuals. It doesn’t seem to me that the current annotation logic deals with them better than the consensus approach I proposed. In both cases the observer can add a comment to say “This observation is for the larvae” and other users can add comments such as “Your photos include both females and males”. Supporting multiple individuals within one observation would be a subject for a different feature request I think.
I would love community consensus logic to annotations.
I would also love to be notified when someone disagrees with the annotation I’ve made so I can go back and re-asses my understanding of the observation.
@rupertclayton made a very compelling point and has really thought it through. I fully support a solution as proposed, although it might be technically complicated to integrate?
Possibly, it could also be an opportunity to deal with observations concerning multiple indviduals. Typically, sightings of a bird incubating or feeding its youngs, or of an insect laying its eggs would beneficiate from the possibility to select multiple annotations for a single picture, while it would probably be silly to duplicate the observation.
Thanks for the kind words, @tommy_andriollo. To me the idea of a consensus process for annotations is separate from the issue of supporting multiple individuals in an observation, although there are some connections between the two.
I do see some value in being able to note more than one individual in an observation (e.g. “Adult females: 1; Juveniles, sex not determined: 4”), but I suspect the added value may be outweighed by the extra complexity. My understanding is that annotations are surfaced in three places:
In the seasonality graphs on taxon pages, so that you can see peak flowering season, breeding periods, etc.
In the photo search page to filter observation photos to show just fruiting individuals, larvae, eggs, etc.
In data exports that might be used by researchers, etc.
Let’s consider an observation of a female Mallard and four ducklings. Let’s say the user took three photos: one of the whole group, one of the mother, and a close-up of two ducklings. With the current focus that there is a primary individual in each observation, then the observer or an identifier has to choose between annotation options and goes with “Adult”, “Female”. The result is that the seasonality graphs include the mother duck, but not the ducklings, and the image search returns a photo of two ducklings when the user chooses “Adult”.
If iNat observations supported multiple individuals, we get a different result but maybe not a better one. Now the observation is tagged “Adult females: 1; Juveniles, sex not determined: 4”. The seasonality graph for Juveniles would now be more accurate. But the photo search would be a little worse, with the mother duck showing up in a search for Juveniles as well as the two ducklings showing up in a search for Adult.
The only real fix to this issue would be to associate annotations with specific images. Image #2 is “Adult females: 1”; image #3 is “Juveniles, sex not determined: 2”. But that makes the data structure and the interface massively more complex. Even if iNat were redesigned to apply annotations (and IDs?) on a per-image basis, do we really think that observers or identifiers are going to add all that data? Reworking the existing 30m+ observations would also be problematic.
To me this is the point where it’s clear that sticking with the constraint of one target individual per observation is preferable to the “ideal” solution that would sap huge amounts of development time and possibly reduce usability.
Actually, according to this comment by @tiwane, it does seem like per-photo annotations could be on iNat’s roadmap. If that thorny issue can be solved, then it seems to me the independent issue of community consensus for annotations should definitely be fixable.
Although I totally agree in principle, before I’d prioritize this over other feature requests and bug reports, I’d like to better understand the value of annotations to the iNat data set. Are researchers using the annotations? When iNat data goes to GBIF or anywhere else, are the annotations going along in a useful way? I certainly find annotations useful for filtering iNat data when I’m identifying, but wrong annotations have been very rare in my experience.
Maybe?
I’m revinded of the blog post about a researcher using iNat to analyze the effects of temperature on male dragonfly wing coloration, but it doesn’t say whether he filtered by the male annotation, or looked at everything and sorted out the male observations on his own.
I’m making heavy use of Annotations in my current gall phenology project. They are extremely useful but in these taxa it is very common for users to misapply Annotations in several different ways and it’s frustrating that there’s no way for me to correct them without asking each user to do it themselves (only works if the uploader still uses the site!). My preference is that the Annotation should, like Observation Fields, simply take the most recently added value. I imagine there are some cases where this could cause a contentious back and forth but requiring multiple votes to overturn bad annotations would be inconvenient for moving quickly through large datasets (the only speed you want to be moving when the datasets are large!) Though a consensus system as outlined in the OP would be a major improvement over the current implementation.
Just using the most recently applied annotation would certainly be a simpler approach @megachile. And it’s reasonable to think there will be a lot fewer real disagreements about annotations than there are about IDs, so we can hope that edit wars would be a rare issue. But I guess there will still be some disagreements and iNat would really need some policy about how those should be resolved.
Of course, where annotations are mutually exclusive (Adult/Juvenile or Female/Male) someone who wants to correct an annotation needs to have the ability to do that. That means iNat still needs to change the current UI logic where selecting one value hides the option for anyone else to apply that annotation.
I think simply adding a comment in notes as to which organism is the focus of an observation could help remedy that potential issue, though of course, it won’t fix the problem entirely.
Even so, I believe the benefits of this proposed system would out way the cons.
I have written a journal post that shows how some of us use the annotations - The power of annotations
Wrong annotations mean that one may not find the obs when looking for images to compare or to identify single life stages. Also incorrect obs end up (or don’t end up) in collection projects that are collecting based on annotation.
Thank you @rupertclayton for adding this feature request!!
Yes! I identify a lot of insects and Life Stage is extremely important for filtering those observations. Identifying a caterpillar (Larva) and a moth or butterfly (Adult) are totally different skill sets.
There are also insect collection projects (for example, the Caterpillars of Eastern North America project) that use Life Stage to add observations. If I come across a sawfly larva that was identified as a caterpillar while identifying for the project, I can identify it as a sawfly and once the consensus ID shifts, it will no longer be included in the project. If someone misclicked the Life Stage annotation and marked an adult moth as a Larva, the only way to get it out of the project is to mention the original annotator and ask them to fix it.
Yes, annotations of “dead” combined with life stage annotations are really helpful for investigating sources of Rallidae mortality (and I assume other birds as well).
I see wrong annotations for Rallidae often. Most of these are individuals who are no longer downy chicks but do not yet have adult plumage. Part of this is caused by insufficient categories (I really think “juvenile” should be broken down into either “chick” and “subadult” or “juvenile” and “subadult”). But regardless, “juvenile” is more appropriate in these cases than “adult”.
Also, many people mistake subadult Purple Gallinules for adult females, because other bird species often have drab females and showy males. Purple Gallinules are not sexually dimorphic (except for subtle differences in average size).