Insight in disagreed annotations

General
8

there’s not a straightforward way to look for the particular information you want because it’s a complicated request, and there are a lot of potential variations in how exactly any particular persion might want to filter for observations in their particular case. maybe one person wants any disagreement on a particular annotation. (that’s the way i did it.) but maybe another person wants only disagreements by a particular person on any annotation. and maybe another person wants another person wants only disagreements on any annotation by a particular person. each of these variations requires very different ways of parsing the data to get the desired result.

see the link to my Jupyter Notebook in my previous post. as it is currently saved, you would:

  1. uncomment (remove the leading #) from items / lines in the parse_fields definition for the async def get_obs which begin with label: 'annot_score_ls...'. (this will tell the code to return the overall scores for life stage annotations, which i’ve defined as 1 for the annotation, plus 1 for each agreement, minus 1 for each disagreement)
  2. set req_params_string = 'user_id=rudolphous&term_id=1'. (this tells the code to get your observations that have a life stage annotation.)
  3. set obs = await get_obs(req_params, get_all_pages=True, use_authorization=False). (the get_all_pages=True parameter will get up to the first 10,000 records for the parameters you defined in #2, which will be a high enough limit in your case.)
  4. starting at the top, run each cell that has code, through to the cell that includes obs = await get_obs.
  5. run the cells in the section that will write a CSV.
  6. look in the pane on the left, and you’ll find the resulting CSV. you can browse it directly in the browser or download it to view it using tools / applications on your own machine. look for any observations / records where any of the socres <1 in the life stage annotation score columns. these would be observations that have disagreements.

note that although score <1 is not a perfect indicator of disagreements, it should work in most cases and was the easiest way to find these without doing new coding. you could make it better by filtering the votes on the individual annotations for just disagreements and counting those vote records. any vote count > 0 would indicate there was a disagreement.

instead of looking yourself for scores <1 in step #6, you could also tell the code to filter for records that have scores <1 directly by adding a parameter in step #3 that would be something like post_parse_filter_function=(lambda x: x['annot_score_ls_adult'] < 1 or x['annot_score_ls_juvenile'] < 1 or ...), where you would define which particular life stage annotations you want to look for overall scores less than 1.