The value of expert identification blitzes

[thebeachcomber]

About a year ago, I posted some preliminary results from a study I conducted for my PhD, looking at an expert assessment of identification accuracy for Australian iNat records. That research has just been published today, and is open access here: https://nph.onlinelibrary.wiley.com/doi/10.1002/ppp3.70005

One of the main focuses of the paper was of course the identification accuracy findings. To sum up that element:

• in three weeks, almost 11,000 records of Western Australian plants had at least one identification added to them by 52 experts (taxonomists, botanists, herbarium curators etc)
• for the 7,000+ records that started at a species ID or finer before the event, 92% of them were correctly identified
• the Research Grade only subset of those 7,000 (= ~3,500) was 97% correctly identified
• ID accuracy was variable taxonomically and geographically, but some groups had exceptional accuracy, e.g., for 510 records of the genus Drosera, just 0.2% were misidentified at the time of the event.

But the main reason I’m making this forum post is to highlight the actual expert ID blitz itself, which I strongly believe is a really valuable approach that we need to be implementing more across the world. To pull directly from the paper:

The vast majority of concerted citizen science ‘blitz’ events are BioBlitzes, i.e., organised events during which participants aim to record as many species as possible in a defined location and time period (Parker et al., 2018). Thousands of BioBlitzes are held each year, ranging from global or hemisphere-spanning events such as the City Nature Challenge and Great Southern BioBlitz (Palma et al., 2024) to far more localised and focused events (Rice, 2022; Rokop et al., 2022). Although BioBlitzes often have multiple goals and outcomes, including inventorying a defined location, discovering new species and building community engagement with the natural world (Meeus et al., 2023), these events fundamentally revolve around increasing the value of biodiversity datasets by adding entirely new occurrence records. In comparison, expert ID Blitzes ─ which increase the value of biodiversity datasets by instead adding new identifications to existing occurrence records ─ are far rarer and greatly underutilised despite their clear value. Beyond our specific event, and regardless of the taxa or regions involved, there are numerous benefits facilitated by expert ID Blitzes. Although these mostly centre on the improvement of citizen science data quality, there are also benefits derived by the experts themselves.

So I wanted to make this thread to hopefully inspire other people from around the world to run their own expert ID Blitzes. We go into extensive detail about expert ID Blitzes, including how to organise them, tips and tricks, etc., in section 4.3 of the paper. I also want to explicitly note here that we define an ‘expert’ quite broadly, and do not limit this only to professional taxonomists and researchers. We note:

4.3.1 Diverse expert recruitment
Recruiting a diverse range of expert contributors is invaluable for maximising the knowledge base available during an ID Blitz. While professional taxonomists will often be the first group of experts to be considered for participation, there are many other experts with important contributions to make. Given that most iNaturalist records are associated with photographic vouchers, and that identifying organisms from images often requires a different skillset to identifying physical specimens (Chulif et al., 2023), recruiting field workers with significant experience identifying living plants is also crucial. Many amateur naturalists are highly skilled identifiers with significant knowledge of their focal taxa and often have intimate local knowledge of the organisms that occur in a particular region. Despite not having professional qualifications in the field, these naturalists are often outstanding amateur experts who have invested significant time and effort mastering a particular group of organisms (Viola et al., 2022). We strongly emphasise the importance of recruiting both amateur and professional experts to contribute to ID Blitzes as their skillsets and knowledge domains are often complementary. Knowledge of natural variation within a species can be key to providing a confident identification for sub-optimal image sets. Individuals with field experience are often far more likely to be able to recognise species that lack critical features usually required for identification (e.g., flowers or fruit), or juvenile plants that may differ significantly in appearance from adult plants, but are poorly represented in herbarium collections and field guides which naturally emphasise reproductive material.

I’d love to hear about similar events other people have run, and also very happy to provide advice for prospective future events.

[DianaStuder]

Skimmed. Will read it later.

Were you able to recruit some new identifiers for IDs going forward? Perhaps to curate their own slice (now they have it sorted)?

[thebeachcomber]

@DianaStuder

Were you able to recruit some new identifiers for IDs going forward?

yes, some of the new identifiers now use iNat regularly

[jhbratton]

Did you have to pay for the time the professionals put in? I see a few professional UK entomologists on iNaturalist. If you wanted to greatly increase the input from, say, London Natural History Museum staff, I imagine the museum administrators would be asking who is going to pay for all this staff time.

[thebeachcomber]

I didn’t have to pay anyone, everyone involved was generous enough to donate their time. Many of them made their IDs on weekends or outside of work hours.

In some cases, experts making IDs on iNat can include that time within their work I believe (from a few previous discussions on a couple of Australian observations, if I recall correctly). Especially the case for the increasing number of experts who use iNat data as part of taxonomic revisions they’re working on

[josscarr]

This is really fascinating and important stuff Thomas – thank you :)

I’m just starting out my Master’s dissertation which will be looking at iNaturalist data so papers like yours are immensely helpful to me right now.

[cthawley]

I would also note that many “traditional” bioblitzes (ie, those focused on creating rather than IDing observations) themselves are volunteer events. Apart from the organizers, most participants are there in a volunteer capacity.

[lynnharper]

Fantastic and fascinating! So, if I’m reading the paper correctly, your experts identified both Research Grade and Needs ID observations, while you and your co-authors were particularly interested in measuring how accurate those added IDs were - is that right? Here is the northeastern United states, we’ve been running an annual 48-hour plant ID-a-thon in February for four years now. This hasn’t been focused on measuring accuracy; it’s really been aimed at reducing the number of Needs ID observations, so very little attention was paid to checking the IDs of Research Grade observations. Over those eight days in four years, I estimate we’ve reduced the Needs ID pile by a little over 80,000. (It’s shocking to realize we were making 10,000 IDs a day.) Depending on the year, we had around 20 to 60 members of the ID-a-thon projects, some of whom were recruited from the New England and New York botanical associations, but most were already active plant identifiers on iNat.

So, what I’m wondering is if there is some useful information to be gleaned from comparing your work to our work. I expect we’ll be running another ID-a-thon next February - should we focus on checking the accuracy of RG observations? Maybe concentrate on bringing older observations in line with current taxonomy? Expand the ID-a-thon to a week? Recruit specific experts to work on “difficult” taxa (sedges, hawthorns, etc.)? Just keep on having fun seeing the Needs ID numbers go down? What would you do with a band of experienced, enthusiastic identifiers like ours?

[thebeachcomber]

@lynnharper

So, if I’m reading the paper correctly, your experts identified both Research Grade and Needs ID observations, while you and your co-authors were particularly interested in measuring how accurate those added IDs were - is that right?

not quite. We operated under the assumption that any ID added by an expert during the blitz was right unless corrected by a second expert (with all the caveats this entails, namely that experts aren’t infallible of course, and we discuss this in the paper). So what we were comparing for each observation for our main analysis was:
species ID at the moment the blitz began (= before)
vs
species ID the moment the blitz ended (= after)

if these two were the same, ie the expert ID confirmed the existing ID, that was +1 for an accurate/correct ID

so rather than measuring how accurate the added IDs were, we were measuring how accurate the pre-blitz IDs were, using the added IDs as our point of reference.

@lynnharper

Here is the northeastern United states, we’ve been running an annual 48-hour plant ID-a-thon in February for four years now.

awesome! I wish more of these existed! And I’m definitely keen on setting up something similar that runs on a regular basis rather than only a one-off

@lynnharper

should we focus on checking the accuracy of RG observations? Maybe concentrate on bringing older observations in line with current taxonomy? Expand the ID-a-thon to a week? Recruit specific experts to work on “difficult” taxa (sedges, hawthorns, etc.)? Just keep on having fun seeing the Needs ID numbers go down? What would you do with a band of experienced, enthusiastic identifiers like ours?

I think all of these things have value, and if you have the time and scope to cram them all in to your 48 hours, the more the better. Personally, I think “Recruit specific experts to work on “difficult” taxa (sedges, hawthorns, etc.)” is one of the most valuable things you could do, and we explicitly address this recommendation in the paper:

One important challenge to acknowledge is that some taxa are disproportionately ‘ignored’ (i.e., left unreviewed; Table 1) during expert ID Blitzes. Records of these taxa tend to be disproportionately ignored due to one, or a combination, of three common drivers: an absence of images showing key diagnostic characters, especially where these characters may be small, obscure, or non-intuitive for someone without knowledge of that group (e.g., leaf attachment to stems, hair types, or petiole glands); inherent difficulties in identification due to cryptic diagnostic characters, high infraspecific variability, or many similar congeners; and, a paucity or absence of available experts with knowledge of these taxa. In some cases, these taxa may also have higher rates of misidentification, and so their non-review and thus exclusion from identification accuracy statistics may possibly skew the overall results. For example, although we found a greater than 92% identification accuracy across more than 7,000 vascular plant records, it is unlikely these statistics hold true for some genera such as Acacia Mill., Dampiera R.Br. or Synaphea R.Br., which are often difficult to identify from photographs ─ and for which there were few or no active taxonomic experts on iNaturalist ─ and thus for which a large proportion of the records in our overall dataset went unreviewed during the event. Targeted recruitment of experts with knowledge of taxa that are known to be disproportionately ignored across citizen science records will likely 1) improve the proportion of these records that are reviewed during expert ID blitzes and 2) improve the quality of future observations of these taxa, especially where experts have provided comments and guidance on which characters are important to photograph. While a shortage of taxonomic specialists is a factor that limits the success of expert ID Blitzes, unreviewed observations may be subject to future expert verification if existing constraints on taxonomic capacity are addressed, and as such have the potential to provide valuable data to aid future research.

so this would be at the top of my list for both the short-term and long-term positive impacts.

It can definitely fall down the list of priorities given so many records are posted every day, but I do think there is strong value in checking RG stuff as well, because there are clearly still mistakes in there regardless of how high the ID accuracy is.

[lynnharper]

Ah, I wrote incorrectly - I did mean you were looking at the accuracy of the pre-existing IDs on your group of observations.

Finding experts for difficult taxa is not easy, if only because there are so many observations that can never get even to genus due to a lack of photos of the necessary characters. I think experts get a little discouraged by seeing dozens of unidentifiable observations. But I wonder if non-experts like me can do an initial clean-up - I’m thinking of something like marking non-flowering, non-fruiting hawthorns As Good As Can Be at the genus level (assuming that’s appropriate) - and then call in the real experts.

I think many people enjoy the opportunity to participate in a concentrated event such as our ID-a-thon, but I agree that having something that runs on a regular basis would be ideal, especially if we can figure out a way around iNat’s somewhat cumbersome communication and notification systems. We’ve just started an on-going project aimed at cutting down on the need to tag in additional identifiers; maybe in a month or so I can let you know how that goes.

[phoekman]

Congrats on the publication! Great that you were able to quantify and quality the contributions of such events. I agree that putting more focus on improving the capacity of identifiers would be very helpful to keep up with iNat’s growth. These ID Blitzes sound like a very good way of doing so!

In this proposal to set up a new “Identification Center”, I also suggested a section dedicated to identification events. I think combining events around ID’ing WITH additional training and resources, and making it very place- and/or taxon-specific would go a great deal in increasing identification capacity by making it accessible to beginners and seasoned identifiers alike.

[jasonhernandez74]

@thebeachcomber

In comparison, expert ID Blitzes ─ which increase the value of biodiversity datasets by instead adding new identifications to existing occurrence records ─ are far rarer and greatly underutilised despite their clear value.

There used to be ID-a-thons.

[a_emmerson]

Hi @thebeachcomber these are a couple of great projects that have both ID and annotation blitzes on a regular basis, it is amazing how the numbers of research grade observations can be increased over a very short time period:
https://www.inaturalist.org/projects/uk-hoverflies-syrphidae
https://www.inaturalist.org/projects/syrphing-europe
Take a look at the journals to see the regular updates on how the identifications and annotations improve over time.
I am working up a couple of other projects on UK Galls and UK Leafmines where I hope to start a similar thing.

[sedgequeen]

We’ve done ID-a-thons for Oregon plants. Gotten lots of ID’s.

[charliealice13]

This is a really interesting perspective and has me questioning the most effective way for me to use iNaturalist for my current project. I’m focused on Vascular plants in Spain at the moment and wondered whether people think it’s more effective to reach out to experts external to iNaturalist and attempt bring them onto the platform to do an identification blitz? Or should I be focusing my efforts on reaching out to identifiers already in the community to try and mobilise identifications of key taxa in my target regions? I’ve been involved in plenty of citizen science projects in the past but normally face to face with the public doing outreach and running workshops so navigating the digital community is new territory for me!

For context I’m interning on a project aiming to use satellite images and remote sensing to build better biodiversity models and it’s in the early stages where we’re trying to “calibrate” the models by making sure that they line up with existing biodiversity data. However some areas don’t have enough data available on GBIF to meet required threshold and I’m trying to bridge the gap using citizen science data by ensuring as many observations as possible are uploaded.

Hope this is okay to ask, would love to know whats worked best for anyone in a similar situation in the past.

[DianaStuder]

Is it this project?
https://forum.inaturalist.org/t/getting-paid-to-id/53630

[lynnharper]

I think it is most effective to pursue both ways of recruiting experts. Certainly people who already do lots of IDs of vascular plants in Spain will be likely to want to help, but if they can’t help with some groups of plants, it’s worth asking outside experts to help.

[sedgequeen]

In my experience, experts who aren’t already on iNaturalist are resistant to signing up, because they have so much else to do, but are likely to help with individual observations if asked. (I e-mail the URL with the request.) Nonetheless, it is worthwhile to ask for the help, explaining the time limit if there is one (help just next week? for two weeks in June?). You may manage to lure them in once they have helped a little bit.

[spiphany]

It isn’t clear to me what the context is, but if this is a commercial undertaking it seems like rather poor form to ask IDers to volunteer their time for your benefit.

[charliealice13]

No it’s not its a collaborations between a few universities and charities to do a big modelling project of EU biodiversity data and they’re trying out all different routes of acquiring data from different online sources to. see whats an effective way to fill the gaps in our current understanding. My internship is looking at citizen science data sources and seeing where there is untapped potential in the data available.