City Nature Challenge - issues & suggestions for improvement

General
1

As I am sure most people reading this are aware by now, this year’s CNC event has caused a lot of disruption and frustration for many iNat users and curators. In the interests of being proactive and solution-seeking, I thought I’d put together a post compiling the exact issues, and potential actions that could be taken to alleviate them in the future.

Please feel free to comment with your own issues and suggestions, and I will incorporate them into this post.

The Issues

1. Overwhelming numbers of falsified observations, including:

  • Users uploading photos that are not their own (this is the biggest issue);
  • Uploading old photos with the date/location falsified so they become eligible for the project;
  • Uploading the same photo dozens of times with a different ID each time to increase their species “count”;
  • Marking cultivated organisms wild to prevent them from becoming casual-grade

2. Large amounts of “sockpuppet” activity - for those who may not know this term, this is when a user creates a secondary account to agree with themselves. Usually it involves adding IDs, but sometimes it is used for Data Quality votes (for instance, a user might get upset someone marked their houseplant “cultivated” and made it casual, so they create 3 more accounts just to mark it “wild” again).

3. Retaliatory harassment by new CNC users against users who flag their content or ask them to abide by iNat rules - Thankfully there were only a handful of instances of this, although it was quite unpleasant for those targeted. But since it was mostly done via DQA votes, which can’t yet be tracked, there are probably other targets who have not yet noticed the vandalism to their observations.

4. So. Many. Misidentifications. Bad IDs are a fact of iNat, but in some areas it seems like upwards of 90% of the observations are identified wrong. Generally, it appears to be due to extreme over-reliance on CV and accepting the first suggestion that appears - even if that means identifying a soda can as a great blue whale. Worse, their friends and classmates often immediately confirm the ID and make it research grade.

5. No disincentives for cheating at the Challenge. The ‘winning’ project this year has very few legitimate observations, but since the project settings allow casual observations, all the cultivated, incorrect location, and copyright-flagged observations still count towards the total. This means that a malicious user doesn’t lose any progress by being suspended or flagged - their old observations still count, and they simply make a new account and continue the same behavior.

6. Few CNC organizers have meaningfully contributed to helping with the workload generated by any of the above points

Since the beginning of CNC this year, iNat users have contributed around 8,500 copyright flags. Matter of fact, I myself did 3,600 of those (and I have the wrist strain to prove it!). This, unfortunately, is only the very tip of the iceberg - there are many times this number still up.

Many of us flaggers have felt the responses by CNC organizers in recent threads have been overly dismissive of this issue - and I have come to believe it may be because they simply do not understand the effort involved in what we are doing.

The top 20 CNC projects this year have generated a total of 1,222,099 observations, or 38% of the overall total. I checked the managers of all of those projects (77 accounts in total) and came up with this:

  • The admin of one of the largest and most problematic projects (over 900 members) has been inactive since the day CNC started

  • 7 out of 20 projects do not have a single admin or manager who has ever submitted a flag of any kind in the entire history of their account. Do they even know how to do it?

  • Only 3 out of 20 projects have had an admin or manager who has submitted even a single relevant CNC - related flag of any kind this year. The winner is Monterrey, Mx, with 98 copyright flags, followed by Sao Paulo with 2, and South Florida in third place with a single flag.

  • Another 3 projects have at least one admin who has submitted a CNC-related flag, but it was not a flaggable issue (2 observations flagged for being drawings, one flagged for no organism, one flagged for being cultivated, and one seemingly normal observation flagged ‘inappropriate’ - the flagger did not respond to being tagged to ask why).

  • In 4 out of 20 projects, no project manager has submitted a single ID to any observation in their project

  • all told, managers in the top 20 projects have submitted 30,761 IDs for 1,222,099 observations.

Potential actions by CNC organizers

  1. CNC coordinators should take a MUCH more active role in curating this event if it is to re-occur in a form that involves iNat. It is unconscionable to create this much extra work for the rest of us without sharing in the burden. I do not mind contributing, but this feels more like exploitation.

  2. Casual observations should not be allowed. Any regional projects that include casual observations should be excluded from the overall challenge. This will allow falsified observations to be removed from the challenge, and reduce the incentive for adding them. This one is very important to me personally - I despise seeing people get rewarded for cheating. Especially when it has happened multiple years in a row in the same region, and each year they get applauded for “winning” and cheat more the next time around.

  3. Participants need to be provided clearer instructions about the purpose of the event, and what constitutes an appropriate observation with emphasis removed from the competition aspect and placed on the accuracy aspect.

  4. New or inexperienced users should not lead projects. It is impossible for someone to guide new users about appropriate iNat behavior if they have no experience themselves. At a bare minimum, a user should have a decent amount of time as an active site user, a familiarity with the rules, a demonstrated ability to communicate with other users when contacted on the site, and a knowledge of how the flag system works.

Potential actions by iNat staff

I have submitted a feature suggestion for having copyright flags trigger automatic warnings and (eventually) account suspensions. You can find it here. I think this, or something like it, would be a very valuable tool to have during an event like CNC.

44 Likes
2

(reserving in case I run out of space above)

4 Likes
3

this is one of the most important ones imo

In the past, when I was a CNC organiser for Greater Sydney back in 2020-2022, there were lengthy discussions about whether casual obs should be included in some of the slack chats, email chains, and video calls. Inclusion of casual obviously ‘won’, and one of the main reasons used by proponents of inclusion was that some cities are very heavily built up and urbanised, so participants may find it difficult to find wild organisms, and that interacting with cultivated/captive species is still nature engagement, and so shouldn’t be ‘punished’.

I personally think this is a terrible argument, and anyone who tries to claim that you cannot find myriad wild organisms in even the most urbanised areas probably needs to do some more observing themselves.

32 Likes
4

Exactly!
I’ve found several first-on-iNat observations from the most urban areas around me. Manicured landscaping has all kinds of weird stuff living in it if you actually look.

14 Likes
5

I’ve done a lot of CNC IDing in previous years, concentrating on certain areas where there are lots of observations and very few IDers. This year I swore I was going to ignore the whole thing, but got sucked in somehow. But I am a bit over cultivated plants, and the weird IDs given to those cultivated plants. And the duplicates. And all the rest of the issues outlined above.
I’m always surprised that there are so few observations of insects amongst the garden plants. There are usually a few weeds, but they are outnumbered by the Pelargoniums and Roses and all the rest.
Maybe next year I’ll stick to my plan and avoid it!

7 Likes
6

Both iNat and CNC need to move away from vanity numbers and quantity - a million obs in one day. wow. underwhelm. :sob:

Turn the focus to quality. Looking for supposed extinct. Confirming a small population of. First obs (an HONEST first). Set up some careful targets that EVERY project can aim at - however small the number of participants, and wherever they are.

iNat at least could choose to display the results by RG.
Or by rare threatened endangered
Or by Invasive - since most projects are urban
Or by Endemic.

As I work on our backlog, yesterday I was IDing for a scout. Before I leave yet another - please … comment - I check if they are still active. He was on iNat yesterday. iNat could send a welcome and encouragement email / PM - thank you for doing CNC - let us know if you have questions - or feedback. Catching active new iNatters should be iNat’s goal at least. Then those no longer so newbies can / could be a positive influence on their circle next year.

CNC bubbles to the tip of the iceberg - 1% of identifiers make 75% of IDs (I wonder if you removed the 4th etc confirmation of RG, what that 1 for 75 would skew to) The 6 top identifiers also have millions of IDs - including johnascher curating his bees.

17 Likes
7

Thank you for this thoughtful summary of the issues around the CNC and the suggestions for improving it. I’d like to respond to each of your suggestions.

CNC organizers should take a more active role. Absolutely! I’m assuming you mean the local organizers here, not the global organizers? How, exactly, do the global organizers ensure that the local organizers are more active? Here, I’m assuming you mean the local organizers should make more IDs themselves, should ensure that local identifiers identify some reasonable percent of observations in their locale, should be responsive to messages from at least iNat staff, if not other iNaturalists, and should train the local observers how iNat works, what wild means, etc. I suppose the global organizers could ban problematic local CNCs from future participation, could require local organizers to have, say, at least 500 RG observations and 500 IDs on iNat before becoming an organizer, could require all observers to complete a short quiz on how iNat is supposed to work, and so on. Any other suggestions?

Casual observations should not be allowed. I absolutely agree, period. I’ll push for that in the CNC I co-organize next year, if the global organizers don’t choose to go that way.

Participants need to be provided clearer instructions. Good idea! I would add that those instructions need to come from the global organizers, not iNat staff, and that every participant should be required to sign a statement agreeing to those instructions. That’s a bunch of extra work for everyone, but how else can this work?

New or inexperienced users should not lead projects. I agree. I think requiring a minimum of RG observations and IDs by ALL local organizers, as I suggested above, is one way to ensure this. Are there other suggestions to this point?

It is a shame that all this hoop-jumping may be necessary to clean up future CNCs, but otherwise, how can the CNC fulfill its real purpose, which is to engage people with the natural, non-human world?

9 Likes
8

On the competition aspect of the thing:
CNC results shouldn’t be set on stone as the IDing process is endless. A city might lose or gain a new species months after the event was over. Places with low ID activity such as non-european non-North America areas need several weeks before all the major ID fixes are done and they will never get a chance to actually compete with the most active areas. If by all means there should be an announcement of the event numbers, do so in 2 months or something like that.

Anti-sockpuppeting:
Prevent new accounts from ID-ing CNC stuff in the first week(s) of the event. And here also a call to CNC organizers that say “…from now on observation posting stage is done. Go visit the observations of others and ID them”. This is also very problematic and it is not sockpuppeting per say, but it leads people agreeing to wrong IDs or doing mass and wrong CV ID-ing just to get some medals or merit mentions or whatever.

Force CV to check again if no location was given:
When people are submitting obs they tend to click on what the CV suggest BEFORE adding a location to the obs. I propose removing whatever the CV said if the person add/changes the location of the obs. (or some detection protocol where the CV is run again under the hood for the new location and then an alert saying “oh you might want to revise what i proposed earlier” if it detects it can be a different thing).

Instructing the CNC organizers themselves:
I sense most CNC organizers are people that like doing crowd related activities, but that don’t actually know iNat so much to be effective in the use the platform and transmit that skill to its group. So, create a How-to-effectively-iNat tutorial for CNC organizers focusing on teaching them things such as:

  • What to post on iNat during CNC (no cultivated, no zoo-stuff)
  • How to handle the group in a way they don’t post the same blurred cacti 100x times (split the group to cover a big area? some people check trees, some check bushes, some focus on plants, some focus on insects etc)
  • Extra care during obs posting (make sure people know how to post all pics in the same obs instead of multiple ones; Put location before trying CV suggestion; teach and suggest people to post with a higher taxa level: fungi, plant, insect, etc… easy to fix levels);
6 Likes
9

Well it’s City Nature Challenge, not City Agriculture Challenge. It’s the organizers duty to educate participants on how to find wild organisms. Even spiders in your kitchen qualify if they are not escapees. It’s not really hard to understand if organizer is naturalist. Preferrably not a “power user” who posts a hundred blurry crow photos from the same place too. Quality should be over quantity. Deep understanding over engagement numbers. iNat is not social medium, it’s sitizen science platform, and science has rules to prevent devaluation of knowledge.

8 Likes
10

One more reason to push for separating the “captive/cultivated” category from the rest of the casuals (here’s another). I think including captive/cultivated makes sense for a project focused on urban areas, since parks and gardens (and maybe even the local zoo) are places city dwellers may visit to “interact with nature.” However, I think all the “observations” missing a critical piece of what makes an observation (picture missing/removed for copyright violation, wrong time/location, no evidence of organism etc.) should be excluded.

Also, some kind of duplicate detection and a better way for dealing with these might be helpful. E.g. as an identifier, I’ve sometimes found myself wishing for a way to detect multiple copies of an image across iNat observations. There are legitimate reasons why someone might upload an image already in the database, so if this could be implemented, people uploading an image already on iNat could be prompted to provide an explanation for why they’re doing it (e.g. identifying another organism in the picture, sharing an observation with a hiking buddy). Even better if it was possible to choose to link all the observations to the same instance of an image in the database the way this happens when you use the “duplicate” feature on an observation.

6 Likes
11

In my area there were a few users who would upload different images of the same organism, sometimes with the same ID (best case), sometimes with different IDs. Before the CNC I had rarely seen where people would post evidence of one organism across multiple posts. This was the opposite problem I’ve been facing with newer users who upload 5-10 organisms in one observation and then never get to splitting them.

I wish there was a way that these could be tagged other than putting comments in the observations so that moderators can combine them.

There was one user in particular who was just gaming (really scamming) the system by uploading over 600 photos of rocks, dead branches, the soil, and/or incredibly zoomed in parts of e.g. plants so that it became unrecognizable. While I commend the dedication, it’s at the same time absurd and I had to exclude this person from my ID pages. I didn’t want to deal with 600+ nonsense observations.

I loved the CNC, but there definitely needs to be a few more regulations in place to prevent pure spam.

7 Likes
12

Speaking of misidentifications, there were multiple instances where a user misidentified other user’s observations intentionally so that the user who misidentified it could win the CNC or a contest. I’m still looking over CNC results for Long Island, NY, but aside from many false CV suggestions and cultivated plants marked as wild, there appear to be no blatant Inat rule violations.
Looking over the flags reveals thousands of copyrighted images, including screenshots, other people’s observations, observations from a checklist, etc. Inat needs to be a safe place for everybody, and everyone has the right to be identified fairly and not have their observations be copyrighted. Thank you to everyone who has helped make Inat a better and safer place, especially with the massive surge in traffic during CNC.

6 Likes
13

This is a radical idea that may be too unkind for implementation, but . . . . Warn all organizers that if their project next year has a larger than usually amount of copyright violation and other activities that are obviously cheating, next year that challenge (and any other set up to cover approximately the same area) will experience this: next year finalizing the uploading of all photos from that area will be delayed until after the CNC is over. (We welcome your observation but too many participants are treating this as a contest.)

Message this beforehand to everyone who seems to be making observations in that area. And/or have a pop-up with an explanation for each attempted upload.

9 Likes
14

Identifier speaking here. Can we organize the CNC every four years, like the Olympic Games?

9 Likes
15

And perhaps like the Olympics certain bad-behaving teams can be disqualified from participating.

8 Likes
16

How would we track this? I know the La Paz CNC won but had over 500 copyrighted observations. Maybe if the amount of copyrighted photos exceeds 0.2% of observations we could flag the entire project?

3 Likes
17

@graysquirrel these are all excellent suggestions. IMO the way the CNC has been working up to now is just no longer sustainable.

The current system just rewards bad behaviour and dilutes the usefulness of iNat data.

There is an issue that has been bugging me a lot since the CNC: some accounts post what are fairly obviously copyrighted photos, but these are not easily identifiable as such by Google lens or TinEye.

I feel very frustrated at the amount of work required to prove that a photo is copyrighted. Some users post hundreds of observations with photos that are all different sizes and aspect ratios, and contain no EXIF data. At a small scale, one can spend time searching to determine if these photos were stolen from elsewhere, but at CNC scale this just does not work.

17 Likes
18

Or just get rid of CNC altogether and think of other ways to promote interest in biodiversity/iNaturalist? These kind of shenanigans undermine the notion that iNaturalist can be a place for serious citizen science and data collection.

Also, CNC was started back in 2016 when iNaturalist was not well known. It now is widely known throughout the world. The need for something like CNC to publicize iNaturalist is much less than it used to be.

25 Likes
19

for that we have DQA - not a Single Subject.
If you use that the obs goes to Casual.

3 Likes
20

If CNC is to continue I strongly support making Casual observations ineligible for inclusion in the counts. Also allowing much more time before a winner is announced since it takes months to really deal with all the submitted records.

11 Likes