# Many IDs have ID category: null **URL:** \ **Category:** Bug Reports\ **Tags:** web\ **Created:** [December 27, 2020, 6:15pm UTC](https://forum.inaturalist.org/t/many-ids-have-id-category-null/19061 "2020-12-27T18:15:50Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![bouteloua](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.inaturalist.org/bouteloua/32/1180_2.png) [@bouteloua](https://forum.inaturalist.org/u/bouteloua)\ **Post date:** [December 27, 2020, 6:15pm UTC](https://forum.inaturalist.org/t/many-ids-have-id-category-null/19061/1 "2020-12-27T18:15:50Z")
Found this when trying to square my various ID stats [not matching](https://forum.inaturalist.org/t/year-in-review-id-stats-exclude-those-on-casual-obs/19052/2) when looking at my Year in Review. | ID Category | My 2020 IDs via [inaturalist.org/identifications](http://inaturalist.org/identifications) | My 2020 IDs via API [GET /identifications](https://api.inaturalist.org/v1/docs/#!/Identifications/get_identifications) | | --- | --- | --- | | Improving | [2,505](https://www.inaturalist.org/identifications?for=others¤t=true&d1=2020-01-01&d2=2020-12-31&user_id=bouteloua&category=improving) | [2,505](https://api.inaturalist.org/v1/identifications?own_observation=false&user_id=bouteloua¤t=true&d1=2020-01-01&d2=2020-12-31&category=improving) | | Leading | [2,650](https://www.inaturalist.org/identifications?for=others¤t=true&d1=2020-01-01&d2=2020-12-31&user_id=bouteloua&category=leading) | [2,650](https://api.inaturalist.org/v1/identifications?own_observation=false&user_id=bouteloua¤t=true&d1=2020-01-01&d2=2020-12-31&category=leading) | | Supporting | [4,965](https://www.inaturalist.org/identifications?for=others¤t=true&d1=2020-01-01&d2=2020-12-31&user_id=bouteloua&category=supporting) | [4,965](https://api.inaturalist.org/v1/identifications?own_observation=false&user_id=bouteloua¤t=true&d1=2020-01-01&d2=2020-12-31&category=supporting) | | Maverick | [2](https://www.inaturalist.org/identifications?for=others¤t=true&d1=2020-01-01&d2=2020-12-31&user_id=bouteloua&category=maverick) | [2](https://api.inaturalist.org/v1/identifications?own_observation=false&user_id=bouteloua¤t=true&d1=2020-01-01&d2=2020-12-31&category=maverick) | | Total | [10,122](https://www.inaturalist.org/identifications?for=others¤t=true&d1=2020-01-01&d2=2020-12-31&user_id=bouteloua) | [10,489](https://api.inaturalist.org/v1/identifications?own_observation=false&user_id=bouteloua¤t=true&d1=2020-01-01&d2=2020-12-31) | Totalling up each of the four ID categories, you get 10,122 IDs - i.e. the sum of the obs in the pie charts. But if you search [IDs via the API without a category specified](https://api.inaturalist.org/v1/identifications?own_observation=false&user_id=bouteloua¤t=true&d1=2020-01-01&d2=2020-12-31), there are 10,489 IDs, meaning there are 367 obs in that time period that do not have a category assigned. > [@Year in Review: ID stats exclude those on Casual obs](https://forum.inaturalist.org/t/year-in-review-id-stats-exclude-those-on-casual-obs/19052/2): > > for example: [https://api.inaturalist.org/v1/identifications?id=84340111](https://api.inaturalist.org/v1/identifications?id=84340111) ([https://www.inaturalist.org/identifications/84340111](https://www.inaturalist.org/identifications/84340111)). > > i suspect category=null is a bug(?) / data issue in at least some of these situations. for example, here’s another identification that’s almost the same as the one above, but it’s categorized as supporting (as i would expect): [https://api.inaturalist.org/v1/identifications?id=84340067](https://api.inaturalist.org/v1/identifications?id=84340067) ([https://www.inaturalist.org/identifications/84340067](https://www.inaturalist.org/identifications/84340067)). > > it looks like identification records with category=null tend to show up in clustered in time. so maybe there was something going on with the system at those times. for example, in your case, you have a lot of category=null identifications from 2020-01-17 21:26:22 (-06:00) to 2020-02-05 23:02:14 (-06:00), although not every identification during that period was uncategorized. [https://api.inaturalist.org/v1/identifications?id=84340111](https://api.inaturalist.org/v1/identifications?id=84340111) ![image](https://us1.discourse-cdn.com/flex016/uploads/inaturalist/original/2X/3/35cd1a37a41e1b764d6751a4fe6422e013519f50.png) 2020-01-17 to 2020-02-05: [https://api.inaturalist.org/v1/identifications?own\_observation=false&user\_id=bouteloua¤t=true&d1=2020-01-17&d2=2020-02-05](https://api.inaturalist.org/v1/identifications?own_observation=false&user_id=bouteloua¤t=true&d1=2020-01-17&d2=2020-02-05) ---
**Author:** ![astra\_the\_dragon](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.inaturalist.org/astra_the_dragon/32/37251_2.png) [@astra\_the\_dragon](https://forum.inaturalist.org/u/astra_the_dragon)\ **Post date:** [December 27, 2020, 6:46pm UTC](https://forum.inaturalist.org/t/many-ids-have-id-category-null/19061/2 "2020-12-27T18:46:55Z")
Maybe withdrawn IDs have something to do with it. ---
**Author:** ![pisum](https://avatars.discourse-cdn.com/v4/letter/p/91b2a8/32.png) [@pisum](https://forum.inaturalist.org/u/pisum)\ **Post date:** [December 27, 2020, 6:48pm UTC](https://forum.inaturalist.org/t/many-ids-have-id-category-null/19061/3 "2020-12-27T18:48:46Z")
> [@astra\_the\_dragon](#): > > Maybe withdrawn IDs no, in the examples / analysis referenced above, we’re dealing with current IDs. so whatever happened is unlikely to be related to withdrawing IDs. ---
**Author:** ![pisum](https://avatars.discourse-cdn.com/v4/letter/p/91b2a8/32.png) [@pisum](https://forum.inaturalist.org/u/pisum)\ **Post date:** [December 28, 2020, 2:32am UTC](https://forum.inaturalist.org/t/many-ids-have-id-category-null/19061/4 "2020-12-28T02:32:48Z")
not sure how useful this might be, but i went ahead and made a page to summarize identifications by category (including an “Other”, or uncategorized, category). it could provide a way for folks without direct access to the database to quickly get these stats for different sets of parameters. page: [https://jumear.github.io/stirfry/iNat\_id\_summary\_by\_category.html](https://jumear.github.io/stirfry/iNat_id_summary_by_category.html) code: [https://github.com/jumear/stirfry/blob/gh-pages/iNat\_id\_summary\_by\_category.html](https://github.com/jumear/stirfry/blob/gh-pages/iNat_id_summary_by_category.html) example usage: [https://jumear.github.io/stirfry/iNat\_id\_summary\_by\_category.html?own\_observation=false&user\_id=bouteloua&d1=2020-01-01&d2=2020-12-31](https://jumear.github.io/stirfry/iNat_id_summary_by_category.html?own_observation=false&user_id=bouteloua&d1=2020-01-01&d2=2020-12-31) i might add a pie chart or something like that later… ---
**Author:** ![michaelpirrello](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.inaturalist.org/michaelpirrello/32/1039_2.png) [@michaelpirrello](https://forum.inaturalist.org/u/michaelpirrello)\ **Post date:** [December 28, 2020, 8:02pm UTC](https://forum.inaturalist.org/t/many-ids-have-id-category-null/19061/5 "2020-12-28T20:02:30Z")
So I tried that URL and got 77 category Other identifications (0.4%). I tried it again with a much earlier start date and got the same number, suggesting that they were all in 2020. Is anyone seeing any in this category from previous years? ---
**Author:** ![pisum](https://avatars.discourse-cdn.com/v4/letter/p/91b2a8/32.png) [@pisum](https://forum.inaturalist.org/u/pisum)\ **Post date:** [December 28, 2020, 8:09pm UTC](https://forum.inaturalist.org/t/many-ids-have-id-category-null/19061/6 "2020-12-28T20:09:00Z")
yes, there definitely are a few cases from previous years ([https://jumear.github.io/stirfry/iNat\_id\_summary\_by\_category.html?own\_observation=false&d2=2019-12-31](https://jumear.github.io/stirfry/iNat_id_summary_by_category.html?own_observation=false&d2=2019-12-31)), but the bulk of the cases are definitely from 2020 ([https://jumear.github.io/stirfry/iNat\_id\_summary\_by\_category.html?own\_observation=false&d1=2020-01-01](https://jumear.github.io/stirfry/iNat_id_summary_by_category.html?own_observation=false&d1=2020-01-01)), and Q1 2020 in particular ([https://jumear.github.io/stirfry/iNat\_id\_summary\_by\_category.html?own\_observation=false&d1=2020-01-01&d2=2020-03-31](https://jumear.github.io/stirfry/iNat_id_summary_by_category.html?own_observation=false&d1=2020-01-01&d2=2020-03-31)). are you doing some additional investigation, or are you just trying to understand the issue better? ---
**Author:** ![astra\_the\_dragon](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.inaturalist.org/astra_the_dragon/32/37251_2.png) [@astra\_the\_dragon](https://forum.inaturalist.org/u/astra_the_dragon)\ **Post date:** [December 28, 2020, 8:19pm UTC](https://forum.inaturalist.org/t/many-ids-have-id-category-null/19061/7 "2020-12-28T20:19:38Z")
I’ve got a full 2.2% Other. I wonder what the difference is between us that I’ve got proportionately so many more… and 65 maverick! time to go fix those… ---
**Author:** ![pisum](https://avatars.discourse-cdn.com/v4/letter/p/91b2a8/32.png) [@pisum](https://forum.inaturalist.org/u/pisum)\ **Post date:** [December 28, 2020, 8:24pm UTC](https://forum.inaturalist.org/t/many-ids-have-id-category-null/19061/8 "2020-12-28T20:24:49Z")
> [@astra\_the\_dragon](#): > > proportionately so many more… overall, the uncategorized identifications for others make up 6.4% of identifications for others in 2020. so i don’t think your 2.2% is above average in that respect, however, during Q1 2020, 54.5% of your identifications for others were uncategorized ([https://jumear.github.io/stirfry/iNat\_id\_summary\_by\_category.html?own\_observation=false&d1=2020-01-01&d2=2020-03-31&user\_id=trh\_blue](https://jumear.github.io/stirfry/iNat_id_summary_by_category.html?own_observation=false&d1=2020-01-01&d2=2020-03-31&user_id=trh_blue)), which is quite bit more than average. … for what it’s worth, it looks like there was some experimentation with how identifications trigger observation reindexing early in 2020 (see [https://github.com/inaturalist/inaturalist/commits/main/app/models/identification.rb](https://github.com/inaturalist/inaturalist/commits/main/app/models/identification.rb)), which may coincide with the appearance of most of these uncategorized identifications. probably what needed to be fixed got fixed, but maybe doing a lookback to see why these changes may have triggered the problem might be warranted (just to understand what not to do in the future). i suppose the data should be cleaned up at some point, too (which i guess probably is the ultimate fix for this bug report at this point). ---
**Author:** ![michaelpirrello](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.inaturalist.org/michaelpirrello/32/1039_2.png) [@michaelpirrello](https://forum.inaturalist.org/u/michaelpirrello)\ **Post date:** [December 29, 2020, 12:35am UTC](https://forum.inaturalist.org/t/many-ids-have-id-category-null/19061/9 "2020-12-29T00:35:41Z")
Is cleaning up that data something we can do on our own? And if so, how? ---
**Author:** ![bouteloua](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.inaturalist.org/bouteloua/32/1180_2.png) [@bouteloua](https://forum.inaturalist.org/u/bouteloua)\ **Post date:** [December 29, 2020, 12:40am UTC](https://forum.inaturalist.org/t/many-ids-have-id-category-null/19061/10 "2020-12-29T00:40:35Z")
There are around 700 from just this past week, so it would probably be something staff should look into rather than an ongoing user clean-up effort. ---
**Author:** ![pisum](https://avatars.discourse-cdn.com/v4/letter/p/91b2a8/32.png) [@pisum](https://forum.inaturalist.org/u/pisum)\ **Post date:** [December 29, 2020, 2:40am UTC](https://forum.inaturalist.org/t/many-ids-have-id-category-null/19061/11 "2020-12-29T02:40:20Z")
> [@michaelpirrello](#): > > Is cleaning up that data something we can do on our own? just looking at the code history, it looks like changing anything on the observation – annotations, identifications, quality flags, etc. should trigger recategorization of all the identifications. i tested this by adding a flowering annotation on an observation where my id was previously uncategorized, and the identification was categorized properly after the fact. observation before and after: ![tbdscreenshot1a](https://us1.discourse-cdn.com/flex016/uploads/inaturalist/original/2X/3/3165558bd6344003e74dd023da715148d2c1c224.jpeg) ![tbdscreenshot1b](https://us1.discourse-cdn.com/flex016/uploads/inaturalist/original/2X/1/17000768a1d3a174c879f0f84a97d188eb1e2781.jpeg) api response before and after: ![tbdscreenshot2a](https://us1.discourse-cdn.com/flex016/uploads/inaturalist/original/2X/9/99a257909e3801ffdbf2bc1362e42292a01a8f9b.jpeg) ![tbdscreenshot2b](https://us1.discourse-cdn.com/flex016/uploads/inaturalist/original/2X/6/61f8a4d80640138a3c9ccaaefff768f10e30831b.jpeg) that said, doing this for the millions of affected observations seems inefficient at best. so i agree that: > [@bouteloua](#): > > it would probably be something staff should look into rather than an ongoing user clean-up effort