This is a known issue, see https://forum.inaturalist.org/t/community-id-display-confusing-neutral-ids-display-as-disagreement/12744 and https://forum.inaturalist.org/t/withdrawn-ids-still-counting-against-community-taxon/15219.
There’s also already a github issue for it, so I’m going to close this thread to keep discussion focused at the open bug report.