I’m not sure I understand the algorithm that you’re proposing, but if you can share pseudocode or a proof of concept, I’d love to read more.
Also, I mentioned in the blog post linked above that we plan to continue to look at new approaches to including or excluding nodes in the taxonomy. So we’re on the same page with you, we see a problem here and we want to tackle it. I’m really focused on two other projects right now (speeding up training / releasing models more often, and new approaches to geofrequency) but once I have some time, I want to do some evaluation in this area.
I hope you can understand that we’re trying to balance a few competing priorities here: we are excited to have more and more taxa represented in the models and we would like to see predictions made for those taxa, but we’re also disappointed when the model makes mistakes on taxa that are in the model and we wish it was more conservative in those cases.
In the past Cassi made this great thread https://forum.inaturalist.org/t/computer-vision-clean-up-archive/7281 and it would be great to have something like this as a starting point for finding taxa the community thinks the vision model struggles with.