Hi Matthias,
These are super good questions. We haven’t done this kind of analysis, but it’s an excellent idea and I’ll add it to my list. We have been asking ourselves “what is the best way to produce a computer vision dataset from the iNaturalist community created dataset” and this is exactly the kind of question that will point us to better answers. So thanks!
Unfortunately we only train new models twice a year, and we regularly change how we export the data to our training system in order to either improve accuracy, add more taxa, or decrease training time, so not all our historical models are comparable in this way. We also don’t do very much additional computer vision experimentation because it’s simply too expensive for us.
However, I believe I have enough data to do compare our current production model with the previous production model, since they were both trained with the same database export rules. And for sure we’ll add it to the analysis we do of new models after they’re trained, where we vet them to make sure they’re ready to be released to the community.
I can’t promise when I’ll be able to report results, but when I do I’ll share them here in the forum and @mention you, @matthias55!
Thanks,
alex