Audio recordings of bird song/calls

[cthawley]

Given the way that the iNat CV model works, this wouldn’t really be possible. It is certainly possible to train a machine learning model to recognize spectrograms (this is what Merlin does for instance). But that model is a separate one from their photo ID model. Both are trained on their specific class of inputs.

iNat’s model doesn’t know that the picture uploaded is a spectrogram - it will be treated the same as other pictures of the organism. So adding spectrograms will introduce unnecessary error into model training. An additional issue is that spectrograms would need to be produced in a consistent manner to allow for comparisons (again, this is what Merlin et al. do). Using unstandardized spectrograms, even in a spectrogram specific model, would lead to poor results.

If you want a more “official” take on spectrograms:

@kueda

Observation photos are intended and assumed to be photographic evidence for the recent presence of an organism, i.e. they should communicate what you saw in the field. Not spectrograms, not habitat shots, not pictures of the sky to show what the weather was like, not photos of photos, just actual photos that show someone what you saw, and hopefully look like what others might see when seeing similar evidence for the recent presence of the same taxon. We make that assumption when showing observations photos on the taxon page, when training our computer vision system, when sharing data with partners like GBIF, etc., and all those non-organism shots break that assumption and cause us to use and share inaccurate information (we claim something is a photo of an organism when it’s actually a spectrogram). If at some point we support some way to categorize observation photos or support some other form of ancillary photographic material to be attached to an obs, then that stuff would be ok, but at present we don’t. I realize tracks & signs screw that up and I admit my tolerance for them is a lot higher than it is for spectrograms, but I think that’s b/c they at least show something unique about the organism that helps others learn to recognize it in person (“but what about microscopy” etc etc). Spectrograms are great evidence and really interesting (as are habitat shots, microscopy, most of the other kinds of images that people upload as obs photos), but if we’re not going to distinguish them from photos of organisms then I don’t think people should upload them. Maybe post them elsewhere and embed them in the description or a comment or something.