New Vision Model Training Started

@alex just published a new blog post about the latest vision model, which we just started training. Check it out!

26 Likes

I’m excited to see how it performs! I think my favorite little Macrotera bees finally have enough RG observations as a genus to be eligible for the model.

5 Likes

This time around we went from 38,000 to 47,000 taxa

That’s a big jump. Any idea of the breakdown between additional species vs other taxonomic ranks that are being included this time around? The announcement on the July 2021 model didn’t mention it, but the announcement for the March 2020 model did and it had ~21000 species and ~2500 genera. How many species & genera are there now and how many will be in the forthcoming iteration?

Also, the “leaf model” terminology is confusing, given that a species may or may not be a leaf node and genera are never leaf nodes.

3 Likes

In the tree that our vision system sees, genera can be leaves if we don’t have enough photos to train any of the child species but we do have enough photos to train at the genus level.

5 Likes

So genera are booted from the model as soon as a single child species is separately included in the model?

2 Likes

Yep, Genus Macrotera will be in the model this time around (as a leaf node, so no children in the vision model). It looks like we have just over 350 training images for the genus.

Also, with a name like goblin bees, maybe my new favorite, too!

3 Likes

@alex have you played around with any solutions for recordings? At one time, I looked into using computer vision to ID bird sounds by comparing audiograms, though I didn’t get very far along that path.

Unless something has changed, this is correct.

It’s honestly a big problem in many insects (and probably other groups), where a single species in a genus (or even family) is common enough and easily enough identified from photos to be included in the CV model, whereas there are tons of genus-level observations (and a few ID’d as other species). In that case, none of the other photos make it into the training set, and the AI happily, and erroneously, suggests that anything that looks similar must be the one distinctive species.

9 Likes

Which causes the feedback loop identifiers are always complaining about; incorrectly IDed species → CV thinks all similar species are that one → more incorrect IDs → more incorrect CV… etc etc.

2 Likes

I’ve seen CV-IDs that make me suspect this too.

If the model only works with leaf-nodes in a tree, then it would make more sense a pseudo-genus to bin all of the observations for species that don’t have sufficient numbers of observations or photos. If the model can work with branch nodes, then those should be included.

2 Likes

I have thought the same thing! I spoke with iNat staff about this issue, and they said that the way the CV is currently set up, it can’t consider groups that are nested within each other. Furthermore, adding all these extra groups (whether you do branches or pseudo-leaves) will dramatically increase the complexity of the model and lengthen the training time. What levels would you include: genera, families, classes, orders? Subfamilies, tribes, subtribes, subgenera? I think ideally you would include as many branch taxa as you could, but just duplicating all eligible genera (that is, using at least one leaf for the species in the genus and then a second pseudo-leaf for the remaining observations) would be a lot! I can appreciate that concern, since the training set is already growing very quickly on its own.

3 Likes

Any chance that in future learning algorithms the community disagreements will be included, so that the overconfident CV suggestions for difficult taxa will be mitigated?

1 Like

In this case, if there exist additional species within, say a genus, that don’t have enough observations to enter the model, the most conservative and (imo) best approach would be to include only the genus in the model, and not the one species that does have enough images. We don’t need the model to always suggest species; we need it to be right as often as possible with what it does suggest.

6 Likes

I bet that would be largely equivalent to not using the species level at all. I think you will find at least one species with very few or even no observations in most genera. So it’s not just a somewhat more conservative approach, it’s an entirely different one. (I’m not necessarily saying worse though, haven’t really thought about it.)

1 Like

I do think that if the CV only has enough training for one species within a genus, ID suggestions should be at genus. Two or more, at species. With an exception for genera with only one species. That may require a whole different approach, or it may be something that can be tacked on right at the end. Idk.

7 Likes

Ah, right, I missed the reference to ‘genus with one species with enough observations’ above.

I’m still not sure if this would always be a good idea. The reasons for only having one species with enough observations could be very different. For example

a) Only one species in the genus can be identified with images alone. All the others of which serveral are very frequent can’t.

b) The vast majority of observations belong to that one species, all others in the same genus are super-rare.

In case a) only training gainst genus would be very useful, I doubt many would disagree on that. In case b) you would give less information for avoiding a rare mistake, if that is a good idea is much more open for debate.

1 Like

This could be a criterion. If more than some threshold of observations within a genus are identified as one species (e.g. 50% of total, or 90% of RG) then it might be reasonable to include that species. But if there are 200 observations of species A, and 2000 others only identified to genus, then it would be better to include the genus in the model.

7 Likes

I agree - I also would have thought it would be significantly better if the CV model would only be trained on species where there are sufficient obs to cover at least two members of the genus.

I could imagine if this was one of the basic criteria ( alongside 50 RG and 100 obs), we would see a very positive shift in the type of work identifiers have to do in complex taxa.

2 Likes

Presumably monotypic genera would get their single species included too? But how would iNat determine whether a genus is truly monotypic, or only has had one species imported into the iNat taxonomy? Maybe that’s enough of an edge case that it wouldn’t be an issue. (or, at least would be less of an issue than the one we’ve got now!)

3 Likes

Not complaining, but can you please, please, please make it country specific!
The “old” one is an amazing bit of tech, but in Australia we are constantly having to lift id’s out of US species that don’t occur here. Which of course annoys the hell out of newbies (the last thing we want to do).
If there was just a filter that said “recorded in Australia” at the top level, it’d become really useful here; rather than being a right pain in the proverbial.
Perhaps as part of it’s training you can monitor how often the image match is agreed to/disagreed to by others?
Cheers
Brett