Use computer vision on each photo in an observation

Currently, the CV only suggests species for the first image in an observation. It would be useful to be able to get suggestions on any image in an observation. There are ways to achieve this in the current system, but they are cumbersome and only work on a user’s own observations, not those of other people. Intuitively I want this to happen when I scroll to the next image in the viewer, though there might be better ways to implement it.

I agree that would be useful. But shouldn’t the computer vision suggestions be on the whole set of photos anyway, it doesn’t make sense for the system to only look at one photo when suggesting an ID for the observation.

1 Like

I’m no programmer, but that seems a lot harder to implement. The computer vision is not programmed to identify organisms, rather pictures of organisms. It seems unlikely it would be able to collate information from different pictures the way a human can (i.e. it can’t really think along the lines of the leaf is shaped this way, the flower is shaped that way, therefore…).

Being able to choose what picture to invoke the computer recognition on, though, would be invaluable.

2 Likes

I agree. Asking the CV to weight and combine its guesses for multiple pictures in an observation seems to cross the line from helpful tool to scientific judgment. Integrating those guesses and sorting them by overall plausibility is the job of human experts, at least for now.

1 Like

Determining the common ancestor of the taxa recognized in different pictures from the same observation sounds like a straightforward proposition though, unless I’m missing something.

2 Likes

I agree, either that or the original image-by-image request should be straightforward. The crux, of course, will be whether the added computational load will be feasible.

1 Like

From my non-technical perspective: the ability to choose which photo to run the model on is probably doable, but use multiple photos at once to determine a suggestion is probably not. Or at least would not be doable in the near future.

2 Likes

I would have thought just process each image and take a majority vote, and default to the first image if indeterminite (although as noted above, a high computational load). No agreement between images might suggest going for a higher taxon, or less confident suggestion.

But being able to select an image would work fine on the website version. Combining that with the ability to draw a square/circle around the element that is the focus of the observation would be a bonus (that’s another feature request too I think).

1 Like