Real-time Computer Vision predictions in Seek by iNaturalist 2.0

[tiwane]

We’re excited to announce the release of a new version of our Seek by iNaturalist app, available now for iOS and Android! You can read more about it in the blog post. Just like iNat’s computer vision suggestions, Seek would not be possible without the contributions of our incredible community of observers and identifiers.

In this post, we’ll try to answer some of the questions we expect the iNaturalist community will have.

How is Seek different than iNaturalist?

• It is not a social network; no accounts are created, and no observations are publicly shared.
• It only uses computer vision results for identifications rather than computer vision + crowdsourcing.
• You can earn badges by observing different species and by completing challenges.
• It’s not contributing to citizen science (because it doesn’t share data).
• It’s kid-friendly (because it doesn’t share data).

How does the Seek camera work (technically)?

When you use the camera in Seek, our vision model analyzes the frames of the camera feed to provide live ID suggestions as you point the camera at an organism. Unlike the iNaturalist online-only vision model, the Seek camera uses an on-device computer vision model which allows the in-camera predictions to give you instant feedback, including the level of specificity of its current prediction. Here’s a preview video:

In addition to being byte-wise smaller than our online model so it will fit on mobile devices, this new model was trained to include more photos representing more species. The model includes species as well as non-species “leaf nodes” in cases where individual species do not have enough photos to train on, but their collective genus does. We hope this will help Seek identify more organisms, if sometimes to a rank higher than species.

Will it work on my device?

You can download Seek on iPhones and iPads running iOS 9.0 or higher, but you must have iOS 11.0 or higher to see the in-camera predictions. For Android devices, you need 6.0 or higher. Note that older devices may experience much slower performance of the in-camera predictions.

Why might I get different suggestions from the same photo on Seek and iNaturalist?

The model we trained for Seek isn’t implemented yet for iNaturalist. We are still learning a lot about how to get the highest success and lowest error rate, and the use case for iNaturalist and Seek are different enough that the two may not ever have precisely identical results.

What else should I know about using Seek if I’m used to using iNaturalist?

• Photos taken in the Seek iOS app won’t have geotags, so be aware of that if you’re planning to use the photos later for iNaturalist.
• Your badges are stored only on your device, and there’s no backup (because there are no Seek accounts). If you delete the app, you lose your badges.

Who made it?

The first version of Seek (supported by HHMI Tangled Bank Studios) was initially iOS only, with @alexshepard and @joelle doing the heavy lifting on programing and design (respectively). Thanks to the additional support from WWF/Our Planet, @albullington joined the team as the primary engineer and @abhasm as designer and product manager. The computer vision innovations are thanks to @gvanhorn with support from @alexshepard, @pleary, and @loarie. @budowski and @kueda helped with the Android implementation and @alexshepard helped on iOS. @tiwane and @carrieseltzer did whatever was needed that didn’t require programming or design (like this post). Our contacts at WWF/Our Planet @mlarsendaw and @noah51 designed the challenges and coordinated the translations. Everyone contributed to its evolution and provided feedback.

Please write reviews if you can!

Once you’ve taken Seek for a spin, we’d appreciate if you would rate the app and share a brief review that describes where you used it and what worked (or didn’t). You can see that earlier app store reviews have some feedback that isn’t very specific, or some expectations for its performance that weren’t met (e.g. common houseplants). We’re trying to do a better job of messaging Seek’s intended use case (e.g. it’s not for breeds of dogs or cultivars of daffodils), and app reviews from the iNaturalist community would be valuable in steering potential Seekers in the right direction when they download it. After all, Seek only works because of you, the iNaturalist community. Thank you!

[carnifex]

Awesome! When receiving the notification, I immediately went out to give it a try. The augmented reality mode is really great.
Here some things I noticed:

• There is no focus option within the integrated camera, so I struggled to get small flowers or insects sitting on a branch to get detected.
• The import module should be improved. I would like to provide Seek with photos I took with my mobile camera, but choosing a photo includes all the other images from WhatsApp etc. as well. Those should be excluded or at least there should be an option from which source the images can be taken
• I am using the English version, but I saw a friend using the German translation and a major glitch I saw was that the taxonomic level ‘Genus’ was translated as ‘Geschlecht’ - which means ‘sex’. It should be ‘Genus’ in German as well. Also, upon successful identification there was only the scientific name displayed, although a German name exists, and under ‘My Observations’ most of it is English (including the wikipedia stuff)

Using Android, if this is important.

Looking forward to testing it out more thoroughly,
cheers, Lorin

[carrieseltzer]

Thanks @carnifex! We expect to be quickly fixing some known bugs and we’ll add the German issues to that list.

[charlie]

ohmygosh this is so cool. I am now imagining a form of inaturalist like this but oh boy, i’m gonna add every individual tree along a hike and annoy everyone (maybe just kidding, kinda? :) )

[carnifex]

Forgot one issue: looking at a specific species badge under ‘Achievements’, it is difficult to return to the overview - the back arrow is not working on my mobile in that window. One has to click that small space between the pop-up and the window screen edge

[carrieseltzer]

I’ll add we know about the bug where if you choose a photo from your camera roll it incorrectly says you’ve re-sighted the species, so no need to report that one. A fix for that will be in the next update soon (which we couldn’t queue up until this was released).

[nathantaylor]

So, is it possible that we could use this to help fill holes in the computer recognition by targeting areas or angles of the organism where it is identifiable to us but not by the computer? Regardless, it’s still pretty amazing!

[mutolisp]

I tested Seek for a while (testing platform: iOS 12.x, locale: traditional Chinese [Taiwan]). Here are some problems/questions I noticed:

• When showing result of an identified organism, the common name is English. Could it show localized common name based on user’s locale?

• In the block of “About”. The system automatically fetch the content of Wikipedia, but it shows simplified Chinese version. Could it be adjusted to traditional Chinese (Taiwan) version?

• Is there any translation interface just like iNaturalist mobile on crowdin?

thanks!!

Cheng-Tao

[optilete]

Great idea, localized common names :-).

[joemiller]

This is great, I would love to see this technology applied to wildlife camera trap data analysis. Is there such a project happening somewhere?

[tiwane]

@mutolisp

Is there any translation interface just like iNaturalist mobile on crowdin?

Yes, we plan to open it up to Crowdin translation, hopefully soon, and integrate better common name support. As for Wikipeda, I’ll ask.

[ahospers]

@joemiller

This is great, I would love to see this technology applied to wildlife camera trap data analysis. Is there such a project happening somewhere

Wild trap analysis ? I think in Holland they are working very hard on it, so i was very surprised reading this forum to see that is actually exists. I only know the were working on computervison in stead of determine species from photos to determine the species from film. The will use it to estimate the degreadation of insect life in nature reservers in Holland and Germany.
And in such a way it can also be used to understand the degratdation of biolife with migratory birds and weidevogels (Plovers, Grutto (limosa limosa), Kievit (Vanellus vanellus)

[carnifex]

In addition to swipe to see badge requirements for a specific taxon, I’d recommend clicking on the bronze/silver/gold/empty icon at the bottom also to show the requirements. Earned a silver badge for plants, I thought it was a bug that it still showed bronze. Just figured out one has to swipe to the left instead of clicking on the ‘>’ arrow

[danaleeling]

@tiwane

Once you’ve taken Seek for a spin, we’d appreciate if you would rate the app and share a brief review that describes where you used it and what worked (or didn’t).

What might be most useful in an educational application of Seek would be a setting that would allow the user to enter an identification. In a class situation the teacher might be/should be assisting students with identifications. When the app simply drops through to “Dicot” or family level ID, the option to enter a species level identification would be helpful. Perhaps a dropdown species appearing as in iNaturalist when one starts typing a genus and species. This should not be a default behavior, the app should remain simple for elementary school children. The option to specify species when not identified by the computer vision should be a setting tucked away on a settings menu. Getting “duress” secondary school users off of iNaturalist will likely require having the option to specify the genus and species. When a commonly cultivated plant such as Jasminum sambac is left at the dicot level, that feels problematic and seems likely to drive a secondary school instructor back to iNaturalist where they can have the students specify the plant. I look forward to the evolution of Seek as a way to help those of us who teach.

[danaleeling]

@tiwane

Once you’ve taken Seek for a spin, we’d appreciate if you would rate the app and share a brief review that describes where you used it and what worked (or didn’t).

An example of what did not work: “the first cultivated plant on the right after going out the door.”

This plant is likely to be Baleria cristata, a cultivated plant with a somewhat distinctive look to the flower, a range from India to the Pacific, but Seek could not get the plant past Magnoliopsida. I know the instructions were to put a review in the app store, but I would rather share what does not work in the more limited reach of this forum. Too, I am fully aware that distinctions below the species level such as

@tiwane

cultivars of daffodils

are not included. That said, too many common garden plants - at least around here - wind up at the dicot level. I will continue to take Seek out for the occasional spin and once the computer vision gets better at the kinds of plants exuberant young students are likely to image, I will be recommending the app to teachers here. Thanks for the great work, I look forward to better incarnations of Seek and the possibility to provide genus or species level hints to the app from the keyboard.

[mtank]

I have noticed difficulty in identifying organisms as well, but that is to be expected, given the higher confidence that should be required to prevent misidentifications.

One thing that I think is resolvable though is when my camera is clearly pointed at something borderline identifiable. I notice the ID will swing back and forth from genus to species, and when I try to time my capture to get the species ID, I’m not always successful (due to lag in the realtime ID I assume).

I think you could introduce a tolerance into the realtime camera feature, so that it will look at the submitted image, compare it to the last second or two of previous ID data for that camera, and assuming the image is roughly of the same thing (ie, you haven’t quickly moved your camera from your dog to your cat in the meantime), it takes the most granular ID. This would mean even if the ID dropped slightly below the species threshold just as you pressed the button, it would still ID the organism as what the app was showing as the ID at the time you pressed the button.

As it is, I get to the point where it IDs to species level, then just continually click and hope

Zooming might also be helpful. While it won’t improve the quality of your subject in the image, it will help to add a boundary to what the CV classifies on (ie, the bird in the tree, and not the tree)

[charlie]

i think one of the biggest issues here is still that the algorithm isn’t trained as much on cultivated plants but those are what newbies go for first. That being said i’ve had good results with it for cultivated plants and even houseplants.

[bobby23]

Here’s the review I wrote for Seek:

Seek is a very promising extension of iNaturalist and has a lot of potential as a family-friendly game. The Badge System is pretty cool, but it’s unclear if there is any long-term “goal” or incentive to getting higher badge rankings other than bragging rights. I think this is ultimately what will hinder long-term use of the app.

The interface is great though - much better than the actual iNaturalist app and more aesthetically pleasing. It feels like Seek can serve as an actual learning tool, and I sincerely hope that the core iNaturalist app adopts some of its concepts. In terms of how accurate it is… well, it’s limited and not always 100% accurate (it recognized my Goat observation as a Dog) but it’s also as complex as it needs to be.

My biggest grievance is that when a genus lacks a Common Name, it doesn’t show up in a species’s taxonomy. It‘s a little jarring.

^Note that Ursus is absent.

[andypatrick]

I found these apps today and I’m LOVING them! I love how easy Seek is to use and how accurate it’s been. My whole days has been going around my yard and through old photos to ID things I didn’t know. So thanks!

My request would be this:

If Seek ID’s something, it only seems to be in the Seek app and not my iNat account. If Seek doesn’t ID it I can upload it but then I get things ID’s over on iNat and now I’ve got two running lists.

I would love to be able to push all my ID’d items on Seek over to iNat as well. I don’t want to go through the trouble of uploading them twice. A “send to iNat” option would be awesome (individually or via a “select” button where I could do a bunch at once).

Thanks!