Real-time Computer Vision predictions in Seek by iNaturalist 2.0

seek
update
#1

We’re excited to announce the release of a new version of our Seek by iNaturalist app, available now for iOS and Android! You can read more about it in the blog post. Just like iNat’s computer vision suggestions, Seek would not be possible without the contributions of our incredible community of observers and identifiers.

In this post, we’ll try to answer some of the questions we expect the iNaturalist community will have.

How is Seek different than iNaturalist?

  • It is not a social network; no accounts are created, and no observations are publicly shared.
  • It only uses computer vision results for identifications rather than computer vision + crowdsourcing.
  • You can earn badges by observing different species and by completing challenges.
  • It’s not contributing to citizen science (because it doesn’t share data).
  • It’s kid-friendly (because it doesn’t share data).

How does the Seek camera work (technically)?

When you use the camera in Seek, our vision model analyzes the frames of the camera feed to provide live ID suggestions as you point the camera at an organism. Unlike the iNaturalist online-only vision model, the Seek camera uses an on-device computer vision model which allows the in-camera predictions to give you instant feedback, including the level of specificity of its current prediction. Here’s a preview video:

In addition to being byte-wise smaller than our online model so it will fit on mobile devices, this new model was trained to include more photos representing more species. The model includes species as well as non-species “leaf nodes” in cases where individual species do not have enough photos to train on, but their collective genus does. We hope this will help Seek identify more organisms, if sometimes to a rank higher than species.

Will it work on my device?

You can download Seek on iPhones and iPads running iOS 9.0 or higher, but you must have iOS 11.0 or higher to see the in-camera predictions. For Android devices, you need 6.0 or higher. Note that older devices may experience much slower performance of the in-camera predictions.

Why might I get different suggestions from the same photo on Seek and iNaturalist?

The model we trained for Seek isn’t implemented yet for iNaturalist. We are still learning a lot about how to get the highest success and lowest error rate, and the use case for iNaturalist and Seek are different enough that the two may not ever have precisely identical results.

What else should I know about using Seek if I’m used to using iNaturalist?

  • Photos taken in the Seek iOS app won’t have geotags, so be aware of that if you’re planning to use the photos later for iNaturalist.
  • Your badges are stored only on your device, and there’s no backup (because there are no Seek accounts). If you delete the app, you lose your badges.

Who made it?

The first version of Seek (supported by HHMI Tangled Bank Studios) was initially iOS only, with @alexshepard and @joelle doing the heavy lifting on programing and design (respectively). Thanks to the additional support from WWF/Our Planet, @albullington joined the team as the primary engineer and @abhasm as designer and product manager. The computer vision innovations are thanks to @gvanhorn with support from @alexshepard, @pleary, and @loarie. @budowski and @kueda helped with the Android implementation and @alexshepard helped on iOS. @tiwane and @carrieseltzer did whatever was needed that didn’t require programming or design (like this post). Our contacts at WWF/Our Planet @mlarsendaw and @noah51 designed the challenges and coordinated the translations. Everyone contributed to its evolution and provided feedback.

Please write reviews if you can!

Once you’ve taken Seek for a spin, we’d appreciate if you would rate the app and share a brief review that describes where you used it and what worked (or didn’t). You can see that earlier app store reviews have some feedback that isn’t very specific, or some expectations for its performance that weren’t met (e.g. common houseplants). We’re trying to do a better job of messaging Seek’s intended use case (e.g. it’s not for breeds of dogs or cultivars of daffodils), and app reviews from the iNaturalist community would be valuable in steering potential Seekers in the right direction when they download it. After all, Seek only works because of you, the iNaturalist community. Thank you!

12 Likes

The category of "cultivated" is problematic for plants in urban landscapes
Importing photo from Seek to iNaturalist doesn't have location
#2

Awesome! When receiving the notification, I immediately went out to give it a try. The augmented reality mode is really great.
Here some things I noticed:

  • There is no focus option within the integrated camera, so I struggled to get small flowers or insects sitting on a branch to get detected.
  • The import module should be improved. I would like to provide Seek with photos I took with my mobile camera, but choosing a photo includes all the other images from WhatsApp etc. as well. Those should be excluded or at least there should be an option from which source the images can be taken
  • I am using the English version, but I saw a friend using the German translation and a major glitch I saw was that the taxonomic level ‘Genus’ was translated as ‘Geschlecht’ - which means ‘sex’. It should be ‘Genus’ in German as well. Also, upon successful identification there was only the scientific name displayed, although a German name exists, and under ‘My Observations’ most of it is English (including the wikipedia stuff)

Using Android, if this is important.

Looking forward to testing it out more thoroughly,
cheers, Lorin

2 Likes

#3

Thanks @carnifex! We expect to be quickly fixing some known bugs and we’ll add the German issues to that list.

1 Like

#4

ohmygosh this is so cool. I am now imagining a form of inaturalist like this but oh boy, i’m gonna add every individual tree along a hike and annoy everyone (maybe just kidding, kinda? :) )

3 Likes

#5

Forgot one issue: looking at a specific species badge under ‘Achievements’, it is difficult to return to the overview - the back arrow is not working on my mobile in that window. One has to click that small space between the pop-up and the window screen edge

2 Likes

#6

I’ll add we know about the bug where if you choose a photo from your camera roll it incorrectly says you’ve re-sighted the species, so no need to report that one. A fix for that will be in the next update soon (which we couldn’t queue up until this was released).

0 Likes

#7

So, is it possible that we could use this to help fill holes in the computer recognition by targeting areas or angles of the organism where it is identifiable to us but not by the computer? Regardless, it’s still pretty amazing!

1 Like

#8

I tested Seek for a while (testing platform: iOS 12.x, locale: traditional Chinese [Taiwan]). Here are some problems/questions I noticed:

  • When showing result of an identified organism, the common name is English. Could it show localized common name based on user’s locale?

  • In the block of “About”. The system automatically fetch the content of Wikipedia, but it shows simplified Chinese version. Could it be adjusted to traditional Chinese (Taiwan) version?

  • Is there any translation interface just like iNaturalist mobile on crowdin?

thanks!!

Cheng-Tao

3 Likes

split this topic #9

4 posts were split to a new topic: Seek vs. Questagame

0 Likes

#10

Great idea, localized common names :-).

0 Likes

#11

This is great, I would love to see this technology applied to wildlife camera trap data analysis. Is there such a project happening somewhere?

0 Likes

#12

Yes, we plan to open it up to Crowdin translation, hopefully soon, and integrate better common name support. As for Wikipeda, I’ll ask.

3 Likes

#13

Wild trap analysis ? I think in Holland they are working very hard on it, so i was very surprised reading this forum to see that is actually exists. I only know the were working on computervison in stead of determine species from photos to determine the species from film. The will use it to estimate the degreadation of insect life in nature reservers in Holland and Germany.
And in such a way it can also be used to understand the degratdation of biolife with migratory birds and weidevogels (Plovers, Grutto (limosa limosa), Kievit (Vanellus vanellus)

1 Like

#14

In addition to swipe to see badge requirements for a specific taxon, I’d recommend clicking on the bronze/silver/gold/empty icon at the bottom also to show the requirements. Earned a silver badge for plants, I thought it was a bug that it still showed bronze. Just figured out one has to swipe to the left instead of clicking on the ‘>’ arrow

1 Like