Hello - does anyone have any ideas for how to find observations from doorbell/security cameras? I recently started a ‘traditional’ project to try and collect these observations: https://www.inaturalist.org/projects/wildlife-on-home-cameras
So far, I’ve separately queried for observation descriptions that contain “doorbell” and “ring camera”. There are a few more searches like this I will do. Any other ideas/suggestions would be appreciated!
Maybe search specifically for Obscured or Private observations, since people often do that for observations made at home? Depending on your exact goals, I guess the Private ones may not be of as much interest. You could also filter out observations in the Never Home Alone project, since those have to be inside a house or other building
Maybe restrict your searches to vertebrates (I assume small stuff and plants are unlikely to be captured on these cameras) or try searching within a restricted urban geographical area (like Paris or Miami).
These are all off the cuff ideas that I haven’t looked into, but I think are promising.
I would try to see if you can recover exif data from the images first. Camera manufacturer is pretty standard (e.g., iphone, android, Nikon, etc), and will allow you immediately filter out a huge percentage of the possible observations.
Next I would do a bit of GIS work and look for observations that occur near a road. This won’t help you for obscured observations, but by virtue of being a doorbell camera, it should be near a house, and therefore a road. Openstreetmap is where I would look first for road vectors.
You can also filter by taxa of course. Looking at your project, it is largely mammals and birds. That will narrow the total pool down quite a bit as well. No need to look at plants or marine life.
You can use cheap LLMs to surface observations that have notes which seem promising, instead of simple word matches. If you hate LLMs, you can take an embedding approach and calculate similarity then manually review. Simply filtering out blank notes would remove most anyway.
Another LLM approach is to pass the image to another small and cheap vision enabled model and ask if it looks like it was taken by doorbell camera or trail camera, since the two appear quite similar. I think you would be surprised how effective these image classifiers can be.
I would use all of these ideas to just reduce the total number of observations to review, because eventually you will want to verify them.
All of these ideas are well within the capabilities of frontier AI right now to help you setup the tooling to make this possible. As for the actual grunt work of classifying everything, local models that can run on well spec’d laptops are sufficient, and you can do it with your own machine. Just pay power!
Hope there’s something useful here for you.
Sounds great - but how do you get the image to get the EXIF data? There is no API for photograph pages, AFAIK, so you are reduced to requesting them by hand, which isn’t going to scale well. Asking because I want to use the photograph data to match back to my image collection for labeling and cataloging my images.
Slugs, snails, and geckos trigger them pretty often.
Thanks @bugbaer! That’s a good idea but there will likely still be a TON of ‘regular’ observations obscured around homes. I like the idea of filtering out the never home alone ones though.
Thanks for taking the time to share these ideas @kylenessen! I think the vision model approach might be the most promising - we have the capabilities in our research group to train a custom model. I wonder if it can even learn to differentiate camera trap and doorbell camera photos…
The exif data is also a good idea but the problem is that most of the images seem to be screenshots or photos from a cell phone. So I don’t think that will be reliable.
You’re definitely right that the goal is to find a way to reduce the number of observations to manually review!
I have a handful of observations from security cameras. I think they’ve mostly been vertebrates (mostly black bears, whitetail deer, I think a red fox, and a couple birds), but I do get inverts triggering the cameras occasionally (I just can’t recall if I’ve made any observations with those). But I can’t really think of any search terms that would pull them up specifically, especially as opposed to my camera trap observations.
I don’t necessarily think that using obscured coordinates would help, either. I have a LOT of observations from home with obscured coordinates which include many insects from my rain garden and native meadow plantings as well as plant observations from uncultivated areas. All of those observations have obscured coordinates. Also quite a few bird (often audio observations, but not all) and herp observations.
It won’t be quick, but you might consider attempting to use annotations for this. You’re going to have to do some legwork to search for observations YOU can apply those annotations to before other people will start using them. One plus is that if you find someone who has uploaded an observation from a security/doorbell camera, there’s a good chance that they will have more. So if they become aware of relevant annotations and projects, they might start using them.
Good points @naturalist_nate! It’s definitely true that people won’t upload everything that they see on the cameras. For example, I’ve found very few squirrel observations so far.
Interesting idea about the annotations. But how is that better/more likely to be used than the project itself? I.e., why add the annotation if someone can just add it to the project?
I think you mean observation fields. Calling them annotations is likely to create confusion, since annotations are set up rather differently. I agree that I don’t see the advantage of observation fields vs. a traditional project for collecting such observations.
My mistake, I did mean obs fields.
It’s not an either/or. Lots of projects out there make use of obs fields for metadata they want to collect that isn’t otherwise contained within the common searchable fields. Some projects even require specific observation fields to be filled in before they can be added.
There’s a couple aspects to it. The obs fields help ensure that you’re collecting the information you want to collect. As you do the time-consuming work to find observations, add them to your project, and add obs fields relevant to what you’re seeking, other inat users will get more aware of your project and start to add observations themselves.
You can also use observation fields to as search terms to help you find observations for your project. Maybe there are existing obs fields that some people already use for home security/doorbell cam observations? Or maybe there are obs fields people use that might narrow your searches some?
The iNat team is working on developing computer vision models for purposes just as you described. See this query I tried on the iNaturalist Vision Language Demo: https://www.inaturalist.org/vision_language_demo?q=Photo+taken+from+doorbell+security+camera+or+wildlife+camera+trap&taxon_id=40151
I also made my own version of the vision language demo with more customizability using a general CLIP model - you can try it yourself at: https://ktorres23.github.io/ecology-vibe-coding/pages/secondary_explorer.html . For context, I used the filters of “Mammalia” taxa, research-grade observations, and a sample size of 100. You can see the text queries I used and the results in the screenshot below (higher “similarity scores” at the top-right of the image = higher visual-language match; This basically means that the image is likely to contain to the target feature, which is a photo taken from security camera or camera traps):
