I noticed that when I do a species search in Explore for a certain geographic area that includes the entirety of observations of a geo-obscured species, that species is missing in the search results. The geographic area is roughly the size of the uncertainty rectangle for geo-obscured observations. Many of the randomized occurrence markers for the species do in fact fall within the search area but still the search yields zero observations. I find such a search engine behavior odd and misleading. How can a species that is restricted to a certain area not come up at all when a species search is done for exactly that area? It would make sense to me if only some observations were found (those whose randomized place markers fall inside the search area), but why would it not return anything? I think this needs some improvement.
Do you have an example search URL? For example, I just put in Shenandoah Salamander and Shenandoah National Park, and the observations show up - 139/146.
https://www.inaturalist.org/observations?place_id=9012&subview=map&taxon_id=27136
I think when the obscured box goes within the specified place much at all, it gets counted.
I’m not sure without specific examples, but possibly the obscuration box does cross the bounding box of the place (red rectangle in this example).
Are you using a geographic place for your search or a bounding box?
I don’t think that the criterion for inclusion is whether the pseudo-randomized marker falls within the search but whether the obscuration box is contained within the search area. This is how it works for places at least.
I think that this is preferable behavior to basing search results on the pseudo-randomized location, as this could lead to false positives - a taxon is not found in an area, but the search says it is. For instance, when searching in a ocean area, one could gets lots of records of nearby terrestrial taxa because their pseudo-randomized points were out in the ocean - not very useful and quite confusing.
The current system will be conservative at least - unless the entire obscuration rectangle falls within the place, the taxon will not be present. This will lead to missing taxa (false negatives) - which is what I would intuitively expect with obscuration - but not false positives.
@cthawley That might be it. The geographic area (a National Park), due to its shape and size, never includes the entire obscuration rectangle, regardless of where the species is observed. That means the species is completely excluded from the Park in iNat search results, even though it only occurs in the Park and nowhere else.
Generally, false positives should be avoided, but this situation looks even worse to me. Your example would be a good point why this needs to be changed. If an oceanic species only occurs near shore (and therefore the obscuration rectangles always include land) that species would never show up in a search for this particular body of water. It would literally drop off the map. Not a good situation (in my opinion).
To me the whole idea of randomized place markers is, that for practical purposes a definite locality is needed, even if it is inaccurate. And that needs to be used in searches. The same problem exists, BTW, with large uncertainty radii of non-obscured observations. I have not looked into how those behave in iNat place searches.
@arman Actually, your example behaves differently than mine. I randomly picked one of the salamander observations that was returned in the search to which you provided the link: https://www.inaturalist.org/observations/63589303. In this case, the obscuration rectangle has large portions outside the National Park, yet it was included in the search results. So there must be something else going on.
BTW: my example concerns Astragalus schmolliae in Mesa Verde National Park: https://www.inaturalist.org/observations?place_id=69108&taxon_id=139556 (0 observations). I have to go to county level to get results: https://www.inaturalist.org/observations?place_id=1288&subview=map&taxon_id=139556
I believe that
behave in the same way.
Though as above, I think this is the best approach to handling searches of uncertainty in records, not a problem.
I don’t think that
I would say that one of the primary reasons for these markers is display on maps which doesn’t require any specific search functionality.
Something else that is odd about my specific example that most of the randomized locality markers fall south of Mesa Verde NP. That is inconsistent with the distribution of the observations, which are mostly centred in areas heavily visited by people (around Park HQ and main trails). Something strange going on there.
@cthawley Arman’s salamander search results are inconsistent with your explanation.
The pseudo-randomized markers will all be in the same 2 degree by 2 degree box. So even if the observations are all in one location, if that location is in the upper left corner of the bounding box for that area, the markers will be spread across the box. If the pseudo-randomized markers were centered on the actual location, this would defeat the purpose of obscuration. You may want to check out the geoprivacy documentation for more details of how obscuration works: https://www.inaturalist.org/pages/help#geoprivacy
Addition:
One note from the documentation might explain the issue:
“Geographic searches on most places use the publicly blurred latitude and longitude (searches on countries, states, and counties use the private latitude and longitude).”
I think that some/all US National Parks were added somewhat officially. The Shenandoah one was added fairly early (based on its lower number for Place ID than Mesa Verde) so perhaps it functions as a county for its location search, not sure? But I think this is the exception, and the Mesa Verde location search is functioning as intended.
@jwidness may know better.
That still does not explain why the salamander search yields results and the plant search does not. In both cases the obscuration rectangles intersect the park boundary.
Yes, the parks that were added in 2016, during our collaboration with the US National Park Service, are all Standard Places on iNaturalist. And place 69108 is a Standard Place. If the observation was made within the park boundaries it should come up in that search, I believe, unless there’s a bug.
@tiwane Thanks for the info! Based on what you are telling me there appears to be a bug. Zero observations for Astragalus schmolliae for Mesa Verde NP: https://www.inaturalist.org/observations?place_id=69108&taxon_id=139556. The entire known range of this species is within the park. Perhaps someone can look into this and get it fixed. Thanks for everyone’s input! Very helpful.
For what it’s worth, there are two Mesa Verde National Park locations in case that is part of the issue, though I don’t think that either of them turn up the species in the search.
One other potential issue could beif the observations themselves had accuracy values that were larger than the park (ie, not due to geoprivacy). This appears to only be the case for perhaps one.
I also noticed that the place guesses for all the observations are just “United States” whereas those for Shenandoah are more specific (all at least “Virginia” if not more specific).
Place ID=69108 shows the correct park boundaries for Mesa Verde, Colorado.
The total number of observations currently is 22. I recently identified about a dozen of them and saw the actual localities before they were obscured by the system. All the ones I saw had very small accuracy radii.
The counterpoint to this argument is that it could cause species to be reported for countries they don’t actually exist in if the obscuration rectangle where it is found (in the neighboring country) overlaps the border. There really isn’t a perfect solution for this problem. Every approach is going to have different issues and different trade-offs.
@zygy That’s an interesting topic for discussion, but what we are dealing with here is actually a bug. Normally, iNaturalist does return obscured observations for places, even when the obscuration rectangle is partially outside the place. I tested this on some other examples, e.g. Cypripedium montanum for Waterton Lakes National Park, and there results are properly displayed. This is a specific problem with a specific place (Mesa Verde NP) that needs fixing.
I don’t think this statement
is entirely correct.
It is true for Standard Places in both the Explore and Identify interfaces. This is what my earlier quote of the geoprivacy guidelines referred to:
National Parks are considered to be standard places as noted previously:
It would probably be good to add the specific language about Standard Places (with the link) to the documentation on geoprivacy as it would help clarify situations like this. It is currently in:
https://www.inaturalist.org/pages/help#placeindex
but would definitely be good to have in Geoprivacy (or on the new Help pages)
But any places that are not Standard Places will not index observations of obscured species to them based on obscuration boxes/accuracy circles, relevant info from the link above is:
"Each place boundary has what is called a “bounding box,” which is a rectangle of latitude/longitude lines that inscribe the entire boundary. For example, below in red is the approximate bounding box for Lake Merritt, in Oakland:
iNaturalist will not index an observation as being in Lake Merritt if either the observation’s accuracy circle or obscuration rectangle break that bounding box. We do this to prevent observations from being added to a place when there’s a chance they were not found there and, more importantly, to prevent users from narrowing down the location of an obscured observation. This means that if you have a Collection project for a small place, obscured observations as well as observations made near the edge of the boundary may not be displayed in your project and you may want to consider using a traditional project.
Note that this does not apply to counties, states, and countries and their equivalents, which are “standard places” in iNaturalist (as opposed to “community curated places” that anyone can add."
On the other hand, if someone wishes to see the obscured markers or search on them, they can do this in the Explore interface, using a bounding box, searching Google places, etc.
The apparent bug here specifically involves the Mesa Verde National Park place not functioning as a Standard Place (which it should).
For Waterton Lakes, since this is a Canadian National Park, I’m not sure if it counts as a Standard Place or not (I don’t think so?). It looks like a community curated place based on creation, but I don’t know of an easy way to determine whether a specific place is a standard place or not.
I think that the Park may be showing some Cypripedium montanum but not all because it looks like one obscuration rectangle does fall inside the park’s bounding box but not all of the obscuration rectangles for the area. You can see that only the central part of the park shows obscured observation markers from one rectangle in the map for the relevant project:
https://www.inaturalist.org/projects/waterton-lakes-national-park-parc-national-des-lacs-waterton
If the park were a Standard Place, I think we’d expect to see obscured markers to the east and west. Eyeballing the park’s bounding box, we can see it looks like it would juuuuuuuust exceed the north and south edges of the relevant “central” obscuration 2 degree X 2 degree rectangle, but the eastern and western edges of the park are not wide enough to include the next obscuration rectangles to the east and west.
In Explore, a search for Cypripedium montanum turns up 121 observations for Waterton Lakes NP (place_id=55198), even though the obscuration rectangle exceeds the park boundary considerably in the northeast. Two of my own observations from the Park also show up. There are only about 140 observations of this species in the wider area on the Canadian side so it definitely captured a high percentage of the total observations for this park.
The geoprivacy information page (https://www.inaturalist.org/pages/help#geoprivacy) states that “Geographic searches on most places use the publicly blurred latitude and longitude (searches on countries, states, and counties use the private latitude and longitude).” That seems to imply that searches for standard places should actually yield all observations because the search uses the private (i.e. actual) coordinates. Searches on non-standard places should yield all the observations whose publicly blurred coordinates fall within the search area.
The query should never come back empty for standard places if there is at least one observation for the place. It should only come back empty for non-standard places if none of the publicly blurred coordinates fall within the search area. That’s at least what the info page says. Perhaps it’s inaccurate, in which case it should be updated.
I am personally happy with how the searches behave overall, and I am not asking for any changes. Only this bug for Mesa Verde NP needs to be fixed.