Not so obvious is how to download data with obscured localities.
I do have a project that allows me to do so, but it has over 445 000 observations and the only way I can get the obscured coordinates is by trying to download all 445 000 observations which always fails to finish. If I try any filtering for a taxon, then the obscured data does not download.
How can I download observations with the actual coordinates for hidden coordinates, without having to create new projects each time and getting special permission for each project from each of hundreds of users?
1 Like
I split this report off from the feature announcement.
Is it possible to get a reply to this?
I am still having this problem.
Basically, I do have a project for which iNaturalist users have granted me rights to see their coordinates, but the download function for this does not work as there are lots of observations.
Whereas applying to iNaturalist for permissions allows me to obtain obscurations by Red List status, the user obscurations only appear to be extractable by this means, and it does not work.
Iâm having the same problem with trying to download hidden coordinates from the Australasian Fishes project. Several project curators have told me that they have also experienced this problem. Help please!
One âsolutionâ (doesnât solve the problem per se, but is a workaround), is to download the data piecemeal.
@tonyrebelo Given the limit of 200,000 observations per download and your 445,000 observations, you would have to download 3 CSVs (150x3 is probably best), and then combine them after downloading. You can cut up the data by date, so youâll need to tinker with date ranges to find 3 discrete blocks with ~150,000 observations in each.
@markmcg Aus Fishes should be ok given itâs under 100,000 observations? Have you tried to download a CSV containing everything?
Note that a download of 100-150,000 will take a fair while, perhaps 30-40 mins
Sorry about the slow reply @thebeachcomber. I only just now saw your comment. I donât imagine that downloading a csv will help because it wonât contain the hidden coordinates.
We did some testing and unfortunately much of this just have to do with the fact that the project exporter isnât very good. At this scale (pun intended) it just takes a long time.
Ken-ichi said Australasian Fishes took about 20 minutes to complete, so we recommend clicking the Export with Hidden Coordinates link, which kicks off the job. The job should run even if you get that error message, but it takes a long time. So wait 20-30 minutes and click the link again. Not a great solution, I know, but a workaround for the current situation.
2 Likes
After digging into this specifically for the case of @tonyrebeloâs Habitats (s Afr) project which currently has 596K observations, I want to clarify how anyone running into issues with this kind of export should proceed.
- Above a certain size (we arenât sure exactly what), the link you see as a project owner/manager/curator to âExport with Hidden Coordinatesâ will stop working.
- Instead, you should go to the export page https://www.inaturalist.org/observations/export
- Enter the project name in the project field (an essential important step, or you wonât get the hidden coordinates)
- Use some filters to focus on the observations of interest, or otherwise break it into manageable sections under 200K observations
- Select desired columns to include (the fewer you can get away with, the faster it will be)
- Click âCreate Exportâ. If it takes a while, revisit the export page later and scroll to the bottom and/or request an email notification (if prompted).
Based on my testing, this method of exporting project data should still give you access to the hidden coordinates that you have been trusted with through projects. I think it will also include hidden coordinates from users who have trusted you.
1 Like
Thanks. I tried that (I have tried it before - I had just assumed that this was the way to obtain data for which I had been entrusted with coordinates by users).
But the fields Private_latitude and Private_longitude were not included in the download. (I presume from previous experience that the fields Latitude and Longitude are useless for observations with coordinates_obscured=True.)
I did have the two fields ticked for inclusion in the download. The Habitat (s afr) observation field for the Habitats (s Afr) project was included.
Here is the download summary:
|Download|2.47 KB|17 Oct 13:07|17 Oct 13:07|17 Oct 13:07|Query quality_grade=any&identifications=any&taxon_id=40057&projects=habitats+(s+afr)
Columns id, observed_on_string, observed_on, time_observed_at, time_zone, user_id, user_login, created_at, updated_at, quality_grade, license, url, image_url, sound_url, tag_list, description, num_identification_agreements, num_identification_disagreements, captive_cultivated, oauth_application_id, place_guess, latitude, longitude, positional_accuracy, geoprivacy, taxon_geoprivacy, coordinates_obscured, positioning_method, positioning_device, species_guess, scientific_name, common_name, iconic_taxon_name, taxon_id, field:habitat (s afr)|
So no: this does not give the user- or taxon- obscured data, assuming that I correctly interpreted your instructions above.
Curiously, if rather than selecting fields in the export page https://www.inaturalist.org/observations/export page, I click the Export option in the filter box of the explore page:
https://www.inaturalist.org/observations?place_id=any&project_id=habitats-s-afr&subview=grid&taxon_id=40057&verifiable=any
it generates this url:
https://www.inaturalist.org/observations/export?verifiable=&spam=false&place_id=any&user_id=&swlng=&swlat=&nelng=&nelat=&lat=&lng=&taxon_id=40057&projects=habitats-s-afr
and this summary:
quality_grade=any&identifications=any&taxon_id=40057&projects%5B%5D=habitats-s-afr
| File size | Queued | Started | Finished | Inputs |
|---|
|Download|2.15 KB|17 Oct 13:45|17 Oct 13:45|17 Oct 13:45|Query quality_grade=any&identifications=any&taxon_id=40057&projects=habitats-s-afr
Columns id, observed_on_string, observed_on, time_observed_at, time_zone, user_id, user_login, created_at, updated_at, quality_grade, license, url, image_url, sound_url, tag_list, description, num_identification_agreements, num_identification_disagreements, captive_cultivated, oauth_application_id, place_guess, latitude, longitude, positional_accuracy, private_place_guess, private_latitude, private_longitude, private_positional_accuracy, geoprivacy, taxon_geoprivacy, coordinates_obscured, positioning_method, positioning_device, species_guess, scientific_name, common_name, iconic_taxon_name, taxon_id, field:habitat (s afr)|
Except that this time, notice the private longitute and latitude are included.
Curiously, although the filter is the same, the previous version downloaded 15 observations, whereas this version downloaded 11 observations. So I have the private coordinates, but I have lost 4 observations.
Note about the private coordinates: summary (in parentheses, after all observations added to project)
visible user obscured: na
visible taxon obscured:3 (3)
visible user and taxon obscured:1 (1)
not visible user obscured: na
not visible taxon obscured: 1 (5)
not visible user and taxon obscured: 6 (6)
Ah! The missing four observations had not joined the Habitat project, so what happened with the previous filter if the Habitats project was not included?
(it appears that habitats+(s+afr) does not parse correctly as generated by the export query, but that habitats-s-afr from the explore filter does parse. ))
Confirmed. line 2 of the instructions does not work.
Carrie, in your instructions, replace line 2
2. Instead, you should go to the export page https://www.inaturalist.org/observations/export
and suggest that the user goes to the explore page, add the project to the filter and download from there::
2. Go to the explore page: https://www.inaturalist.org/observations and select the project in the grey âFiltersâ box, and at the bottom of the filters box click the âdownloadâ option.