Tool for exporting iNaturalist data to iRecord (or elsewhere)

I´ve made a tool to help download data in a way the current on-site export tools don´t allow. (e.g. with multiple photo URLs). This is particularly directed towards UK users like myself who may wish to send their data as a batch to iRecord now the data-link has been severed, but can be used by anyone needing a more versatile data download tool to get spreadsheets.

To use it, just go to this link. Once Binder has loaded, click on the .iynb file.
Inside the file, just follow the instructions one at a time.

NOTE: The code may look intimidating if you’ve not worked with code before, but you only need to alter one or two little bits to use the basic setup. ( no more complicated than using the URL edits inside iNaturalist really). The instructions should walk you through it. But I’m also happy to guide folks through any issues and explain how to change to obtain different versions of the data - just give me a shout.

With thanks to @jcook for the starting points on how to use PyiNaturalist, as detailed here.

11 Likes

I don’t mean to be critical of what you have done, but it is somewhat unusual to have to install a compiler for users to run your application as those are somewhat confusing tools to people unfamiliar with them, only used by developers.

Have you considered making an executable file and a simple user interface to run your code instead?

2 Likes

Well done Sam, I have been testing your procedure and it works admirably - I have now transferred over 50,000 records to iRecord from iNat complete with all the relevant photographs to aid ID and verification.

Excellent and so, so needed !!!

Steve

4 Likes

Sure! Its just mark one.
As I say at the bottom of the post, I’m hoping to make a version that can be used interactively directly in the browser instead.

In the mean time, its not perfect, but then not sure how many people will use it…so posting it partly to see if there’s interest. @SteveMcBill will likely be one of the biggest users - and, as he says, he found it easy enough.

…and though the steps might sound like a bit of a rigmarole, there is really nothing complicated about using it. The installation of Anaconda itself is just like any other programme. The installation of the libraries is just a few pip commands. After that its basically just pressing the run button.

It would be really nice to have a GUI version as you say, but this would be a fair amount of work when iNaturalist already has an export tool page which they might well change soon anyhow to incorporate the things which lead to me doing this.
Could be a good challenge for me though!

I think its also good to try and help “demystify the API” as @pisum mentioned on another thread - it makes sense to me to help open people up to the use of the API / pyinaturalist through doing simple tasks.

3 Likes

As an alternative, the Jupyter Notebook file (.ipynb) linked above can be imported into Google Colab and run in your browser without installing anything on your computer.

5 Likes

Yes! Thats the plan - I’ve just not had time to explore how to set it up yet.
Maybe its worth fixing sooner rather than later if its quick to sort out (?)
@jcook also mentioned Binder.

1 Like

Binder may be the way to go, Colab requires a google login. In Binder and Colab it looks like just the imports would need a tidy, adding the following at the beginning of your ‘getting started’ imports code block should install the non standard packages:

!pip install pyinaturalist
!pip install OSGridConverter
!pip install pandas
3 Likes

nice! thanks @lawnranger - will take a look later :)

1 Like

Great work. Were planning to use iNaturalist with most of the uk national parks next year. We have a plan to get data to iRecord. But this will make it much easier. Not had a chance to have a proper look yet (I need corporate IT to allow me to down load stuff) but looks like just what we need. Thanks for setting up.

3 Likes

Good to hear!
Hopefully can make a more advanced version by then - this is really just skin and bones atm.
Let me know if there is anything in particular you need or if anything is unclear.

1 Like

I think i wait for an easier version. I think i quit the programm while it was still busy and i did not understood the the only 2 lines were the programm which i had to execute. I am not familiar with Anaconda en Jupiter so it was a bit hard. Probably the 2nd time could be much much easier but i think i wait for a next version. I did not now there should be a .csv file and where it should be in first instance.

First i tried https://colab.research.google.com/ but uploading was a bit weird so i stopped there.
A bit more feedback what is he doing
( starting…busy…finished) is sometimes very informative for an innocent user.

Is Binder easier than Colab ? I do have google accounts so Colab is not a strange choice for me (accuatlly my google account was the reason i started with Colab)

= = =

BlockquoteRequirement already satisfied: pyinaturalist in /Users/samuelrees/pyinaturalist (0.11.0)
Requirement already satisfied: python-dateutil>=2.0 in /anaconda3/lib/python3.6/site-packages (from pyinaturalist) (2.8.1)
Requirement already satisfied: python-forge in /anaconda3/lib/python3.6/site-packages (from pyinaturalist) (18.6.0)
Requirement already satisfied: requests>=2.24.0 in /anaconda3/lib/python3.6/site-packages (from pyinaturalist) (2.25.1)
Requirement already satisfied: six>=1.5 in /anaconda3/lib/python3.6/site-packages (from python-dateutil>=2.0->pyinaturalist) (1.15.0)
Requirement already satisfied: idna<3,>=2.5 in /anaconda3/lib/python3.6/site-packages (from requests>=2.24.0->pyinaturalist) (2.10)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /anaconda3/lib/python3.6/site-packages (from requests>=2.24.0->pyinaturalist) (1.26.2)
Requirement already satisfied: chardet<5,>=3.0.2 in /anaconda3/lib/python3.6/site-packages (from requests>=2.24.0->pyinaturalist) (4.0.0)
Requirement already satisfied: certifi>=2017.4.17 in /anaconda3/lib/python3.6/site-packages (from requests>=2.24.0->pyinaturalist) (2020.12.5)
Requirement already satisfied: pandas in /anaconda3/lib/python3.6/site-packages (0.24.2)
Requirement already satisfied: python-dateutil>=2.5.0 in /anaconda3/lib/python3.6/site-packages (from pandas) (2.8.1)
Requirement already satisfied: pytz>=2011k in /anaconda3/lib/python3.6/site-packages (from pandas) (2020.4)
Requirement already satisfied: numpy>=1.12.0 in /anaconda3/lib/python3.6/site-packages (from pandas) (1.19.2)
Requirement already satisfied: six>=1.5 in /anaconda3/lib/python3.6/site-packages (from python-dateutil>=2.5.0->pandas) (1.15.0)
Requirement already satisfied: OSGridConverter in /anaconda3/lib/python3.6/site-packages (0.1.3)

1 Like

ok good to know, thanks for the feedback!

I have updated it now so you can run it directly in the browser using Binder.
I also added some print statements to try and make it clearer whether a cell has run or not.
:)

I will update the main post, but here is the link.

Thanks everyone.

1 Like

For me it would be more clear, it was for by adding in [8]:

Excute/Run next two lines in the cell to see the recent data

Blockquote VIEWING THE DATA
Now we’ve got it ready, lets check it out inside a dataframe so its a bit more legible.

It took me one hour to understand what i should do. And that was run those two lines of code.

1 Like

Ok thanks - will try and make this clearer too.

How about,
“Run the following cell to see the data in a spreadsheet format” ?

1 Like

Good idea, But remember, i never saw those things so it should be more easy and clear. I tried to run and refresh the data cell below [8] but not those two lines of code in the box above.

1 Like

Maybe its best if I split it into two versions.
One with only one or two cells thats just click and run…
and another version which explains a bit more how to edit and which bit does what.
This version is a little inbetween.

1 Like

How much work is it ?
I think you should wait on other feedback. With little extra’s like a bit more tekst and explaination it will get clearer is think. I am not good in this as i do not recognise what is the executable code, executble cell.

1 Like

I’ll try and make an update in the next week :)

1 Like

thankyou. will it deal with hidden coordinates at all? we are starting a traditional project to get people to join so we can view these. but not sure how to handle getting them into iRecord.

Not sure I understand exactly what you mean…
But at present, it has an option to include those with obscured/private coordinates or not, so will place them in whatever way the API / iNaturalist deals with them. The notebook just converts to OS and offers a rough conversion of the position accuracy into grid reference form.