Bulk import of csv failed

I have 108 observations to upload - too many to experiment with, so I created a 3 observation csv file to learn with.

Platform (Android, iOS, Website): website

Browser, if a website issue (Firefox, Chrome, etc) : Chrome

URLs (aka web addresses) of any relevant observations or pages: https://www.inaturalist.org/observations/import?project_id=113069&tab=1

Screenshots of what you are seeing (instructions for taking a screenshot on computers and mobile devices: https://www.take-a-screenshot.org/):

Description of problem (please provide a set of steps we can use to replicate the issue, and make as many as you need.):

The upload fails with this message:
translation missing: en.we_tried_to_process_your_upload_named_filename

  • Latitude must be greater than -90
    • Row(s): 2, 3, 4

Please fix your CSV and try again.

Here is the full contents of the file - though the first (header) line is wrapped in this display, but of course not wrapped in the actual file.

“id”,“date”,“user_id”,“lat”,“lon”,“Estimated Height of Tree (ft)”,“DBH inches)”,“DMR.1”,“DMR.2”,“DMR.3”
“Western Hemlock”,“2021-07-14”,“paul-shannon”,47.55855,-122.25139,31,10,2,0,0
“Western Hemlock”,“2021-07-14”,“paul-shannon”,47.558316,-122.250828,121,35,2,1,0
“Western Hemlock”,“2021-07-14”,“paul-shannon”,47.55785,-122.25077,42,8,2,2,1

welcome to the forum.

when i go to the import from CSV tab under the import page, the last rule noted is:

We can only understand CSV in the following format:

Taxon name Date observed Description Place name Latitude / y coord / northing Longitude / x coord / easting Tags Geoprivacy
text YYYY-MM-DD HH:MM text text dd.dddd dd.dddd tag,tag obscured

so it seems like your file doesn’t conform with that standard. for example, it looks like your latitude value is in the 4th column in your file, but it’s expected to be in the 5th column; and your longitude value is in the 5th column in your file, but it’s expected to be in the 6th column.

the “latitude is must be greater than -90” error message is consistent with your longitude being loaded as latitude.

use the standard, and all should be fine, i think.

probably no bug here. just a data problem.

3 Likes

Thank you, pisum. Very helpful.

Here is a elated question: I have 5 custom fields in my project . How do I represent them in the csv file so that the fields are added properly to your database, and displayed properly on the project page?

DBH (inches)
DMR.1
DMR.2
DMR.3
Estimated Height of Tree (ft)

i’m not really familiar with the bulk import process, but just based on the rules noted on the import page, it doesn’t look like the bulk import itself allows for the simultaneous import of observations and related observation fields.

that said, you could probably load the observations, then go back and batch edit your observations, manually adding the necessary values.

the only way that i think regular folks like us would be able to import both observations and observation fields in an automated way would be to use the API. (how comfortable with coding are you, and which languages are you most comfortable with or would you be willing to learn?)

someone else might know of a more automated way to do this without coding though – so maybe see if someone else responds. the only other thread i can find where someone has asked a similar question seems to have gone unanswered though: https://forum.inaturalist.org/t/import-via-csv-can-this-include-additional-observation-fields/13013.

@paul-shannon do you plan on adding photos and such to the observations?

1 Like

@tiwane - no photos

@pisum - coding to the API is definitely an option. I see the R client is no longer supported, but perhaps JSON over REST? That’s a common language semi-neutral strategy, maybe iNat uses it also? A quick look at the documentation suggests to me that data access, rather than data upload, seems to be the emphasis.

Any suggestions, anything I’ve missed?

Add them at the end of the row. The caveat is that the fields must be associated with your project first. Right now, your project has three fields, but they aren’t any of the ones you mentioned. Once you add the fields to the project, they should populate just fine from the csv upload.

if the import process described by jwidness doesn’t work, or if you want to try the coding approach anyway, then…

if you want to use R, unfortunately, i’m not aware of an existing R package that includes support for posting (writing) data. (rinat only gets data from the old API, and so i wouldn’t advocate that anyway. hanly’s R package gets data from the current API, but it includes no methods to post either.) so then, yes, if you want to use R, you would have write something from scratch to post to the /v1/observations and /v1/observation_field_values API endpoints.

if you’re willing to use Python or Javascript, then you could use pyinaturalist or (iNat’s own) inaturalistjs packages, respectively. (or write your own code.)

posting will require that you get a JWT. the easiest way to get that is directly (manually) from https://www.inaturalist.org/users/api_token. but if you want to programmatically get it, you’ll have to go through a more complicated authorization process, which you can read about here: https://www.inaturalist.org/pages/api+reference.

Thanks, @jwidness. The csv format you describe worked well. Or appears to - the 3 trial observations I uploaded are not yet visible in the project. What sort of latency is normal?

Thanks also to @pisum and tiwane for your help.

I’d be grateful for an update on this. My 3-row trial upload apparently completed without error, but after 3 days have yet to appear here: https://www.inaturalist.org/projects/seward-park-western-hemlock-study.

when you imported your CSV, did you select a project from the Add to Project section?

if not, you can go back after the fact and add your observations to your project, since it’s a traditional project.

I have no hard evidence of doing that, but I am sure that I did. Looking at the csv upload page now, I’d have to blind to miss that!

Maybe you could find my file in your csv-import pipeline?

I will also, right now, upload the full data set, minus the hand-entered first observation, and the already-uploaded fileinat-upload-2-4.csv

not sure how exactly imports are accomplished by the system, but if it’s through something like a flow task, only staff would be able to see the details of that, i think.

since there were just a few observations already uploaded, you could just open up each observation and add it to the project:

(there are other ways to add observations en masse, but it’s just as easy this way since you have just a few.)

I have more than 100 observations - the 3 I uploaded were just a test.
Are staff unavailable to help?

maybe load a couple more observations, making sure you definitely add a project, and see what happens. if that isn’t adding a project, then i would still load the rest, and then add the project to the observations on the back end.

the way i would do that on the back end is to click on your profile photo in the top-right section of the iNaturalist website, click on Edit Observations in the Edit menu, and then from there, search for and batch edit the observations:

there are only so many staff to answer many questions. if you really need staff help, i would just wait to see if they respond here or try e-mailing help@inaturalist.org.

1 Like

I just uploaded the full set of 116 western hemlock observations, in a csv file

inat-upload-all.csv

To the Seward Park western hemlock study.

The system immediately redirected me to

https://www.inaturalist.org/projects/seward-park-western-hemlock-study

and notified me that the upload had been queued.

A few minutes later I refreshed the page; that notice had been removed.
I have no idea how to check the status of the queue. Nor how much delay I should expect before the observations are visible.

No error, no progress report, no observations. I may have stumbled into a bug.

Thank you for your many attempts to be helpful! I will now follow your advice, and send this information to help@inaturalist.org.

Thanks again!

  • Paul

This time it worked! Thanks again for your multiple attempts to help.

1 Like