Observations missing from batch upload

Platform: Website

Browser: Firefox

URLs (aka web addresses) of any relevant observations or pages: n/a (observations failed to upload)

Screenshots of what you are seeing: nothing - observations are missing

Description of problem (please provide a set of steps we can use to replicate the issue, and make as many as you need.):

Step 1: Used the upload link on the website (https://www.inaturalist.org/observations/upload) to prepare a batch upload (added 72 pictures), combining photos, editing locations, adding notes, tags, and observation fields.

Step 2: Uploaded the observations (27 total). This was the batch upload I did at 9:06 AM EDT this morning.

Step 3: Went back through my observation list to add annotations for plant phenology, noticed that only 25 observations uploaded - 2 observations (8 photos) were missing. Waited ~1 hour to check again in case they’re taking a while to come through and they’re still missing.

This seems somewhat unusual website behavior, especially since the other 25 observations uploaded just fine and I can’t think of anything that might have been different about the two that didn’t. I can upload again but don’t want to end up duplicating effort if these are still in limbo somewhere.

I doubt that’s the case, but with computers you never know.

Did you see any error messages? I assume not since you submitted the observations.

No, I didn’t get any error messages at all. It told me it was uploading 27 observations, but only 25 seem to have made it.

I have had this a few times lately when uploading via the website. One ob out of a batch just vanishes without any warning or error. I figured out that it only ever happened in cases where two of the observations were based off the same photo (like if a photo has both a swan and a duck in it, say). It’s happened at least four or five times to me but I haven’t been able to reproduce the error reliably… It doesn’t always happen when I have a duplicated pair of observations like that, just sometimes, and I have yet to fathom out more precisely the conditions under which it occurs.

I’ve uploaded a lot of observations over the past few weeks, using Firefox, with some duplicated in the uploader, but I don’t think I’ve noticed any that have been lost… It would be difficult to check now, though.

@DanielAustin which browser are you using?

I had this every now and then as well in the last weeks (Browser Firefox). I did not have the feeling that duplicated observations were affected, but am not sure. Maybe also observations that were last (or 2nd last) in the upload line?

Chrome. I should clarify that this wasn’t using the Duplicate button in the uploader, but rather manually creating two observations that had a photo in common (usually as well as at least one unique one).

I did wonder if it was something to do with dragging and dropping the same photo twice onto the uploader. Maybe when it gets to uploading the observation with the second instance of the same photo, something in the system says “oh this photo has already been uploaded successfully so this can be skipped”. Some sort of internal flag marking that the first photo has been uploaded also gets applied to the second instance because it has the same filename, or something like that. That was my best guess at what is happening, but since it doesn’t happen in all cases there is clearly more to it.

So you dragged and dropped photo A into the uploader window. Then dragged and dropped photo A into the uploader again?

Were you dragging photos from within the system, or directly from a photo editing program? Some people do the latter and it can be buggy.

None of mine were duplicates, nor were the missing observations the first or last in the batch. It seemed pretty random and I can’t think of any pattern/reason why these two in particular didn’t make it. Interesting to see that others are experiencing similar issues - I always wonder if it is my computer/internet connection or something I’m doing. I’ll have more stuff to upload so I’ll keep an eye on it and see if it happens again.

Yes, like that. 99% of my drag-n-drops are from the regular Windows Explorer window, but I can’t say for certain it wasn’t from the FastStone window so maybe that’s the issue.

can you tell which of the observations that did get loaded would have been the ones immediately before and after the ones that did not get loaded? (for example, missing obs A should have been loaded between obs E and obs F, and missing obs B should have been loaded between obs P and obs Q.)

@pleary searched our logs for your upload, @annkatrinrose. He says 27 observations were created but 2 no longer exist and he’s now looking into how that happened.

2 Likes

Thanks for basically confirming that I’m not just going crazy and imagining things!

I think they would have been between 164798078 and 164798081 so presumably numbers 164798079 and 164798080. Turns out those two indeed do not exist.

i assume 079 would have included these photos:

i assume 080 would have included these photos:

it’s strange to me that two observations would have been created and no longer exist but that their photo records would still exist. just based on my recollection of some testing i’ve done in the past, i don’t think there’s a way through the API to delete observations without also deleting the observation records associated with them (if they are tied only to the deleted observations), unless you first disassociate the photo records from the observations before deleting the observations.

if i didn’t know anything else, it would make more sense to me that the system ran into some sort of error while creating the observation records but never reported the issue back to your upload page somehow. but i guess pleary and iNat staff will have a better understanding, as they can see the logs, etc…

i don’t see anything unusual about these photo records or any of the observation records in the uploaded set. in terms of the sequence of the missing records, they would have been second and third in the set of 27, if ordered by their observation IDs.

4 Likes

Yep, those photos look about right. There should have been two more for 079. Very strange that the photos would be there but no associated observations.

1 Like

iNaturalist is likely a highly distributed and asynchronous system. This means that immediate consistency of data cannot be guaranteed, but it tries to reach “eventual consistency”.
When something goes wrong during upload, several steps may have already completed with succes. In case of an error, retries are expected to be done, and if the error lasts, the distributed transaction has to be rolled back by executing correcting actions, and notifying the client application.
Things do get terribly complicated under such circumstances, and somewhere there the issue has occured…

This may be a non-indexing problem that I experienced a couple years ago. I could ‘see’ the missing obs on my Calendar page - https://forum.inaturalist.org/t/not-all-observations-uploading/4946/54

See if your missing obs are on your calendar page.

That’s a good idea but doesn’t seem to be the case for mine. They did not make it to my calendar page.

1 Like