Not all observations uploading

just fyi – the matching missing / broken observation is https://www.inaturalist.org/observations/29563341, and its photo is https://www.inaturalist.org/photos/46145360.

i believe the photo records are actually created as you load them, and the observation records are created upon submit. so they are two different processes i think. the observation does appear to be loaded, at least partially, which is why you see a spinning wheel when you try to view it, as opposed to the “i got nuthin’” mole(?).

i’m thinking that for someone with the right access and tools, it’s probably easier (than simply trying to reproduce the problem by loading a duplicate observation) to try to figure out where the creation of the observation broke by looking at the database from the back end to find out what parts of the observation were created (or not). looking from the front end at the photo record, you can see the system knows the associated (missing / broken) observation’s taxon, observer, observation date, and location. so some other part of the observation record must be causing the problem, i would think. once you know what wasn’t created properly in the database, you should be able to look at what code should have created that data, and then focus troubleshooting / debugging efforts on that.

1 Like

Thank you so much, @pisum! Yes, I am hoping the devs can have a look at the links you’ve found and let us know if the problem is on my side or their side, or a combination or a prank by alien space bicycles with too much time on their hands ;-)

I have just done a test by downloading all my obs in a csv to see if any of the missing obs are recorded on it. They are not.

I had good intentions to try a different browser this morning, but have not yet got around to it. Will do it shortly…

Have just tried Firefox (on Ubuntu Studio 18.10).

First batch of 7 uploaded fine.

Second batch of 5 had issues. When I hit Submit, I got this error:

upload1

So I clicked “Stay & try again”, and these TWO had not uploaded:

upload2

When I submitted again, they both uploaded (yay!). URLs of the 2 problem children (which have uploaded correctly) are:
https://www.inaturalist.org/observations/29627057
https://www.inaturalist.org/observations/29627056

I used to use FF all the time and I never had an upload problem once of this nature. So I think this is something new.

1 Like

I have also just discovered a duplicated ob:
https://www.inaturalist.org/observations/29582135
and
https://www.inaturalist.org/observations/29581937

The difference in upload date is 3 minutes. I don’t remember if this was one I reloaded…

ah, that’s interesting. i wonder if maybe Firefox is displaying an error when it runs into a problem when Brave is just skipping over it? if you see this error again on Firefox, it might be helpful to pull up Firefox’s web console and error console and grab screenshots of those.

i also noticed that the photos associated with the 2 errors each have only one good observation record associated with them (the ones created as a result of staying and trying again), which is good. i wonder if you click the “ignore and continue” option, whether or not that will create a bad / broken observation record, similar to what you would have gotten with Brave?

Yes, I forgot to get a console shot, pffft.

What I find interesting is that the error screen must be iNat’s own, not the browser’s, yes? In which case something isn’t being correctly communicated between iNat & Brave? Or FF for that matter, if I’m getting errors on both browsers…

But, as I say, I have never had this error problem on FF before, and FF is much, much slower than Brave on my laidback rural internet.

the pop up code is provided by iNaturalist, but it’s possible that the code is written / tested only for a certain set of browsers or that different browsers handle code / errors differently. for example, on that page, Edge seems to run into authentication errors that prevents things like computer vision from working (on that specific upload page), but it works just fine on Chrome. so i bet in that case, the code was just never tested to be compatible on Edge.

Just as a caution here, I have sometimes gotten this error (some observations failed to save) from the web uploader (FF + Win7). Before resubmitting, I always check my observations in a separate tab, and often find that they have in fact been created successfully with all photos attached. I’ve always chalked this up to a little asynchrony between some set time-out limit and the arrival of feedback from iNaturalist. If I don’t check first in such situations, I risk creating duplicate observations.

One (admittedly very “edge”) case where observations have truly failed to save for me is when I’ve gone past the 750-byte limit for tags. (I generate a lot of tags sometimes!) The uploader doesn’t handle that error very gracefully – no message to say what specifically went wrong, etc.

2 Likes

Thanks, jdmore!

I have been wondering about extras - I don’t use tags, but I add a lot of fields and projects. I have found that the problem is definitely worse when I have added different fields, field values and localities to a batch of obs. But not always…

I didn’t know there was a byte limit!! Is there such a limit on fields and projects? Or description text? How would one know about it?

2 Likes

I can’t now remember or re-create how I discovered the 750 limit for tags, and would love know the answer to each of those very good questions! All I can say is that I have not yet run up against any other data-length limits while using the site.

Thank you.

Maybe some database data types/objects (e.g. longtext) or the constraints need to be changed?

I’m pretty sure I’ve just released a fix for @karoopixie’s problem with the two Drepanogynis observation . Looking at the logs around the time they were added I found some bugs on our end related to updating counts of observation fields. Those issues should have caused the error state you saw with the popup saying “observations failed to save,” and your subsequent experience of submitting again successfully also sounds like what I would expect.

I don’t think the browser has anything to do with this specific problem. I use Brave on a Mac and have never had an issue like this.

Please holler if this is still happening.

1 Like

Thank you so much @kueda!! I will test later today. Is it possible to restore/fix the obs that have not loaded properly? Or at least find all the orphaned pictures so that I (and perhaps others) know what I (we) need to reload?

I have tested upload on Brave 0.66.101 on Ubuntu Studio 19.04 with a heavy field, field value & locality load, and all went well. Yay :notes:

Thank you so much for looking into and fixing this issue, Ken-ichi and Tony. Thank you so much to pisum and jdmore for helping me (and hopefully others) with finding clues to the problem. I am blown away by all your help, caring, generosity and expertise, especially since I seemed to be one of only two people, along with jdmore, experiencing this problem.

Thank you :green_heart:

I will just do a few more uploads to test, and then if I have no more problems, I’ll mark this as solved.

3 Likes

I have done more tests, and even uploading 15 obs with various fields etc. has not broken anything.

Marking as solved :grinning:

1 Like

I’m not sure which obs you’re referring to. If you can provide some URLs maybe we can investigate. The “missing / broken” observations @pisum linked to above all seem to be working for me. I can see that you have a number of “orphaned” photos, but it’s impossible for me to tell which of those were intended to be included in observations but weren’t, which ones were accidentally duplicated but the duplicate ended up in the observation, and which ones you uploaded and then intentionally removed without adding it to an observation.

Ah, they are loading now (they didn’t before). Thanks, Ken-ichi, I will go through all obs I uploaded during June/July and look for duplicates.

Okay, I found 5 duplicates (though I have documented 8 non-uploaders which I reloaded on this thread).

If photos stay in the db even if an ob is deleted, then it will be hard to find the ‘real’ orphans! Shouldn’t they be deleted along with the ob?

Anyway, the duplicates were Drepanogynis figurata, Attagenus, Laelia, Lepidoptera (micromoth) and Lepidoptera (grey moth). If that helps? I forgot to note the URLs :-( - but they include all the ones @pisum found, so that’s good :-) Since (at least some of) the orphans are now working, I’m hoping that all my ‘dropped’ obs are back. I uploaded nearly 1000 obs in July, so I don’t have time (or inclination) to check if anything’s missing.

Thank you again to all who helped me! {This thread can be closed now.}

1 Like

They do get deleted… eventually. We’ve had so many people accidentally (or intentionally) delete observations and then want them back that we started delaying the deletion of photos. These days we only immediately delete photos when someone asks us to remove all their personal info under the auspices of the GDPR.

3 Likes