I have been uploading hundreds of old images. Now I get an error message saying “Too many requests” Is this short or long term? Why? I was specifically told my images would be welcomed.
Hundreds ?
Each with accurate location ? And date ? Wow you were an observer already in 1944 ! And your best (attempt at an) ID ?
PS that obs has since been corrected to 1994.
Almost 8K obs already. Might you try adding some IDs for others too ? (You are under no obligation to do so from iNat)
Try adding smaller batches at a time ?
Maybe its too many at one time for the server to process simultaneously? Try uploading in smaller batches?
iNat can only handle so much traffic at once, and they sometimes rate-limit people if they make a lot of traffic very quickly to protect their servers from abuse. The error should go away soon.
It is pretty impressive that you managed to hit the rate limit by hand, though. You might have to upload in smaller batches.
I have been enjoying and recording in detail wildlife in many forms for 60 years or more. I was told my records would be valued by the managers of this website. The plant records I’m uploading at present are scanned analogue images of UK plants from between 1975 until I went digital. I don’t need a lecture about respecting wildlife. I can cross-reference them all with my notebooks and, in respect of UK plants with the records in the BSBI database and county flora which I co-authored. What have you done?
Well that is great that you care so much about the things that you are posting as observations!
I can’t help but to wonder … if these are old photos that you are scanning, how do you remember the precise locations of each and every specimen? I would think that it would take a significant amount of time to open the map for each observation, and then zoom in and out until you finally find the exact spot where the photograph was taken. And then adjust the location circle manually, to include more, or less, area, depending on how precise your memory is.
It seems to me like this would take so much time, for each and every photo, that it would be impossible to hit that “too many requests” threshold.
Then there’s also the matter of writing notes to accompany each observation. Not every one, of course, but I would think that at least 30% of the obs one submits would have some things worth noting by writing a comment or description in the “notes” field.
And then there is also the “annotation” field, where one can (should) add values for alive vs. dead, “evidence of Presence”, sex, and life stage.
Hence, I am a bit confused about how you can apply so much care and precision to each and every observation that you submit, and still exceed the threshold that the site’s software is capable of processing.
I am NOT saying that I do not believe you. It is just that one of the things that you say seems to be mutually exclusive of another of the things that you say. And therefore I am confused. Any clarification would be appreciated. I want this to make perfect sense to me.
Are you uploading each photo one at a time? And then carefully entering species and location data for each photo, immediately after uploading it? Or are you trying to upload more than one photo at a time?
I mean for me, if I spend a day afield and get 6 observations out of it, it takes me about 4 or 5 minutes to enter the info for each observation. And that is not counting the hour plus that it takes beforehand, to look through all of the photos and select the one that best represents each specimen. So entering those 6 photos, along with the info that accompanies them, takes almost a half hour. And I have never run into the system error message that you have encountered. So that makes me wonder if maybe you are taking some kind of shortcuts, or perhaps not making sure that the location data is really precise by manually entering it in the map, separately, for each observation.
What have I done? I have spent thousands of hours with many wildlife species, observing and photographing them. And as a result of those efforts, I have had thousands of wildlife photos sold and published worldwide. But that isn’t a big deal. Anyone with a decent camera, a lot of free time, and a love for wildlife can do that, and it only takes a few years to build up a massive portfolio of marketable images. So while the phrase “thousands of wildlife photos sold and published worldwide” may sound impressive, it doesn’t really take that much to do it. Anyone can do it as long as they have a decent camera, a computer with editing software and an internet connection, and a way to travel about and get to a good number of disparate habitats. It really doesn’t take much to do what I’ve done. But still, it is not like I have done nothing. What I have done actually is fairly significant, even though anyone could do it. But it sounds like you’ve already done far more than I have, with photo archives going all the way back to 1975! So you have 5 decades of wildlife experience … that’s pretty amazing!
When I’m uploading photos to iNat from my laptop, usually all from the same general area, I usually don’t do more than 8 at a time. I don’t know if that’s far below what the uploader can handle at a time, but I figure if it chokes on an upload attempt, fewer records are better if I have to attempt it again.
I haven’t hit the rate limit by hand myself - only by accessing the API with software - but I imagine you could hit the limit if you try to upload enough photos at once, though whether that would trigger when you first put them into the upload page or only when submitting I can’t say.
If you’re going through years of observations, each potentially with multiple photos, that can easily get to be a lot of API calls. There might be an overall daily limit too; it’s been a minute since I checked all the usage guidelines.
This is very unusual, so it would be interesting to know more about the exact upload method you were using at the time.
I had a look at your observations, and I can see that they were all uploaded using the website rather than an app. Also, looking back over the last week, you’re only uploading around 50-150 observations per day.
The API Recommended Practices describes the circumstance under which you might get the error message you’re seeing:
Query Rate
- Please keep requests to about 1 per second, and around 10k API requests a day
- Requests exceeding this limit might be throttled, and will return an HTTP 429 exception “Too Many Requests”
Each upload requires at least one request for the observation data, plus another for each associated photo. So with an average of, say, four photos per observation, that equates to about 750 requests for 150 observations, which is obviously far below the per-day limit. Adding annotations also requires separate requests, but 2-3 per observation still wouldn’t be enough to overshoot the per-day limit.
The website uploader can make multiple requests in parallel, so it might be possible to trigger the per-second limit by trying to upload several large batches at once in separate browser tabs (which is obviously not recommended). However, if you’re not doing anything like that, it’s not clear why that 429 error would ever occur. Just a temporary glitch, maybe…
These responses to new user comes across to me as very offputting and unnecessarily negative:
This is implying that the user is not thinking of every living thing that they observe as being worthy of some respect and as something to be mass dumped into a database.
This
may be your opinion, but it is only that - users are free to upload observations without any notes or without annotating them. There’s nothing wrong with doing so, and it is uncalled for to make other users feel that they are doing something wrong by not using iNat in the way that you prefer.
This criticism of another user is surprising, since in the past you have asserted that you use iNat exactly how you prefer (which is fine) and don’t alter your preferred way of using it to benefit others:
You also seem to call into question the veracity of a new user’s observations and/or their ability to enter correct data for them at a good speed.
However, you yourself have claimed that you can do exactly this:
It is unclear why you should believe that you have this ability but not another user.
In short, please try to encourage other users, respect their choices, and assume that they are capable of using iNat as well as you in whatever way they so choose.
Knowing the location can be a problem, but most of my old photos are marked with the general location, sometimes a precise one. Accuracy circle (yes applied manually) is often fairly precise but sometimes a mile, sometimes more but rarely as big as a county; I generally don’t post those. In any case, the fact that a photo is old is not evidence that the location is inappropriate.
Yeah, I got so caught up in making a point about how we should put a lot of time and effort into each and every observation that we submit … and in so doing, I spaced on the fact that we have a new user who we should make feel welcome.
Thanks for reeling me back in. I need that sometimes.
@roger_e_n_smith
The problem seems to be that your connection is too fast and so your upload rate is too high.
If you are using the iNat client (web app) Upload page, please post the error message as a bug report (or alternatively as a feature request).
The iNat app should throttle the upload to a rate acceptable to the iNat back end.
Please read the error message as “Too many requests per minute”,
I hope smaller batches will work for you until this is fixed.
This post from a staff member seems to confirm that opening multiple tabs can sometimes trigger the error:
Each observation page initiates several API calls, and with a per-minute request limit, it is entirely possible that opening 15 pages within seconds will make enough API calls to go over the throttling limit. The per-minute request limit is a rolling window, so each second the number of requests during the last 60 seconds is assessed, and if that value is over the limit you’ll be throttled. So if at one second you were just below the limit, a page will load, and if loading that page makes you go over the limit the next page may not load, etc. So it’s not uncommon to see the “random” behavior you saw when you’re right around the limit - that’s just a side effect of the rolling window of the request limit.
I’m not sure whether this applies to the current case, but it’s always worth bearing in mind whenever these errors occur.