API token can no longer be obtained non-interactively

Bug Reports
1

Platform: website

Browser, if a website issue: n/a

URLs (aka web addresses) of any relevant observations or pages: https://www.inaturalist.org/users/api_token

Description of problem:

Recently, scripts that non-interactively obtain a token from https://www.inaturalist.org/users/api_token as per the documentation at https://api.inaturalist.org no longer receive a token. This can be demonstrated with curl as follows.

$ curl -sL 'https://www.inaturalist.org/users/api_token' | grep 'api_token'

What is expected is a JSON response with:

{ "api_token:" "xxxx" }

What happens instead is a redirect to login.

2

Hello,

The redirection is returned when you call this https://www.inaturalist.org/users/api_token without a proper authentication header (ie: it redirects you to the sign in page)

From https://api.inaturalist.org/v2/docs/

Authentication in the Node API is handled via JSON Web Tokens (JWT). To obtain one, make an OAuth-authenticated request to https://www.inaturalist.org/users/api_token. Each JWT will expire after 24 hours

I suspect your issue is before this call. Could you please share (be careful with secrets) your authentication flow before this call?

1 Like
3

Ah, how careless of me!

In our discord bot written in Python, we use pyinaturalist to do the authentication and I observed that the authenticated request was failing - I naively assumed for this reason. But itā€™s probably somewhere else. Because of the similar timeframe if the emergence of the social media preview issue I also reported today I thought they might have a common cause. Probably they do not.

Later today Iā€™ll do some more careful tracing of the failure and come up with an accurate minimal failing test.

4

not sure what youā€™re doing, but i wonder if this could be relevant: https://forum.inaturalist.org/t/api-my-token-does-not-work-anymore-in-pyinaturalist/68600/4.

2 Likes
5

Iā€™m leaning less towards ā€œthe way we obtain tokens is wrongā€ (since we havenā€™t had any issues with that until recently - and none of that code has changed since before the issue emerged when it was working) and more towards ā€œthereā€™s something strange about adding ā€˜must be observed byā€™ rules to our particular projectā€ that makes it an edge case.

In fact, itā€™s not the first time we hit issues with this project. https://www.inaturalist.org/projects/discord-inaturalist-server has about 2,000 users in it. Each userā€™s observations are included via a ā€œmust be observed byā€ rule. It could be that the sheer size of this list of rules is somehow contributing to the issue. We previously discussed this in other threads, but it has been a long time since any changes were made to how that all works.

If this turns out to not be a failure to obtain a token, but something else, then Iā€™ll file a new issue and this one can be closed. Iā€™ll determine that by the end of today.

6

itā€™s possible. there are other threads where people discuss issues adding many taxa to projects in the project update screen. they (indirectly) addressed the issue in the past by increasing the timeout for the endpoint. iā€™ve never personally encountered issues when making changes using the API though, since you can submit only the changes, rather than completely recreating the entire set of rules, if i remember correctly.

7

Yes, thatā€™s a refinement we made to our project management bot code a while back, i think? Iā€™ll check it again, but I believe we only send up the single new user to add, and that has worked well for us up until a couple of days ago.

8

for what itā€™s worth, i copied the user list from your project + myself to my own project in batches of 500 users, and everything copied over just fine via PUT /v1/projects/{id}.

9

Yeah, thereā€™s something strange going on that might have something to do with the request caching layer, and maybe something changed at the server end that makes this problem emerge now. So letā€™s call this issue closed since it is definitely misdiagnosed as something else and mistitled as a result, and would have to be filed again with a correct description of the problem and steps to reproduce. I just havenā€™t had time to do that yet - and have my workaround (i wait a second or two, retry the command, and it succeeds the second time!)

10

For posterity, I record here the traceback triggered by iNaturalist hanging up on us. I will discuss with @jcook (pyinaturalist maintainer). I know pyinat has some retry logic for handling various exceptions that can be encountered, but I donā€™t think that POST is one of them (and I can think of good reasons why it normally shouldnā€™t be!) Now, pyinaturalist ultimately uses urllib3.util.Retry, and the doc for that does not include POST as one of the default allowed methods:

https://urllib3.readthedocs.io/en/stable/reference/urllib3.util.html#urllib3.util.Retry.DEFAULT_ALLOWED_METHODS

Iā€™m pretty sure that the remote hanging up on us is new behaviour, but since that couldā€™ve happened anyway, even if it has recently become more frequent, I think pyinaturalist probably needs to have exception handling for this particular case.

Exception in command 'event join'
Traceback (most recent call last):
  File "/home/redbot/.pyenv/versions/red-Dronefly/lib/python3.11/site-packages/urllib3/connectionpool.py", line 787, in urlopen
    response = self._make_request(
               ^^^^^^^^^^^^^^^^^^^
  File "/home/redbot/.pyenv/versions/red-Dronefly/lib/python3.11/site-packages/urllib3/connectionpool.py", line 534, in _make_request
    response = conn.getresponse()
               ^^^^^^^^^^^^^^^^^^
  File "/home/redbot/.pyenv/versions/red-Dronefly/lib/python3.11/site-packages/urllib3/connection.py", line 516, in getresponse
    httplib_response = super().getresponse()
                       ^^^^^^^^^^^^^^^^^^^^^
  File "/home/redbot/.pyenv/versions/3.11.10/lib/python3.11/http/client.py", line 1395, in getresponse
    response.begin()
  File "/home/redbot/.pyenv/versions/3.11.10/lib/python3.11/http/client.py", line 325, in begin
    version, status, reason = self._read_status()
                              ^^^^^^^^^^^^^^^^^^^
  File "/home/redbot/.pyenv/versions/3.11.10/lib/python3.11/http/client.py", line 294, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/redbot/.pyenv/versions/red-Dronefly/lib/python3.11/site-packages/requests/adapters.py", line 667, in send
    resp = conn.urlopen(
           ^^^^^^^^^^^^^
  File "/home/redbot/.pyenv/versions/red-Dronefly/lib/python3.11/site-packages/urllib3/connectionpool.py", line 841, in urlopen
    retries = retries.increment(
              ^^^^^^^^^^^^^^^^^^
  File "/home/redbot/.pyenv/versions/red-Dronefly/lib/python3.11/site-packages/urllib3/util/retry.py", line 474, in increment
    raise reraise(type(error), error, _stacktrace)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/redbot/.pyenv/versions/red-Dronefly/lib/python3.11/site-packages/urllib3/util/util.py", line 38, in reraise
    raise value.with_traceback(tb)
  File "/home/redbot/.pyenv/versions/red-Dronefly/lib/python3.11/site-packages/urllib3/connectionpool.py", line 787, in urlopen
    response = self._make_request(
               ^^^^^^^^^^^^^^^^^^^
  File "/home/redbot/.pyenv/versions/red-Dronefly/lib/python3.11/site-packages/urllib3/connectionpool.py", line 534, in _make_request
    response = conn.getresponse()
               ^^^^^^^^^^^^^^^^^^
  File "/home/redbot/.pyenv/versions/red-Dronefly/lib/python3.11/site-packages/urllib3/connection.py", line 516, in getresponse
    httplib_response = super().getresponse()
                       ^^^^^^^^^^^^^^^^^^^^^
  File "/home/redbot/.pyenv/versions/3.11.10/lib/python3.11/http/client.py", line 1395, in getresponse
    response.begin()
  File "/home/redbot/.pyenv/versions/3.11.10/lib/python3.11/http/client.py", line 325, in begin
    version, status, reason = self._read_status()
                              ^^^^^^^^^^^^^^^^^^^
  File "/home/redbot/.pyenv/versions/3.11.10/lib/python3.11/http/client.py", line 294, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/redbot/.pyenv/versions/red-Dronefly/lib/python3.11/site-packages/discord/ext/commands/core.py", line 235, in wrapped
    ret = await coro(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/redbot/.local/share/Dronefly/cogs/CogManager/cogs/inatcog/utils.py", line 60, in wrapped
    await coro(*args, **kwargs)
  File "/home/redbot/.local/share/Dronefly/cogs/CogManager/cogs/inatcog/commands/event.py", line 166, in event_join
    msg = await self._event_action(
          ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/redbot/.local/share/Dronefly/cogs/CogManager/cogs/inatcog/commands/event.py", line 135, in _event_action
    update_response = await client.projects.add_users(
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/redbot/.local/share/Dronefly/cogs/CogManager/cogs/inatcog/client.py", line 18, in async_wrapper
    return await asyncio.wait_for(future, timeout=20)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/redbot/.pyenv/versions/3.11.10/lib/python3.11/asyncio/tasks.py", line 489, in wait_for
    return fut.result()
           ^^^^^^^^^^^^
  File "/home/redbot/.pyenv/versions/3.11.10/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/redbot/.pyenv/versions/red-Dronefly/lib/python3.11/site-packages/pyinaturalist/controllers/project_controller.py", line 145, in add_users
    response = self.client.request(add_project_users, project_id, user_ids, auth=True, **params)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/redbot/.pyenv/versions/red-Dronefly/lib/python3.11/site-packages/pyinaturalist/client.py", line 153, in request
    kwargs = self.add_defaults(request_function, kwargs, auth)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/redbot/.pyenv/versions/red-Dronefly/lib/python3.11/site-packages/dronefly/core/clients/inat.py", line 40, in add_defaults
    _kwargs = super().add_defaults(request_function, kwargs, auth)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/redbot/.pyenv/versions/red-Dronefly/lib/python3.11/site-packages/pyinaturalist/client.py", line 111, in add_defaults
    client_kwargs['access_token'] = get_access_token(**self.creds)  # type: ignore
                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/redbot/.pyenv/versions/red-Dronefly/lib/python3.11/site-packages/pyinaturalist/auth.py", line 97, in get_access_token
    response = session.post(f'{API_V0}/oauth/token', json=payload)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/redbot/.pyenv/versions/red-Dronefly/lib/python3.11/site-packages/requests_cache/session.py", line 138, in post
    return self.request('POST', url, data=data, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/redbot/.pyenv/versions/red-Dronefly/lib/python3.11/site-packages/pyinaturalist/session.py", line 273, in request
    response = self.send(
               ^^^^^^^^^^
  File "/home/redbot/.pyenv/versions/red-Dronefly/lib/python3.11/site-packages/pyinaturalist/session.py", line 322, in send
    response = super().send(
               ^^^^^^^^^^^^^
  File "/home/redbot/.pyenv/versions/red-Dronefly/lib/python3.11/site-packages/requests_cache/session.py", line 230, in send
    response = self._send_and_cache(request, actions, cached_response, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/redbot/.pyenv/versions/red-Dronefly/lib/python3.11/site-packages/requests_cache/session.py", line 254, in _send_and_cache
    response = super().send(request, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/redbot/.pyenv/versions/red-Dronefly/lib/python3.11/site-packages/requests_ratelimiter/requests_ratelimiter.py", line 95, in send
    response = super().send(request, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/redbot/.pyenv/versions/red-Dronefly/lib/python3.11/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/redbot/.pyenv/versions/red-Dronefly/lib/python3.11/site-packages/requests/adapters.py", line 682, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/redbot/.pyenv/versions/red-Dronefly/lib/python3.11/site-packages/discord/ext/commands/bot.py", line 1366, in invoke
    await ctx.command.invoke(ctx)
  File "/home/redbot/.pyenv/versions/red-Dronefly/lib/python3.11/site-packages/redbot/core/commands/commands.py", line 825, in invoke
    await super().invoke(ctx)
  File "/home/redbot/.pyenv/versions/red-Dronefly/lib/python3.11/site-packages/discord/ext/commands/core.py", line 1650, in invoke
    await ctx.invoked_subcommand.invoke(ctx)
  File "/home/redbot/.pyenv/versions/red-Dronefly/lib/python3.11/site-packages/discord/ext/commands/core.py", line 1029, in invoke
    await injected(*ctx.args, **ctx.kwargs)  # type: ignore
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/redbot/.pyenv/versions/red-Dronefly/lib/python3.11/site-packages/discord/ext/commands/core.py", line 244, in wrapped
    raise CommandInvokeError(exc) from exc
discord.ext.commands.errors.CommandInvokeError: Command raised an exception: ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
11

Out of curiosity, do you have any exception handling for the remote hanging up on the first try to get /oauth/token? Iā€™m just realizing now that in my original post I had the wrong API endpoint that was failing! Itā€™s this one, not /users/api_token.

Since POST is not idempotent, the retry handler by default doesnā€™t retry if this fails for any reason: https://github.com/pyinat/pyinaturalist/blob/21465db9c08251fc658d09843c1181729072a549/pyinaturalist/client/oauth.py#L123

12

Since you are using pyinaturalist to connect to iNat API, the first people you should talk to about problems with iNat API is pyinaturalist. If the folks at pyinaturalist discover the connection issue isnā€™t caused by their code, then post a bug report on this forum.

1 Like
13

Absolutely! But since this is a newly emergent issue since March 28, and other users of the iNaturalist API may be following this, I think itā€™s worth publicly recording the new behaviour weā€™ve observed and the problems this causes for downstreams. Also since my initial attempt to reproduce the issue was with curl and not pyinaturalist (even though it was absolutely wrong) I did try to do the right thing before posting here and isolate the issue from the library I was using!

1 Like
14

For completeness, hereā€™s what the pyinaturalist code should be doing. Iā€™ll try to get it to fail and report back my findings. Currently itā€™s returning a bearer token without hanging up:

$ payload="{\"username\":\"$INAT_USERNAME\",\"password\":\"$INAT_PASSWORD\",\"client_id\":\"$INAT_APP_ID\",\"client_secret\":\"$INAT_APP_SECRET\",\"grant_type\":\"password\"}"
$ curl -XPOST -H "Content-Type: application/json" -d"$payload" https://www.inaturalist.org/oauth/token
15

Hey Ben, there are some unreleased changes related to retry behavior in the latest pre-release build on PyPI, but retrying timed-out OAuth POST requests is a case I must have missed. Iā€™ll take a look at that, and if needed follow up in this issue.

1 Like
16

i donā€™t do any retry logic in my own code, but i donā€™t think iā€™ve ever encountered errors out of requests to this endpoint, except maybe if iā€™ve requested too many tokens within a short period of time. so in that sort of case, retrying probably isnā€™t the best course to handle that error.