User-Agent header disallowed via ‘Access-Control-Allow-Headers’ in API

Bug Reports
#1

Platform: API

Browser: Firefox 87.0

URLs of any relevant observations or pages: any v1 api request. for example: https://api.inaturalist.org/v1/controlled_terms

Screenshots of what you are seeing: Not a screenshot, but here’s a log message:
Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://api.inaturalist.org/v1/controlled_terms. (Reason: header ‘user-agent’ is not allowed according to header ‘Access-Control-Allow-Headers’ from CORS preflight response).

Description of problem:
I’ve started making a web app that interacts with the iNaturalist API directly through javascript in the browser, and I’ve been trying to follow the advice on the API recommended practices page.

I followed the advice on that page to include a custom User-Agent header in all requests. This worked so far in Chrome, but trying it out in Firefox - it hits an error during the pre-flight request which complains that Access-Control-Allow-Headers does not allow User-Agent. Removing the User-Agent header allows it to work in Firefox.

To reproduce
Step 1: Create simple html page with a button

Step 2: onclick of the button execute the following javascript function:

async function testUserAgent() {
    getFetchOptions = {
        method: 'GET',
        headers: {'User-Agent': 'example-user-agent'}
    };
    await fetch('https://api.inaturalist.org/v1/controlled_terms', getFetchOptions);
}

Step 3: Open the page in Firefox, open dev tools (F12) to the console, then click the button

Step 4: see the error appear in the console

I can post the full html + js if you need.

I’m thinking that either the API recommended practices page is no longer correct and needs changing, or the User-Agent header should be included in Access-Control-Allow-Headers returned from the api.
Note: This looks like the code in the api for the access control headers.

1 Like
#2

interesting. i don’t think Chrome even allows you to override the browser’s User-Agent using Javascript (which is maybe why you’re not seeing an error in Chrome).

you can override the Chrome’s User-Agent using other means, but it wouldn’t be something you could realistically ask each user to do.

it looks like inaturalistjs is setting an X-Via header that might function like a backup User-Agent in cases where it can’t be overridden. it looks like X-Via is getting logged in the background. so maybe that’s the header to use for identification purposes in this case?

1 Like
#3

Ah, that’ll be why it’s working in Chrome. You’re right that it doesn’t override it. I’m guessing Chrome has some smarts to either exclude it’s own User-Agent from the request or at least not let it cause a CORS failure.

Thanks for pointing out the inaturalistjs repository. I hadn’t looked at that one yet.
Along with X-Via, it does look like it can optionally add the user-agent header for posts at least .

I’ll switch over to X-Via for now. Maybe the recommended practices page still needs updating? Or maybe it’s just not written with browser based requests in mind.

#4

i suspect this is the case. technically the browser is the user agent for a given request. so, to me, it doesn’t make sense to override that in this case. instead, using a secondary identifier (which, for iNaturalist, seems to be X-Via) seems like the right way provide such information for now…

my understanding is that User-Agent is one of a handful of reserved headers that Chrome explicitly disallows setting via Javascript (although you can use other means to override the browser’s User-Agent string). i can’t locate any official reasoning for this, but my understanding is that Google had an interest in the past to make it hard to spoof the User-Agent because its ad business used it extensively for fingerprinting / tracking purposes.

recently though, the folks at Google seem to have rethought their fingerprinting practices (probably weighed against all the baggage of other misuses of User Agent sniffing), and they now want to deprecate User-Agent altogether in favor of a new User Agent Client Hints framework. (see Intent to Deprecate and Freeze: The User-Agent string (google.com).)

probably. i suspect there’s not a bug to be handled here, but adding some alternate guidance for web development might be warranted.

1 Like
#5

The Same Origin Policy (SOP) is the policy browsers implement to prevent vulnerabilities via Cross Site Scripting (XSS). In other words, the browser would not allow any site to make a request to any other site. It would prevent different origins from interacting with each other through such requests, like AJAX. This policy exists because it is too easy to inject a link to a javascript file that is on a different domain. This is a security risk - you really only want code that comes from the site you are on to execute and not just any code that is out there.

The Cross Origin Resource Sharing (CORS) is one of the few techniques for relaxing the SOP. Because SOP is “on” by default, setting CORS at the server-side will allow a request to be sent to the server via an XMLHttpRequest even if the request was sent from a different domain. This becomes useful if your server was intended to serve requests from other domains (e.g. if you are providing an API).

JSON with Padding is just a way to circumvent same-origin policy, when CORS is not an option. This is risky and a bad practice. Avoid using this.

If you want to bypass that restriction when fetching the contents with fetch API or XMLHttpRequest in javascript, you can use a proxy server so that it sets the header Access-Control-Allow-Origin to *.

If you need to enable CORS on the server in case of localhost, you need to have the following on request header.

Access-Control-Allow-Origin: http://localhost:9999

#6