Establishment Means in two /v1/taxa endpoints either not complete or not consistent

I’m making tentative steps towards having Dronefly Discord bot display establishment means information. These displays should show if an observation is native or introduced in the place it was observed, and a taxon display requested by a user with a preferred place should show the same info. However, I have run up against some difficulties choosing between two API calls, /v1/taxa/autocomplete vs. /v1/taxa/#, as the first returns incomplete results, and the second gives inconsistent results.

For these examples, consider the following three URLs, the first being what I see on the web, and the next two are two alternate API calls available to me to write bot code to do something similar.

https://www.inaturalist.org/taxa/3454
https://api.inaturalist.org/v1/taxa/autocomplete?q=zenaida+macroura&preferred_place_id=6853
https://api.inaturalist.org/v1/taxa/3454?preferred_place_id=6853

Because Nova Scotia, Canada (6853) is my preferred place, when I visit the web URL, I see:

This is my model for the information I want to include in our Dronefly bot displays.

But when I turn to the API to try to get this result using the /v1/taxa/autocomplete call, establishment_means is missing the text of the link to “Canada Check List”. It only has a reduced subset of the fields, containing the establishment means label (native) and id#, but missing the name of the checklist, which is needed to construct the link to the /listed_taxa page on the web.

            "matched_term": "Zenaida macroura",
            "iconic_taxon_name": "Aves",
            "preferred_common_name": "Mourning Dove",
            "establishment_means": {
                "establishment_means": "native",
                "id": 6435212,
                "place": {
                    "id": 6712,
                    "name": "Canada",
                    "display_name": "Canada",
                    "ancestry": "97394"
                }
            },
            "preferred_establishment_means": "native"

I had a glimmer of hope that maybe switching API calls would work for me, when I saw that /v1/taxa returned more info in “listed_taxa”, i.e. full establishment_means records, with “list” included in each, and within “list”, “title” which is suitable for the text of the link to the listed_taxa page. But then I started noticing inconsistent results. Here’s an excerpt from an earlier test run:

            "listed_taxa_count": 89,
            "listed_taxa": [
...
                {
                    "id": 5535438,
                    "taxon_id": 3454,
                    "establishment_means": "native",
                    "place": {
                        "id": 6712,
                        "name": "Canada",
                        "display_name": "Canada",
                        "admin_level": 0,
                        "ancestor_place_ids": [
                            97394,
                            6712
                        ]
                    },
                    "list": {
                        "id": 187791,
                        "title": "NatureServe Canada"
                    }
                },

Note that a completely different listed_taxa id# is shown! It’s still for Canada, but from a different list (NatureServe Canada). So then I start filing this issue and tried reproducing the same result with the exact same API call, and got the original Canada Check List result again.

                {
                    "id": 6435212,
                    "taxon_id": 3454,
                    "establishment_means": "native",
                    "place": {
                        "id": 6712,
                        "name": "Canada",
                        "display_name": "Canada",
                        "admin_level": 0,
                        "ancestor_place_ids": [
                            97394,
                            6712
                        ]
                    },
                    "list": {
                        "id": 7025,
                        "title": "Canada Check List"
                    }
                },

It’s not like it’s doing it every time. Most of the time, I’m getting back the NatureServe listing, but some of the time, I’m getting back the Canada Check List result. And it never gives me both (i.e. it always counts 89 entries in listed_taxa, but non-deterministically filling in one or the other listed entry for Canada each time. Shouldn’t it be always using the “primary listing”?

I went back to the web to confirm my initial result there and now It’s giving me the NatureServe listing instead:

wrong screenshot deleted - see new screenshot in the comments below. but note the API call output below that provides the evidence the flipping is real.

And now I can’t get it to show the Canada Check List result again, neither on the web, nor through the API call. So I have no idea how it’s deciding, but once it determines which record to give me, it keeps giving it to me for a while.

Because of this inconsistency, I have spent a few hours off in the wrong direction, thinking I didn’t really understand how it was supposed to work, or that I had made some error in how I had set up my tests. Frustrating.

Please check it out and see if both the web & the API can be made to always return consistent results for each place.

This isn’t a blocker anymore, now that I understand it’s inconsistent both on the web and through the API call. I’ll go write the code that does both API calls (the autocomplete one to match what the user types, and the /v1/taxa/# one to get the complete record from listed_taxa). It’s a bit inconvenient to do two calls where one would’ve been ideal, but not the end of the world. And if users notice the inconsistent results, I can at least refer them back to this Bug Report.

p.s. once I was done filing this Bug Report, I went back to up-arrow my curl tester and finally it had switched back to Canada Check List again …

$ curl -sl "https://api.inaturalist.org/v1/taxa/3454?preferred_place_id=6853" | python -m json.tool | grep "title.*Canada"
                        "title": "NatureServe Canada"
$ curl -sl "https://api.inaturalist.org/v1/taxa/3454?preferred_place_id=6853" | python -m json.tool | grep "title.*Canada"
                        "title": "NatureServe Canada"
$ curl -sl "https://api.inaturalist.org/v1/taxa/3454?preferred_place_id=6853" | python -m json.tool | grep "title.*Canada"
                        "title": "Canada Check List"
1 Like

If there is more than one checklist defined for a geography, as in Canada with both the Canada place list and the Nature Serve list, one listing is defined as the primary listing.

So for instance for MODO, the Canada place list is defined as the primary list as shown bottom left here.

https://www.inaturalist.org/listed_taxa/6435212

Good chance you are getting the primary one sent back, but you should at least check that.

I’m getting the primary, and then not the primary, and then the primary, and then not for the same taxon … non-deterministically. Yes, I’ve gone back to make sure the record wasn’t changed during testing and so far as I know, NatureServe Canada has never been designated primary, yet some of the time it shows as that (both on the web and in the API).

Do you mean for that specific taxa or in general? There are cases where the NatureServe one is designated as primary.

https://www.inaturalist.org/listed_taxa/5544659

I’m pretty sure so long as the same geography is used if you change on one list it automatically updates any other listing of that taxa, so the data in the multiple lists should always be the same.

For that specific taxon. I saw it displayed as listed on NatureServe Canada at the top (and then screwed up the screenshot … I think what happened there is I navigated to the listed taxa entry and back, and then it changed). While on the NatureServe Canada listing, I looked carefully at which was designated primary and it said that Canada Check List was.

Did you see the evidence I included in my p.s. above? Here’s a screenshot of that:

Same API call over and over. The last one changed.

Here, I finally reproduced it again flipping on the web:

Then I click the NatureServe link and it says:

So you can see there it is not the primary listing, Canada Check List is. Yet it’s showing the NatureServe Canada listing on the taxon page.

My new bot code (based on the /v1/taxa/# endpoint) does the same. Sometimes it lists one, and sometimes the other.

FYI, I have now written my workaround for not having the full establishment_means in the taxon record when preferred_place_id is sent.

The workaround:

  1. does a full /v1/taxa/# lookup even when the user’s query was a text search (i.e. costs a 2nd API call after the initial lookup)
  2. finds the establishment_means place id in the listed_taxa and, if present, uses the matching record’s list title as the link text
  3. otherwise, uses the generic label “Checklist” as the link text, as that’s better than nothing
  4. finally emits a display with native/introduced indicated that somewhat resembles what appears on the web

e.g. with preferred_place_id set to 1522 (Montgomery County, US, PA), a search for persicaria perfoliata yields:

image

Note: listed_taxa only includes the first 100 of the total 509 records in the https://api.inaturalist.org/v1/taxa/131249?preferred_place_id=1522 results. So the fallback kicks in and the county checklist is just described as “Checklist”.

Compare that to a query for mourning dove, with my preferred place set to NS, Canada:

image

Here, there are only 89 listed_taxa, and Canada is near the top in any case, so the label can be more informative.

Listed taxa are a giant mess so I wouldn’t rely on them too directly. /v1/taxa/autocomplete is, as the path would suggest, meant to enable autocomplete interfaces, not to retrieve details about a taxon. /v1/taxa/:id only includes listed taxa that show unique combinations of place and establishment means, not all listed taxa, b/c it exists to show details about that taxon, and multiple listed taxa for the same place with the same establishment means are… essentially the same (this is part of why listed taxa are such a giant mess).

All that said, the inconsistent results you’re seeing are definitely buggy, and I think I see a way to fix that, even if I can’t seem to replicate the results you were getting after 70+ requests.

1 Like

Yeah, I didn’t go about using /v1/taxa/autocomplete to get that info. It’s just what I ended up using to accurately match what the user types to search for a taxon. Using /v1/taxa?q=<query> was too prone to result in the topmost item from the search not matching their expectations, whereas /v1/taxa/autocomplete gave better results for that. Since autocomplete doesn’t give all the info needed now that I have added establishment means and conservation status to the bot displays, Dronefly will now also do a hit against /v1/taxa/:id to get whatever was missing from the first call.

As a side issue, the conservation_status JSON responses are a bit of a mess, too, because /v1/taxa/autocomplete returns the status_name but not the url, and I want both. Conversely, /v1/taxa/:id returns the url but not the status_name. So when the bot does retrieve both, I hacked my way around it by synthesizing the result from both calls. Unfortunately, there are circumstances where the bot retrieves a taxon record without a text query, and in that case it’s only using the id#. In such cases, my displays degrade from saying that polar bears are “threatened (T)” in the US to saying that they are “T” in the US, because it’s missing status_name. But perhaps discussion of this issue really should be split off into a whole other post.

I’m encouraged to hear you have ideas about fixing the original issue though. Looking forward to seeing the results of your efforts.

1 Like

I have posted about my side issue here, so any followups about paragraph 2 can go in this thread: https://forum.inaturalist.org/t/how-do-you-get-all-the-conservation-status-fields-for-a-taxon-for-a-particular-place/15486

I just deployed the change that I thought would make the results more consistent, so if you’re still getting inconsistent results, please paste in your curl command and let me know how many times you had to run it before getting an inconsistent result. I think I was running it in a loop for 30 iterations or something and never got anything other than the “NatureServe Canada” list.

So far I have repeated my curl command several times by hand and I have not seen anything but NatureServe Canada. I have left it running for 1000 iterations with 1 second between. I’ll post my results when it is finished.

Update: the 1000 iteration run always returned NatureServe.

I’m puzzled, though, as to why NatureServe is the result and not Canada Check List, which is the primary checklist for taxon_id=3454. I mean, I don’t care strongly one way or the other, but I’d have expected the primary checklist to be used.

For the purposes of that endpoint, we’re just trying to show a unique establishment means for each place. To achieve consistency, I just chose the earliest ListedTaxon in cases where there were several with the same place and establishment means. We could use the primary_listing attribute instead, but it’s not going to make a difference in terms of functionality.

1 Like