Edit project breaks entirely for project with undefined user

[benarmstrong]

Please fill out the following sections to the best of your ability, it will help us investigate bugs if we have this information at the outset. Screenshots are especially helpful, so please provide those if you can.

Platform: website

Browser: Chrome and Firefox

URLs: https://www.inaturalist.org/projects/discord-inaturalist-yearlisting-2020/edit

Screenshots of what you are seeing:

Ignore the CORS error from google maps API. I get those when editing other projects and it doesn’t break those pages:

Description of problem:

Step 1:

Edit the project (direct link to edit page given above). This reproduces the problem. The page is blank and the project cannot be edited. The console errors tell the story. I looked into the Sources tab in Developer Tools and part of the traceback at UserSelector.render is in the render of the “must be observed by user” rules. Apparently there’s a rule in there for a user that doesn’t exist, and instead of discarding the user, it tries to access the id property of it, causing the whole page to break.

[pisum]

i know it doesn’t address your bug, but i wonder if you get your project back to a working state by removing the invalid user via PUT https://api.inaturalist.org/v1/projects/{id}? (it’s undocumented, but i assume it should work.)

you could get the existing rules using GET https://api.inaturalist.org/v1/projects/discord-inaturalist-yearlisting-2020?rule_details=true.

[benarmstrong]

Yeah. I’ve wanted to do something with the iNaturalist dronefly account with that API call, but because it is undocumented, I didn’t want to start writing for it. Do you think I should? Is it possible it will be supported in a future API version?

As for getting the existing rules, I already have code that does that.

[pisum]

@benarmstrong

Is it possible it will be supported in a future API version?

i have no idea, but since the website it using it (and the page was relatively recently designed), i doubt it’s going to go away anytime soon.

@benarmstrong

Do you think I should?

i guess it just depends on how much time you have an whether you think it would be useful to handle this case and possibly future cases. just for example, if it were something like a time-sensitive bioblitz that i had heavily marketed, then i might devote some time trying to get my project unstuck sooner than later.

[benarmstrong]

My plans are longer term than just this broken project. Since the project is old (for 2020), the edit I wanted to make was just some cleanup and not overly urgent.

What we’re missing for those longer term plans is the ability to add or remove a single “must be observed by” rule at a time, and without tying up much time by our mod team on Discord. If we had such an API, full automation of adding users to events would be simple:

• user presses a button in Discord to register
• the bot checks if they are a valid event candidate
• if they are, the user is added to the event project

Without such an API, the process is:

• user presses a button in Discord to register
• the mod team periodically checks the list of pending valid event candidates
• if there are any to add, they are added by a member of the mod team manually on the web

[pisum]

ok. yeah, if that’s what you’re talking about, then i would probably go down that path of development. i would guess that undocumented functionality is subject to change more than documented functionality, but judging by the speed of change in general in the system, i think a little bit of development would help you for a while.

[benarmstrong]

Thanks for the encouragement. I had talked myself out of doing it last month and settled for just doing the semi-automated process I described, but I have a growing list of reasons to go forward with this. This discussion has convinced me to write the code around PUT /v1/projects/# to update our event project user rules. It’s just too painful to keep using iNat web to keep them updated. See: https://github.com/dronefly-garden/dronefly/issues/161#issuecomment-1057892084

[benarmstrong]

OK. doing this, and using breakpoints, I found the offending user, at least:

{"operand_id":1142264,"id":234161,"operand_type":"User","operator":"observed_by_user?"},

The rule doesn’t have a “user” attribute like all of the others.

I forgot for the moment that operand_id is the relevant id and confused the id for the user id. It’s not. The operand_id is, and that appears to be a deleted user.

[benarmstrong]

How much of the form payload is needed for a succesful PUT /v1/projects/{id}? Somehow, I’m not getting it … error 400 every time.

Example working code (at least it returns 200) that just does an authenticated GET:

import requests
from pyinaturalist import get_access_token

access_token = get_access_token(
    username="dronefly",
    password="REDACTED",
    app_id="REDACTED",
    app_secret="REDACTED",
)
response = requests.get(
    "https://api.inaturalist.org/v1/projects/545640",
    headers={"Content-Type": "application/json", "Accept": "application/json", "Authorization": f"Bearer {access_token}", "User-Agent": "Dronefly/1.0"},
)
print(repr(response))

This is fine and returns:

<Response [200]>

But if I give this as complete a payload as I can as a PUT based on the properties of a test project I created, I get error 400:

payload = {
    "project": {
        "project_type": "collection",
        "user_id": 545640,
        "title": "Moderated membership test",
        "description": "A project to learn about writing code to add or remove user observation rules without destroying a real project in the process.",
        "preferred_banner_color": "#74ac00",
        "prefers_hide_title": False,
        "prefers_hide_umbrella_map_flags": False,
        "prefers_banner_contain": False,
        "prefers_rule_quality_grade": "research,needs_id",
        "prefers_rule_photos": "",
        "prefers_rule_sounds": "",
        "prefers_rule_term_id": "",
        "prefers_rule_term_value_id": "",
        "prefers_rule_observed_on": "",
        "prefers_rule_d1": "2022-01-01",
        "prefers_rule_d2": "",
        "prefers_rule_month": "",
        "prefers_rule_native": "",
        "prefers_rule_introduced": "",
        "prefers_user_trust": False,
        "project_observation_rules_attributes": [
            {"operand_id":545640,"id":831633,"operand_type":"User","operator":"observed_by_user?","_destroy":False},
        ],
        "admin_attributes": [
            {"user_id":1474198,"role":"manager","id":1474198,"_destroy":False},
            {"user_id":545640,"role":"manager","id":1474187,"_destroy":False},
        ],
    },
}
response = requests.put(
    "https://api.inaturalist.org/v1/projects/545640",
    headers={"Content-Type": "application/json", "Accept": "application/json", "Authorization": f"Bearer {access_token}", "User-Agent": "Dronefly/1.0"},
    data=payload,
)
print(repr(response))

Then I just get:

<Response [400]>

I had hoped I could just submit the fields I wanted to change, so I started with a simpler payload that only included the project_observation_rules_attributes. However, that failed (same response: error 400), so then I tried the fuller payload specifying everything that would be on the form instead. Still no joy.

[pisum]

i’m not sure if this is what’s going on here, but one thing i noticed when making requests to other endpoints, i get errors if my payload contains extra trailing commas in the structure.

so something like this might not work:

{
  "record": {
    "item": "blah",
    "type": "huh",
  },
}

… but this would (or something like that):

{
  "record": {
    "item": "blah",
    "type": "huh"
  }
}

i didn’t actually verify this just now, but it was just an odd thing that i remember seeing, and your payload seems to have the same kind of potential issue maybe… i don’t have time to get into it right now, but i might be able to try a few things later.

[benarmstrong]

Because the trailing commas is just a style I use in constructing the python dict, I don’t think it makes a difference to the end result. i.e. with / without them, requests is still going to see the same dict …

Something I was investigating in Discord chat with @jcook was whether this endpoint strictly requires Content-Type: multipart/form-data. If it does, then I have a bit of a tricky encoding issue, because I think I need to flatten the nested dict/array/dict structure into something that can then be encoded like that. Everyone seems to have their own solutions (various things found on stackoverflow) and none of them seem to do the job for me. Or I could just be entirely wrong about that. I might just want to quit now and see if Jordan can dig up something for me later when he has time.

[jcook]

Alas, I haven’t gotten any farther than you have, aside from discovering all kinds of ways to not send these requests. I’ll make some more attempts tomorrow.

[pisum]

@benarmstrong

I had hoped I could just submit the fields I wanted to change

this appears to be possible. i was able to change a test project by including only the items that i wanted to change in the payload:

1. change the project description (to “test”):
   curl "https://api.inaturalist.org/v1/projects/pisum-s-personal-project" -X "PUT" -H "Content-Type: application/json" -H "Accept: application/json" -H "Authorization: JWT" -d "{\"project\":{\"description\":\"test\"}}"

2. delete a particular user (1 of 2) from the “include” list (note destroy=true, and does not work if project_observation_rules_attributes[i].id is excluded):
   curl "https://api.inaturalist.org/v1/projects/pisum-s-personal-project" -X "PUT" -H "Content-Type: application/json" -H "Accept: application/json" -H "Authorization: JWT" -d "{\"project\":{\"project_observation_rules_attributes\":[{\"operator\":\"observed_by_user?\",\"operand_type\":\"User\",\"operand_id\":779571,\"id\":831929,\"_destroy\":true}]}}"

3. add the user back to the “include” list:
   curl "https://api.inaturalist.org/v1/projects/pisum-s-personal-project" -X "PUT" -H "Content-Type: application/json" -H "Accept: application/json" -H "Authorization: JWT" -d "{\"project\":{\"project_observation_rules_attributes\":[{\"operator\":\"observed_by_user?\",\"operand_type\":\"User\",\"operand_id\":779571}]}}"

hope this helps. let me know if you need anything else.

[benarmstrong]

Argh. OK, solved [edit: to clarify, I mean this subproblem is solved … the underlying bug isn’t]. I had not obtained the JWT with /users/api_token from the password token. This post sorted it out for me: https://forum.inaturalist.org/t/how-does-one-turn-an-authorization-code-into-a-jwt/13528

And then my test worked in curl. I’m reworking my python code snippet to do it that way.

[pisum]

@benarmstrong

I had not obtained the JWT with /users/api_token from the password token.

here’s just a little more related info:

@kueda

Some endpoints at api.inaturalist.org pass requests through to www.inaturalist.org, and for those an oauth token will work (though not an invalid one). It’s not something I recommend doing though. Better to get a fresh JWT and use it for all requests to api.inaturalist.org that require auth.

[benarmstrong]

Ah, good. So by always passing a JWT, everything will just work.

I’m back to not being able to fix 2020 because of a bit of key-in-locked-box problem. If I could edit the project with dronefly account, I would, but dronefly isn’t an admin for the 2020 yearlisting project, being one of our older projects, so it can’t edit it - but until I can run some code from an admin account, I can’t add more admins!

I have applied to make a new application under my own account, which is an admin of that project, so I can finally fix it, but now I have to wait for that application to be manually reviewed and approved. Sigh.

Anyway, eventually I’ll get there and get the project fixed. :) Thanks for all your help!

[pisum]

@benarmstrong

now I have to wait for that application to be manually reviewed and approved

i think that approval process should be quick (or at least it was for me). if you just need a JWT to run a quick script to update your admins using your own account in the meantime, you can always just go to https://www.inaturalist.org/users/api_token in your browser and manually get one there. (or you could repurpose one generated by other means.)

[benarmstrong]

Of course! Thanks. The deleted user is now gone from the project, and edit works again.

[pisum]

great. i’m glad your specific project is now unstuck. i guess the general issue of deleted users in the “include” list of collection projects preventing the project rule edit page from loading properly is still a thing though. so bug still unresolved, in case it’s unclear for others looking at the thread.

[tiwane]

I tried to replicate this issue but couldn’t. Here are the steps I followed:

1. With tiwane, I made a collection project and included one of my test accounts (stafftest2, since deleted) as a User requirement.
2. With stafftest2, I joined the project and I also uploaded an observation that met the project’s requirements.
3. Checked that stafftest2’s obs was in the project and that stafftest2 was a project member.
4. Deleted stafftest2 account.
5. With tiwane, edited project and the edit page loaded fine.

Am I missing something?