Ah, interesting. I’m not doing any PUT to /observations/:id; I was testing with DELETE /observation_photos/:id. But it seems I got muddled during my testing, because deleting the observation_photo does indeed delete the photo:
russell@zip:~ $ curl --header "Authorization: $JWT" --header 'Content-Type: multipart/form-data' https://api.inaturalist.org/v1/observation_photos -F "observation_photo[observation_id]=153292740" -F file=@_DSC1365.jpg
{"id":247119772,"observation_id":153292740,"photo_id":264879492,"position":null,"created_at":"2023-04-03T13:39:22.911-07:00","updated_at":"2023-04-03T13:39:22.911-07:00","old_uuid":null,"uuid":"afeb8635-3144-4269-8f02-8bfde80c007c","created_at_utc":"2023-04-03T20:39:22.911Z","updated_at_utc":"2023-04-03T20:39:22.911Z","photo":{"id":264879492,"user_id":4590187,"native_photo_id":null,"created_at":"2023-04-03T13:39:21.344-07:00","updated_at":"2023-04-03T13:39:21.344-07:00","native_page_url":null,"native_username":null,"native_realname":null,"license":3,"subtype":null,"native_original_image_url":null,"uuid":"418a12fb-83f4-4d6b-ac9c-46b52864c965","file_extension_id":null,"file_prefix_id":null,"width":1600,"height":1280,"license_code":"cc-by-nc-nd","license_name":"Creative Commons Attribution-NonCommercial-NoDerivs License","license_url":"http://creativecommons.org/licenses/by-nc-nd/4.0/","attribution":"(c) rcloran, some rights reserved (CC BY-NC-ND)","type":"LocalPhoto","square_url":"https://www.inaturalist.org/attachment_defaults/local_photos/square.png","thumb_url":"https://www.inaturalist.org/attachment_defaults/local_photos/thumb.png","small_url":"https://www.inaturalist.org/attachment_defaults/local_photos/small.png","medium_url":"https://www.inaturalist.org/attachment_defaults/local_photos/medium.png","large_url":"https://www.inaturalist.org/attachment_defaults/local_photos/large.png"}}%
russell@zip:~ $ curl -X DELETE --header 'Accept: application/json' --header "Authorization: $JWT" 'https://api.inaturalist.org/v1/observation_photos/247119772'
""%
russell@zip:~ $ curl -v --header "Authorization: $JWT" https://www.inaturalist.org/photos/264879492 -o /dev/null 2>&1 | grep HTTP/1.1
> GET /photos/264879492 HTTP/1.1
< HTTP/1.1 404 Not Found
It does look like the photo URLs in the POST response are not correct; I’ll report a bug for that, but, it also looks like the objects aren’t properly deleted out of S3 similar to what you reported for observation deletion before.