I have a custom shapefile, how I can I query the API for obs within that complex shape?

Making enough bounding boxes to get a somewhat precise approximation would mean a huge amount of requests. Maybe do the opposite, one singular outer bounding box then write some script to check whether an observation within that bounding box has coordinates within your shapefile?

Edit: Also, https://forum.inaturalist.org/t/new-rectangle-and-circle-drawing-tools-for-web-explore-page/36684/32 might help you