There are all sorts of ways to string together the search terms, including the observation fields (useful tips at https://forum.inaturalist.org/t/how-to-use-inaturalists-search-urls-wiki-part-1-of-2/ and https://forum.inaturalist.org/t/how-to-use-inaturalists-search-urls-wiki-part-2-of-2/). If you wanted to look for Daviesia observations that had the Fruiting Phenology annotation and this observation field, it would be pretty straightforward, but there currently isn’t a way to exclude observations with a particular observation field value.
However, I think you could do that search and download the list of observation ID numbers, then exclude them from your search of fruiting Daviesia using not_id. It would be something you’d have to do each time you want to update the search.