API query search results should ignore underscores used for markdown

Feature Requests
1

Platform(s): API

Description of need:

iNat’s query search strings include markdown chars like underscores (used for italicizing the scientific names on the website) which are returned as a string literal in the query. So in order to find results entered as alnus, _alnus xxx_, or _alnus_ you would need to do three queries even though the _ are not rendered text on the website itself. Like this:

  • www.inaturalist.org/observations?q=alnus
    Only finds results where alnus was typed, shown on the site as “alnus”

  • www.inaturalist.org/observations?q=_alnus
    Only finds any results where _alnus xxx_ was typed, even though the the site will show “alnus xxx”

  • www.inaturalist.org/observations?q=_alnus_
    Only finds any results where _alnus_ was typed, even though the the site will show “alnus”

Since inat caters to a community where italicizing latin names is common this seems like a noteworthy oversight for the inat API to be interpreting these markdown styling chars are part of the search string.

Feature request details:

q=word should return matches for word regardless of the use of _ for styling at the beginning or end of the string itself. These should all be returned for a search of q=word:

word
_word
_word_
word_

… this should include less common double underscores __, but I’ll stop with the list.

It should not exclude underscores in between alphanumerics like IM_A_RESEARCHERS_SPECIAL_CODE, or code_2. These should be preserved and searchable still.

I think this could be done with regex in the backend, but you’ll also need to sanitize query strings in the same way at this point, so a user still using q=_alnus will get the results they want for alnus once the underscore is no longer considered part of the string.

Edit - I updated this because I found that _ is the only char with this quirk. It looks like the search language uses UAX #29 (Unicode Text Segmentation) which treats _ as “Connector Punctuation” and ", ', *, and ` as “Punctuation” or “Word Boundaries”. So only underscores are being indexed as part of the words currently.