Observation Field Viewer

[stockslager]

While working on the Virtual Seed Library idea, I wound up creating an observation field viewer (used to show insect activity on plants offered through the library). The viewer requires a project_id and/or user_id and a comma delimited list of obs_fields.

https://stockslager.github.io/iNat/configurable_obs_field_table.html?project_id=133007&iconic_taxa=insecta&obs_fields=7623,498,4935,7807,254,1711&ofield_iconic=47126&ofv_datatype=taxon&operator=merge

In this case, I’m also telling it to “merge” the observation fields. The reason for this is that I’ve been bad at normalizing my own data! I use all sorts of fields when I see an insect on a plant. Was it nectaring, pollinating, resting, eating, or what? What was the insect doing there? Sometimes I’m not sure and don’t really care but it’s worth noting that for whatever reason, that insect was visiting that plant.

So… it’s useful to be able to merge and un-merge the data. If I navigate to the link above and click one of the obs_field id’s at the top, it filters out everything other than observations with that one obs_field. In this case, I clicked “498 - Nectar Plant”. This allows me to see only those instances where I chose to add a secondary observation where I perceived that the insect was nectaring on a plant…

Because I’ve now filtered out everything other than that one observation field (nectar plant), I can click a plant in the “Nectar Plant” column and only see the list of insects observed nectaring on that specific plant. In this case, I’ll choose “anise hyssop”. And now I can see all the insects I’ve perceived to have been nectaring at anise hyssop…

I can then select a specific insect and only see instances of that specific insect nectaring at that specific plant. All of this data is reachable through the iNat front-end but is organized and presented differently. It’s not realistic for iNat to imagine every possible scenario for presentment of inherently customizable user defined observation fields.

** alternative presentment is like the ropes hanging off the goodyear blimp. they’re probably important for something, but they aren’t the blimp. this is the core/custom model. inat is the core engine but maybe there should be endless customized presentment based on any specific use case.

[stockslager]

Field Study use case for Observation Field Viewer…

Scenario…
Ecologist at a nature preserve asks visitors to document the roosting location of bats in a preserve. The project requires a single observation field… roost type. The roost type obs field is displayed immediately alongside each observation in the table.

https://stockslager.github.io/iNat/configurable_obs_field_table.html?project_id=219480&obs_fields=18588

---

Since the most recent observation was of a “northern little yellow-eared bat”, that user can click that species name and see each observation of that species with the “roost type” listed immediately alongside each observation without drilling down into each observation to see the contents of the OBS field.

https://stockslager.github.io/iNat/configurable_obs_field_table.html?project_id=219480&obs_fields=18588&ofv_datatype=taxon,text,numeric,date,time,datetime,dna&per_page=50&taxon_id=75219

---

The “roost type” can also be chosen from the first link above which shows all the bats observed roosting in the preserve at a particular type of roost site… in this case, a man-made structure

https://stockslager.github.io/iNat/configurable_obs_field_table.html?project_id=219480&obs_fields=18588&ofv_datatype=taxon,text,numeric,date,time,datetime,dna&per_page=50&field:Roost%20type=manmade%20structure/objects

This makes the observation field viewer a decent companion to a project being run at a nature preserve. Contributors to the project wouldn’t need to drill down into individual observations to see the observation fields captured for each observation. They’d see them alongside each observation in the table shared via the “about” or in a journal post.

[cs16-levi]

Thats really interesting! This will be useful for entomology projects used to pinpoint species preferences.

[stockslager]

Use Case ~ Normalizing Interactions Data at a Nature Preserve

Scenario…
Visitors to a nature preserve have chosen to use all sorts of different observation fields when an insect has been observed visiting a plant. The preserve would like to make it easier to search and filter these interactions but the inconsistency in the observation field data makes it difficult.

They can use the viewer without passing in any obs_fields while specifying

&project_id=133007 // the project Id for the nature preserve’s biodiversity project
&iconic_taxa=insects // the iconic taxa whose observation fields they’d like to normalize
&ofv_datatype=taxon // the data type of the observation field
&operator=merge // telling it to merge the OBS fields if there are eventually more than one

The viewer would show all insect observations that have an observation field of type taxon…

https://stockslager.github.io/iNat/configurable_obs_field_table.html?project_id=133007&iconic_taxa=insecta&ofv_datatype=taxon&operator=merge

If they click the OBS photo for the first observation in the list, they can see that observation in inat and also see the observation field used for that observation… “plant that the organism was found on” (7623). This observation field is added to the obs_fields param in the url (&obs_fields=7623).

The viewer would show the contents of that observation field for any observation that uses 7623 (plant that the org was found on).

https://stockslager.github.io/iNat/configurable_obs_field_table.html?project_id=133007&iconic_taxa=insecta&ofv_datatype=taxon&operator=merge&obs_fields=7623

Scrolling down through the list, some of the rows don’t have a plant name populated because a different field was used.

Clicking the OBS photo and looking at the inat observation for the first one shows that the 498 - Nectar Plant observation field was used. 498 is then added to the obs_fields param…

https://stockslager.github.io/iNat/configurable_obs_field_table.html?project_id=133007&iconic_taxa=insecta&ofv_datatype=taxon&operator=merge&obs_fields=7623,498

If the entire table is scrolled… looking for blanks and adding the observation fields for those observations, the nature preserve has a consolidated list of all insect visitations to plants.
https://stockslager.github.io/iNat/configurable_obs_field_table.html?project_id=133007&iconic_taxa=insecta&obs_fields=7623,498,4935,7807,254,1711&ofield_iconic=47126&ofv_datatype=taxon&operator=merge

This also allows them to see which observation fields are thinly used within the preserve and which observation fields might be synonymous with other observation fields. Allowing them to suggest to visitors the use of consistent fields for capturing “insect on plant” data. This could be done by simply linking to the observation field viewer from their biodiversity inat project for that preserve. Most observers would probably want to be on the list… and would use the obs fields that would trigger their observations to appear there.

If the preserve added “plant that the organism was found on” to the observations using “nectar plant” it might seem redundant, but would allow filtration based on insect name and plant name since the data in the table wouldn’t need to be merged… https://stockslager.github.io/iNat/configurable_obs_field_table.html?project_id=133007&iconic_taxa=insecta&ofv_datatype=taxon&obs_fields=7623

Nectaring, pollinating, and eating are somewhat mutually exclusive. In each case, the insect was visiting the plant (albeit with different intentions). They’re somewhat hierarchical. Each preserve might look at them differently but normalization of the data within the preserve might give human visitors a richer experience. They’d more easily see how data they’ve collected fits in with data collected by other observers.

[jwidness]

Can I put in a couple requests for https://stockslager.github.io/iNat/observation_fields/observation_fields.html ?

On the query page, can you have the datatypes be checkboxes? Right now you can only select one option at a time, or you can choose all, but the API can handle any combination, e.g. text and taxon.

(mockup)

And on the results page, can you have the list of observation fields show a count of times used, plus sort them by count?

(mockup)

[stockslager]

These are both really good ideas and they can both definitely be added in. However…

I came up with this for selfish reasons. I wanted to compare my insect on plant observations to other people’s insect on plant observations. The problem being… we were using different observation fields. Basically, everything beyond the “merge” functionality was an afterthought.

If I want to see insect activity on a specific plant but my team of observers at my nature center has all used different observation fields… (nectar plant, name of associated plant, plant that the organism was found on)… I’d choose type of “taxon”, enter the project for that preserve, choose all the relevant “insect on plant” observation fields, and…
visitors to all plants at my nature center

The problem is… they’re still in different columns which makes the data confusing to look at. Like, if the nature center wanted to share the data collected by the visitors with other nature centers, they’d want the columns merged into one. So after the query executes… there’s a “Merge” link that appears next to the obs field drop-down if they’ve chosen obs fields of the same type. If the person preparing the query using my tool chooses “Merge”, they see the columns merged…
visitors to all plants at my nature center (merged)

Here’s my dilemma… when I left programming to raise two kids 18 years ago, my company was just beginning to consider using JavaScript in its front end because of its asynchronous transactions. Rate limited fetching would never have flown in that era. I still think it sometimes places an unnecessary burden on the servers if used in the wrong way. The problem is… I’m not sure if this is the wrong way or not. I think it might be. If the nature center shared the last link I sent with its 18,000 members and 9,000 of them clicked the link… how many fetches would that be?

Basically, this concern nudged me toward considering how we would have handled this back in the day (the late 90s). At that time, the strict rule was… one transaction per click.

The one transaction per click rule forces data normalization. Basically, if I want to see merged fields, I better go ahead and merge them… not execute multiple transactions on someone else’s server (even if they said I could). So… I made sure that any observation at my nature center had a generic “found on” observation field populated (I looked at each one so i’d be sure that it wasn’t the obscure case of… a humming bird was sitting on a tree branch of one tree and nectaring from a neighboring plant). Anyway… this created a consolidated view using “plant that the organism was found on”. This consolidated view, created by my reluctance to fully embrace this new era of rate limited fetching, enabled an alternative UI flow. Since I’m now sure that the relevant observations within my nature center have consolidated plant / insect interactions under a single observation field… I can build a simple viewer for observers at my nature center. Which I have done…
Nature Center Garden Viewer

All of my work borrows heavily from @pisum 's examples. Basically, the app is just a condensed version of many of his scripts that are connected together to create a flow. The point is… I can share the little app with the nature center knowing that each click is a single transaction rather than kicking off a series of rate limited fetches.

I’m not opposed to working on the changes you’ve requested… but I look at the obs field viewer as a utility that supports the other (arguably more important) effort.

[jwidness]

I might be misunderstanding your concerns, but if your current version is within API recommended practices, and you can do the sort/count client-side, without any additional API calls, I’m not quite sure where the issue is?

I was also definitely imagining the sort/count as supporting the goal of a workflow to merge observation fields – if users can see which observation fields are most commonly used, they could prioritize merging the handful that capture most of the data, and ignore the long tail of fields used only once or twice.

[stockslager]

@jwidness

I was also definitely imagining the sort/count as supporting the goal of a workflow to merge observation fields

This is helpful. Yah, I’ll make the sort/count of obs fields in the utility my top priority.

---

All I was trying to point out is... In my little follow on application... when a plant is chosen I either need to -or- the observation fields (which I can't do without violating the api rules since I'd need to do one transaction for each observation field). Instead, I created the utility to see which users collected plant data in a relevant observations field... merged them into "found on" manually, so the app can present the data with one transaction. The counts page can further filter the "found on" counts since observation fields can be "and'ed" together. So a drop-down on the counts page could present "found nectaring" or "found eating" or whatever. Sorry if confusing. I do think it's important to be able to present the data dynamically, rather than expecting each nature center to take an image of the data. The app is driven by a .json... so it'd be pretty easy to configure the app to point at any nature center with an iNat project. initially it was driven just off url parameters but if I don't make it project based and driven off a .json... I don't think there'd be enough incentive for each nature center to participate in helping consolidate whatever the local thinking is on the collection of data in consistent fields.

[stockslager]

Added sort count. here’s what it looks like for the OBS fields you’ve used. Appreciate everything you do for the forum and iNat more generally. If sort/count only gets me partial credit and I need to work on the multi-select for OBS field type. lmk. :)

[jwidness]

Thanks very much for the quick update!
I’m not sure the sort is working though – this is what I got from a query for all datatypes with taxon_id=11867&place_id=1

Another option for the multi-select would be to allow the user’s query to override the drop-down selection, e.g. instead of having this return all, let the user’s preference for only text and numeric go through:

(The use case I’m imagining is that you want to know what fields are being used to record count and some of them are numeric and some of them are text, but none of them are date or dna, etc.)

[stockslager]

@jwidness

taxon_id=11867&place_id=1

my mistake. should work now.

yah, that’s a perfectly reasonable use case. I can already tell you envision using it with a much broader scope. I tend to think in terms of it being a project based tool. I even thought of requiring a project_id to use it… just to force them into going against a smaller dataset right at the outset… which seems less intimidating. i’ll update it do the multi-select… makes perfect sense to let the comma delimited list override the drop-down… especially if iNat gets wild and adds another type!

[stockslager]

Here’s a sample project, and the same tool (but with paging… to prevent the rate limited fetching since it’s embedded in a UI flow). I consolidated what I could (before creating the “field studies”), but the larva example has a lot of empty fields because I can’t use ofv_iconic_taxa to limit the results from the endpoint. (the larva example is showing all the observations where a larva was preying upon a snail that could be filtered out if I could pass in the ofv_iconic to the obs end point).

iNat project…
https://www.inaturalist.org/projects/firefly-larvae-on-plants

Field studies using manual observation field consolidation guided by the utility (as well as a similar version of the utility that uses pisum’s paging rather than rate limited fetching)…
https://stockslager.github.io/iNat/nn/admin/dashboard.html?params=firefly

The second half of the solution requires a .json to tell it which observation field some organization decided to consolidate under… but it’d be easy to open it up again.

Basically… the client side “merge” could get a lot closer to correct if the end-point allowed the ofv_iconic_taxa to be passed in and filtered results returned based on it. Otherwise, you’re doing rate limited fetching or requiring every project that wants cleaner results to take an image of the data (or… waiting on a Lampyridae expert to decide which obs fields to consolidate into and use the meta data tool to consolidate).

[wy_bio]

@stockslager

The one transaction per click rule forces data normalization.

I’m building a site to explore iNat data. Instead of using one transactions per click, I decided to use the iNat website as a guide about how to use the API on my site. I used the browser developer console to examine the network requests on various iNat website pages.

I looked through some of your code. Have you’ve considered using iNat V2 API instead of V1? To me the biggest advantage of building an app with V2 api is the ability to set what fields are returned in the response. iNat website uses V2 on the Explore page, so I did the same.

I took a different approach to observations fields. Instead of building a viewer specifically for observations fields, my app adds observations fields to “Filters”. People can select observations field names, field values, and field type, and combine it with any of the other filters. The app also displays the all observation fields for each observation.

Here’s the results for “Project Firefly Larvae on Plants” with ofv_datatype=taxon.

https://inat-explorer.dataexplorers.info/?project_id=233336&ofv_datatype=taxon&view=observations_observations&subview=grid

I think your approach is better for use cases where organizations want to show their data without having end users to do custom searches. I think my approach is better for use cases where people can do custom searches. Both approaches are valid; it just depends on what type of experience you want the users to have.

[stockslager]

Thank you for the nudge. Yes, sometimes I wake up in the night in a cold sweat worried that the v1s will be shut off before I’ve switched to the v2s! You’re absolutely right. It’s interesting to me that the v2s allow you to limit the data structure returned based only what you need (seems that would mean i could cache data on the client in the structure it’s returned in without having to build my own helpers). Anyway… yes, you are correct… at some point I need to look at all that.

This code was written with very narrow focus and a very specific use case. I didn’t even envision passing anything over to the meta data tool since I was mostly thinking of using it to consolidate fields for a single project. If I were going to use it to consolidate using the meta data tool, I’d make some changes…

1 ~ I’d add a without box. When the observation fields are selected from the drop-down, they’re displayed in a scrollable box in the upper right above the table of data. There is currently no way to say… show me this set of observation fields, but don’t show any row that already includes this observation field (the one I intend to consolidate under). So I’d want to add a “without box” which shows its own scrollable list.

2 ~ Add “without?” url to the column header for each observation field shown in the table of data.

3 ~ When “without?” is clicked for a column displayed in the table, that observation field would appear in the “without box” above the table and next to the “with box”.

4 ~ Any row within the table that includes an observation field in the “without” box would become hidden.

5 ~ A new url would be added which sends the current table of displayed observation id’s into ID flow where they could be manipulated by the meta data tool.

Something like that anyway. I dunno tho… maybe it’s ok as/is and the without stuff would just make it confusing.

[wy_bio]

No rush to switch to V2. API V2 is in alpha, and the docs say “UNDER ACTIVE DEVELOPMENT, USE OF THIS ALPHA RELEASE NOT RECOMMENDED”. I use a mix of V1 and V2 in may app. If the iNat site uses V1 for a feature then I use V1, if the iNat site uses V2 then I use V2. Since your app displays so few fields, using V2 to set the fields could be helpful.

You mentioned reducing the number network calls. I’m guessing the iNat devs added all the nested records (annotations, users, etc) to V1 observations endpoint in order to reduce the number of network calls. But each observation ended being a huge chunk of data. For V2, the iNat devs added the fields params so that devs can fetch only the data they need for the page.

@stockslager

I dunno tho… maybe it’s ok as/is and the without stuff would just make it confusing.

Scope creep is never ending problem in software development. I vote to keep things simple.

@stockslager

So… I made sure that any observation at my nature center had a generic “found on” observation field populated (I looked at each one so i’d be sure that it wasn’t the obscure case of… a humming bird was sitting on a tree branch of one tree and nectaring from a neighboring plant).

Are you concerned that by asking people to just use one field, you might end losing some subtle differences that matter to the people collecting the data? I would rather have multiple observations fields to capture the full range of information and make multiple API calls, rather than use one observation field and loose information in order to make one API call. In an ideal world, the iNat API would have AND / OR option when querying multiple observation fields and annotations.

There is a feature request for adding an OR option to query observation fields.
https://forum.inaturalist.org/t/use-or-boolean-logic-in-url-searches-for-observation-field-values/47924

[stockslager]

@wy_bio

Are you concerned that by asking people to just use one field, you might end losing some subtle differences that matter to the people collecting the data?

I’m actually not asking them to do this. I’m trying to capture additional information. Just as an example…
Nectar Plant &
Host Plant

In both cases I can be fairly sure that organism 1 has been found on organism 2. What I can’t see is the aggregate. This is a shame because there may be new associations that become obvious just by an understanding of how often organism 1 is found on organism 2. Was an insect presumed to always be “resting” on a plant without any realization that it has a strong preference for that plant (or that there is actually something else going on there in need of closer examination). Also, I’m just drawn to the more holistic view of things. What is the overall value of planting native plants… how many critters do you find using them if you plant them? In order to get at this more holistic / aggregate number, I need to merge the fields.

Scenario…
A kid is at a nature center. He participates in all the nature center’s programs, attends some of the summer camps there etc. He’s just starting to develop a deeper interest in earth sciences (maybe Ecology). The kid isn’t sure which way to capture data in observation fields because there are so many fields and some of them are very specific. The ecologist at the nature center just tells the kid… add “plant that the organism was found on”. At night, I’ll see it pop up in the app and I’ll help you fill in the other fields (was it nectaring, was it pollinating, was it eating, is it a larva). In this case, the ecologist would be helping a kid they know better understand the natural world. The ecologist would also be onboarding the kid onto iNat and possibly leveraging the kid to help collect data for a very real field study. If that same ecologist could designate a “list” of plants (possibly by phenology… those in bloom at any given time at the nature center)… they would be directing that kid toward specific plant colonies that warranted more investigation.

In this scenario… there would only be a need to store the secondary taxon once (for found on). The behaviors would become text based yes/no. Over time… the kid would be better onboarded by the ecologist and would populate more and more of the yes/no questions about behaviors. In field study scenario, the ecologist would ask the kid to help collect a packet of observation fields reflected back to the kid (and ecologist) similar to how I have it set up in the firefly example above.

The first page of my app allows designation of a “list” of plants… I’m mostly using “garden list=yes”. The wildlife activity documented for each plant can come from a network of observers at the nature center. The species counts page would eventually have a second drop down for “behavior”… what was the insect doing while there?

What I really want is some ecologist at some nature center who wants to try it. I’m not an ecologist. I’m not even sure I’m tech support. But maybe?

[stockslager]

The first page of the app allows designation of a “list” of plants… I’m mostly using “garden list=yes”. The wildlife activity documented for each plant can come from a network of observers at the nature center. The species counts page would eventually have a second drop down for “filters”… what was the insect doing while there?

The drop-downs would all be configuraable (via the .json) by whatever organization is hosting the app for other organizations in that same region. I realize it’s really simplistic, but iNat is really good at broad and complex functionality… they handle pretty much everything. I think this makes customization even more important sometimes. I dunno… just an idea to explore.

The ecologist would potentially use the observation field viewer (the utility) to see if observers within the preserve were using new granular observation fields without the aggregate… or aggregate observation field without the granular… and could invite those observers to join with the promise that their data would be reflected in the custom app for that preserve.