@tiwane - can this training and observation entry be modified somewhat? A real challenge for those of us looking at insect data is the lack of life stage information - this applies to other annotations of course, but I will use insect life stage as the example for simplicity.
Only approximately 25% of global Lepidoptera have a life stage annotation (based on a filter from the identify page, so I hope this is correct, but it feels right anecdotally). That may not sound like much, but there are over 2 million global butterfly and moth observations currently. So, in raw numbers, thatās 1.5 million observations without life stage information, and with little likelihood of ever being assigned life stages at this point. Incidentally, the 25% figure is supported a bit by volunteers annotating other peopleās observations in projects like Caterpillars of Eastern North America and Moths of Ontario, so it could be 4-5% lower without those efforts. Clearly, the manual system doesnāt work very well, and I expect that this will be a growing problem numerically. From a user perspective, it can be done in bulk on computer entry, but it is a pain to even try it on phones in my experience.
So can you train the CV to recognize an adult monarch separately from a monarch caterpillar, and a monarch pupa? It is very good at identifying larger larvae already in North America. The input screen could then offer a suggestion like āweāre pretty sure that this is a Monarch caterpillarā, and automatically annotate or assign Insect Life Stage Field to the observation. The CV system correctly identifies many larval stage stage insects, so this is likely possible if you can classify the training images properly. This may not work for all species and life stages, but if you could move the life stage percentage from 25% to 60% or 75%, that would be a huge improvement in data quality.
It seems like a far more efficient workflow to enter that data as an observation is created. It must be more efficient than hoping people are going to individually open and annotate 1.5 million+ observation records.
The challenge is similar to the challenge of identifying to species. A body of known images is needed to train so the manual tagging of images with life stage is still needed; first to build the training set and second to confirm the effectiveness of the training by verifying the machine predictions :)
I am optimistic that the situation that you describe with the lack of data is not insurmountable - the magic of cooperation is everywhere in iNaturalist.
Let me just preface this by saying Iām not a tech person, so someone like @alex or @loarie would be able to give a better answer, but I think @marykrieger is right in that we need the annotated data in order to train the computer vision model and tell it which photos are of larvae, which ones are of adults, etc. Right now we feed it photos from, say research grade monarch observations but it only knows those are all monarchs, not at which life stage each photo is. So it would have to be re-trained on the annotated photos, which is a huge job.
So just as the species-level training is drawn from the millions IDs added by humans, I think any sort of life-stage level training will have to be drawn from the hard work of humans. Which always makes me feel humbled by the amount of work the iNat community does and thankful for you all. :-)
I like the idea of having the AI automatically add annotations. My desire would be that it (one day) auto-annotate Plant Phenology with flowering and fruiting, as @tsn is very good at helping me by always adding these annotations (which I canāt do from the Android upload app), or I add the annotations later, (at ~200/day, I always forget a few). Many plants should already have a huge corpus of obs marked āfloweringā and equally many that arenāt. Humm bit of an odd case, many obs are flowering, but are not so marked, so that might be a confusing training setā¦
I saw it mentioned on another thread (https://forum.inaturalist.org/t/lets-talk-annotations/627/34) that itād be cool to have other plant options like āvegetativeā and ādormant.ā I suspect that would be helpful if the CV is going to be trained on plant phenology so that āunannotatedā truly means that and not ānone of the above appliedā
Annotations are a fairly recent thingā¦ so many older observations donāt have annotations yet. Also, for Leps I will always put annotations for life stages other than adult.
Given how problematic AI has been for species identification, do we really want that unleashed on what is actually a fairly ācleanā part of the data?
If something like this were to be done, it would be better to wait until or if individual photo level annotations are implemented. Something needs to stop cases where say I post a cardinal observation with 2 pics 1 male and 1 female, say the male is the 1st.
Do you want pic 2 classed as male, the annotation left blank or not even attempted to be completed as an example.
Depends on your definition of ārecentā: annotations were introduced in the summer of 2017 so we are coming up on two years. I wouldnāt say that is recent. My point about the old observations is that the unannotated list is large, but if 75% of all new observations are also unannotated, and the number of observations is growing, then you will have a growing backlog. I bet if you could rewind to 2017, the list of unnatotated leps was 500,000 or something, and a million last summer (Iām making up the figures, but the growth curve is the point). So even with annotations around, the number keeps multiplying.
Chris - if we wait, I fear that nothing will get done and hundreds of thousands, if not millions, more observations will be entered without the benefit of useful data supplements in the form of annotations. So if the male/female thing for birds is an issue, maybe that is a second phase rollout. But hopefully we can work on the unambiguous issues first, possibly to get a process sorted out.
As mentioned in the Lets talk annotations and Data uses cases threads, the ability to download annotations is also a requested feature. Speaking from a data userās perspective, iNatās insect observations are problematic because of the inconsistent application of life stage categorization, so a development like this could really address what some believe is big problem.
No doubt itās a huge job, but isnāt a technology and workflow solution better than any other option? And if I recall correctly, the CV is retrained on a full dataset, implying each new dataset is much larger than any earlier version.
I guess my concern from a technologist perspective is that rule #1 of any work is to not make the problem worse, or to not simply introduce a new problem which replaces the one you are trying to resolve.
I see your point, but the math would seem to indicate that you could reduce the absolute number of unnatotated observations hugely by dealing with the simple, single-variable observations. The current solution of hoping people will manually deal with it is so lacking in efficacy that a new direction is needed.
If you want to fill a bucket by catching rain in it, you donāt have to catch every single drop. Annotations āadd value to an observationāā¦ but itās not a requirement for anything.
Mark - the data that annotations can provide is valuable to many people (but possibly not to you). Please donāt diminish other peoplesā/groupsā interest in better data.
The joyful news is that the main treasure of iNaturalist is the data set itself. The fact that it is growing in size and complexity means that it is becoming a richer and richer resource for exploring the capacity of computer vision to solve problems - not only problems related to the interests of iNaturalist users but also problems facing those interested in advancing the sophistication of computer vision. As long as the data set exists, it can be reanalyzed and re-categorized. I have been very grateful for this particular ability while the current winds of change are blowing so strongly through the world of taxonomy. :)
This discussion is doing a good job of outlining some of the problems of interest to the computer vision side of things.
how to decide which images are most useful to use in training sets and how to predict how many images are needed for effective training
what to do when images contain multiple subjects that are the focus of the training goals
how to balance multiple training goals
As computer vision continues to develop in sophistication, I expect that more ideas about how to solve these more technical problems will emerge. Hereās a link to the iNat Challenge on the website of the Sixth Workshop on Fine Grained Visual Categorization https://sites.google.com/view/fgvc6/competitions/inaturalist-2019 (warning: this link will contain computer jargon, similar but not identical to the scientific jargon you may already be accustomed to). Part of the attraction to participating in these competitions is the access to portions of large complex datasets like iNat to test out approaches and theories for improving the accuracy and efficiency of machine learning and visual categorization.
I agree that the annotations discussion is a valuable one - and that part of the rewards of working on the annotations problem will be the eventual prize that the long slog of updating annotations could be supported by computer vision aided suggestions.
This could also be a good task for new users/non-experts. I bet there are a lot of people who would like to help improve the data but donāt have the expertise to do species ID. If you encouraged people to help with this (maybe on a project page) and included a link to some basic instructions, I would think even users with little or no experience could correctly fill in this data.
Insect life stages seems pretty straightforward and computer vision would have much higher confidence between them than distinguishing species. Run it automatically and allow users to upvote or downvote the computer vision suggestions.
Observations that show an egg, larva, then adult, or male, then female, in separate photos fail the criteria of what an observation is anyway, but only running it on observations with a single photo would be one step to take in limiting the generation of additional issues.
To me this is still illogical or poorly thought out. When I go to the provincial park where I will be tomorrow, I will likely see 200+ Yellow Warblers. Iām not recording 200+ observations, or even one for every separate individuaI I photograph.
Of course if I record 1 record and enter a count observation of 200, Iām breaking a different āruleā because not all 200 were seen at that exact to the 6 decimal points coordinates.
We could certainly train a computer vision model to predict annotations like life stage rather than species. But remember to train the model, weāll still need lots of training data so the more life stage annotations you can add the better the model will be.