Since May 2013 BowerBird has been the main “citizen science” web site for Australia. Unfortunately today it has been announced that it will receive no more funding and go offline when the current server lease expires in several months. Users have been suggested to switch to iNaturalist. https://researchdata.museum.vic.gov.au/BowerBird_Bugle/Edition_60.pdf
So long BowerBird… Long live iNat!
I submitted about 4000 sightings through BowerBird, most of which have made it to ALA but some unidentified and more recent stuff didn’t make it. [If you click the ALA link then on “Attribution” you see where my records have come from.] Apparently about 50,000 records never made it to ALA so unless those people resubmit to inat those will be lost forever. :(
There is no way to download your record data from BowerBird like there is for inat at least.
It nay be a bit of a hack, but if the data got into ALA and or GBIF I assume you could download data from there and use as a source for input to iNat. It would create a risk of duplication depending on their data stewardship processes on those downstream entities though.
Yes, data can be downloaded from ALA, but there are over 120,000 records on BowerBird and less than 70,000 made it to ALA. Records that made it to ALA should at least stay there and not vanish.
I am slowly moving as many Bowerbird records as I can over here, the ones that did not make it to ALA. Crunch time as there are several hundred affected ;_;
I’m sad to hear BowerBird’s technological troubles proved insurmountable. I’ve enjoyed reading the BowerBird Bugle newsletter occasionally since 2015. Around then I came across BowerBird and added it to this list of platforms like iNaturalist. I have always been so impressed with Ken’s entomological insights and enthusiasm. I think some of the anecdotes in the BowerBird Bugle are some of the best importance-of-museum-specimen stories I’ve read (you can find them all here). The Bugle also highlights amazing insect photography (including from @reiner!) and the value of associated records in great detail. Ken was a great advocate for BowerBird, and these kinds of projects really need deeply knowledgable champions.
It’s a shame about the orphaned data that didn’t make it to ALA. For easy reference, I’m copying the portion of the newsletter that lays out the struggles.
BowerBird was always offered for free and I have since learnt
that it costs money to offer something for free. The main yearly
cost of BowerBird is hosting it in the Amazon cloud. The
BowerBird programmers argued that it needed two virtual
servers – one for the software and one to store the images.
We initially tried using just one server for both software and
images but the website ran very slowly. So, hiring two virtual
servers is more expensive than hiring one. I have been able to
manage to secure sufficient funds to support delivering
BowerBird. However, the situation changed when software
problems occurred about 12 months ago. The first was the
decision by Google Maps to charge for the use of their maps –
previously Google Maps had been a free service. The second
software problem occurred with data uploads to ALA. Every
night, BowerBird creates a complete data dump for all of its
records. Then weekly on a Sunday afternoon, ALA would
hoover up these records and display the BowerBird dataset on
ALA. By uploading the complete BowerBird dataset meant that
any identification that had changed or been updated took only
one week to be corrected on ALA. The software programmers
created a data dump program that had an expiry time of five
hours. At the time of releasing BowerBird, that seemed like a
reasonable if not excessive time limit. As we have now
discovered, the five hour time limit times out at about 70,000
records but there are now over 120,000 records on BowerBird
and so not all data can be downloaded from BowerBird nightly
to be uploaded to ALA. This has been the case now for over 6
months.
I raised over $50,000 to fix these and other BowerBird software
issues – that was the amount the programmers told me was
needed to fix these issues. I paid the programmers these funds
but when they began working on the problems, they realised
the issues were much bigger than they had initially considered
and would cost at least double what I had paid. I was not able
to raise double these funds and so none of the BowerBird
problems have been resolved.
It took quite some time for me to be told there was insufficient
fund to resolve the BowerBird issues. So, I waited for problems
to be fixed but nothing happened despite assurances from the
programmers. Finally, the time has come to say enough.
I believe that the Google Maps and ALA data upload issues will
not be resolved. To me, a core component of BowerBird is the
data uploads to ALA where the data is permanently stored,
added to millions of other data records and displayed for all to
use.
I am now recommending that if you want your citizen data to
reach ALA, you should stop using BowerBird and start using
the iNaturalist citizen science website
(https://www.inaturalist.org/ ). Many BowerBird users have told
me that they have already made this change.
Finally, I have no idea how much longer the BowerBird website
will continue to be delivered. I believe it will stop sometime in
the next six months but I cannot get an answer from the
programmers on when BowerBird supporting funds with run
out.
This is all somewhat disappointing news for everyone involved
in some way with BowerBird. The website really has made
some amazing discoveries. On ALA for many species,
BowerBird records provide the only distribution points and
species images. Several fascinating magazine stories have
been published about BowerBird records.
We should all be very proud of the legacy that BowerBird
contributors have built and the many lessons that citizen
science has learnt from BowerBird.
Thanks from me to everyone involved with BowerBird. It has
now run its course but there are other websites where new
discoveries can be made and shared with the world. I do hope
that BowerBird users will now engage with other citizen science
websites and continue making valuable contributions about our
natural world.
Thank you all and happy photographing nature. Citizen science
is now a well-accepted and valued part of natural sciences.
BowerBird and your contributions have helped to secure this
status. Best wishes – Ken Walker
I went back a couple of issues in the Bugle to find earlier mention of the technological problems. In October 2018 Ken wrote:
The Google Maps issue
I’m sure you have all seen the recent Google Maps issue. If
you click the OK button you do actually see a map but that’s not
really a fix.
The problem has come about due to a change at Google. They
have provided free access to their Google Maps software for
over a decade and now they have decided to charge for that
service so they changed the code. The charges are very
reasonable as the first 25,000 maps per day on a website are
free. BowerBird currently has over 100,000 records so I am not
sure whether we would be charged each day for all 100,000+
possible maps that could be opened or whether the charge only
applies to maps that are generated when someone creates a
new record or opens and existing record. If the charge is for all
100,000+ possible maps, then the first 25,000 maps per day
are free leaving 75,000 maps per day at $0.50 per map making
the charge $37,500 per day – at least! Yikes – we could not
afford these costs.
I have spoken with the old BowerBird programmers. I employed
these two people for two years when they developed BowerBird
back in 2012-2013. Unfortunately, that funding has long since
been fully expended and sourcing new funding is challenging.
However, funding will not easily solve this problem as one of
the programmers explained to me. First up, the BowerBird
software was compiled and made live back in 2013. Any
changes now to the BowerBird code base would require the
programmers to reuse the 2013 compiler. The Microsoft
compiler code has been upgraded each year since and so the
2013 complier is no longer readily available. The programmers
would have to rebuild a 2013 compiler program on a separate
machine and that will take time … and funding.
The other problem is the so called “knock-on” effect.
Programmers hate going back to old software to make changes
as you can never predict the possible knock-on effects of
changing something that inadvertently affects some other
pieces of the code.
So, we will look into the actual costs which will either be $0 or
at least $37,500 per day and then factor in the “gamble” of
changing the code.
Nothing is ever simple is it? Thanks for your patience.
Part of the issue with the BowerBird software is that they used Microsoft development tools. In my experience with them (C++) it is horrendous to try and import an old project into a new development version (In Delphi, for example, its very easy and often as simple as just opening the project in the new environment). The way around this would have been to develop it in a virtual environment (VM) where you kept the old operating system and development software. My understanding was that a significant part of the cost to fix the bugs was just to get the development environment working again.
There also appear to be misunderstanings about google maps’ costs and API keys. Unfortunately Ken was not able to do anything techical with the system - not even adding to the static species list (of which many groups were missing). Yes, Ken is a great advocate for museums and citizen science and itsa a great pity that his pet project floundered and is about to die.
I can check what my last record is that made it to BowerBird and then submit later observations on iNaturalist but its too complicated to go through all the 4000 records to see which ones were missing ID’s and never made it.
According to your profile on Bowerbird, you have 4647 sightings submitted. Therefore after Bowerbird goes down, you will have 4647-3703 = 944 orphaned sightings. Nothing uploaded after 17 Dec 2017 made it to ALA, so by viewing all your sightings and moving from the latest to the earliest, you can safely recognise and copy over if you wish everything submitted after 17 Dec 2017.
I agree that finding unidentified records before the cutoff date will be much harder, but I’m doing similar with about 360+ to-be-orphaned records.
This is a bummer to hear especially about the lost observations. Thanks for sharing your experience with the iNaturalist community. I hope iNaturalist proves more durable!
You can cache Google maps tiles.
And i now observado.org had the same problem and seemed to transfer rather easy to a system which is very popular in Europe and also has better maps than Google and is for free to.
iNat is doubling in my region and also world wide so i guess it is the same in Australia.
We’re very sad to hear about Bowerbird. Thats good that most of the data has been archived at ALA. We’re also happy if iNaturalist can provide a home for any of the Bowerbird community who’d like to keep being a part of a naturalist community and posting observations into the future as Ken suggested.
Unlike ALA, iNaturalist isn’t a good place to archive orphaned Bowerbird observations. But we’d be happy to help Bowerbird community members get up and running on iNaturalist if that means help moving over their own observations from Bowerbird to iNaturalist. If this is of interest to you or if know someone well who might be interested send/have them send an email to help@inaturalist.org with the name of the iNaturalist account (make one if your don’t already have one), the name of the Bowerbird account, and confirmation that you’d like help moving observations.
Two caveats: (1) please only do so if you have enough observations that it would be difficult to move them manually (lets say >100). (2) please only do so if you haven’t already posted many (lets say >10) of the Bowerbird observations to iNaturalist on your own to avoid duplication issues that would be difficult to clean up manually (unless there’s some super easy way of sorting not-already-posted ones).
@loarie Have you developed some sort of script to extract records from BowerBird? I don’t see an easy way to do it manually as the only way to access records is to keep scrolling down. I guess the way to automate it would be to access each record by number and get the details that way.
Each observation has a URL that looks like the one below with the latest one being number 126691. http://www.bowerbird.org.au/observations/119405
Co-ordinates are available in the page source.
I spent a little time this morning and wrote a program to extract the data from any BowerBird observation (currently it doesn’t extract the identifications yet but that is in the source files including the identifiers and “votes”). I got the following from one of my records. The main thing is I can get the media links (and therefore the images). The Filename, Height & Width fields don’t serve much purpose except provide details of the original image (the name could be useful to find the original on someone’s personal backup). Comments could also be collected.
Observer=Reiner Richter
Id=observations/119405
CreatedOn=2018-11-10T07:46:14.9399918Z
ObservedOn=2018-11-10T00:04:50
Latitude=-37.9400861111111
Longitude=145.22260499999993
CommentCount=0
IdentificationCount=1
Title=Asian Paper Wasp in Melbourne
Address=Tirhatuan Wetlands, Rowville VIC 3178, Australia
MediaCount=4
IdentificationComments=Two nests seen in grass beside swamp and also one seen feeding on Leptospermum continentale flowers.
[Media1]
IsPrimaryMedia=false
Filename=DSC02056.JPG
Height=1760
Width=2640
URL=http://media.bowerbird.org.au/251/250955-Full1024.jpg
[Media2]
IsPrimaryMedia=false
Filename=DSC02059.JPG
Height=2624
Width=1024
URL=http://media.bowerbird.org.au/251/250956-Full1024.jpg
[Media3]
IsPrimaryMedia=true
Filename=DSC02058c.JPG
Height=1536
Width=1024
URL=http://media.bowerbird.org.au/251/250957-Full1024.jpg
[Media4]
IsPrimaryMedia=false
Filename=DSC02064.JPG
Height=2624
Width=1024
URL=http://media.bowerbird.org.au/251/250958-Full1024.jpg
I just added an optional user filter and ran my program and got the following log (note most of the time is spent fethcing the data via HTTP sequentially).
12:49:04.406 Starting 119405 to 119405...
12:49:04.828 Observations processed: 1
12:49:04.828 Finished in 422ms
12:49:26.109 Starting 115200 to 115300 for Reiner Richter...
12:49:39.484 Observations processed: 8
12:49:39.500 Finished in 13s
12:53:10.218 Starting 80000 to 79000 for Graeme Cocks...
12:55:29.078 Observations processed: 34
12:55:29.093 Finished in 2min
If anyone is interested, the current code can be downloaded via the link below. Its written in Lazarus and contains a precompiled Win32 executable. I will probably implement the identification extraction and then use it for my records. I don’t need to download the images but if anyone else wants to use it that could be implemented relatively easily. Automatically submitting to inat would be good but I am unfamiliar with the API (I don’t even know how to do the HTTPS & authentication). http://rnr.id.au/temp/BowerBird%202019-05-20.zip
That’s excelelnt and means an editable record exists. I’m still not sure what the best way to go about it is for my few hundred missing records. How much work is involved for you guys? I have downloaded all the BowerBird records from ALA and could then only download the missing ones - can your migration do that too?
I have also realized that my program could generate a CSV database that ALA can import relatively easy (they have imported my 6500 dragonfly records from such a file). I estimate downloading all the observation record data from BowerBird to take about 5 hours (the photos would be extra). This would allow saving the BowerBird data statically but at least all would not be lost. If I use your migration the records could then still be modified.