BowerBird.org.au R.I.P

Since May 2013 BowerBird has been the main “citizen science” web site for Australia. Unfortunately today it has been announced that it will receive no more funding and go offline when the current server lease expires in several months. Users have been suggested to switch to iNaturalist.
https://researchdata.museum.vic.gov.au/BowerBird_Bugle/Edition_60.pdf

So long BowerBird… Long live iNat!

I submitted about 4000 sightings through BowerBird, most of which have made it to ALA but some unidentified and more recent stuff didn’t make it. [If you click the ALA link then on “Attribution” you see where my records have come from.] Apparently about 50,000 records never made it to ALA so unless those people resubmit to inat those will be lost forever. :(

There is no way to download your record data from BowerBird like there is for inat at least.

7 Likes

It nay be a bit of a hack, but if the data got into ALA and or GBIF I assume you could download data from there and use as a source for input to iNat. It would create a risk of duplication depending on their data stewardship processes on those downstream entities though.

Yes, data can be downloaded from ALA, but there are over 120,000 records on BowerBird and less than 70,000 made it to ALA. Records that made it to ALA should at least stay there and not vanish.

We moved the southern African iSpot community to iNat - speak to @tonyrebelo and @loarie about how to do this.

2 Likes

Abotu Bowerbird https://forum.inaturalist.org/t/integrating-records-from-australian-nature-sites/2536/2
As far as i know iNat only wants to support interfaces, APIs.
I got a test script but did not figure out how to run it.

There was a post over duplicates
https://forum.inaturalist.org/t/inaturalist-2019-team-retreat-follow-up/373
(Duplicates and merge!!!)
We’ve written a spec for the team to review and see if this is feasible, so I’m going to close this topic.
https://forum.inaturalist.org/t/duplicate-prevention-notify-observers-if-their-image-checksums-match-others-on-the-site/258/5

https://forum.inaturalist.org/t/photo-analyzer-to-prevent-duplicate-obs/1494

https://forum.inaturalist.org/t/duplicate-prevention-notify-observers-if-their-image-checksums-match-others-on-the-site/258

Oh wow, that really sucks!

I am slowly moving as many Bowerbird records as I can over here, the ones that did not make it to ALA. Crunch time as there are several hundred affected ;_;

1 Like

I’m sad to hear BowerBird’s technological troubles proved insurmountable. I’ve enjoyed reading the BowerBird Bugle newsletter occasionally since 2015. Around then I came across BowerBird and added it to this list of platforms like iNaturalist. I have always been so impressed with Ken’s entomological insights and enthusiasm. I think some of the anecdotes in the BowerBird Bugle are some of the best importance-of-museum-specimen stories I’ve read (you can find them all here). The Bugle also highlights amazing insect photography (including from @reiner!) and the value of associated records in great detail. Ken was a great advocate for BowerBird, and these kinds of projects really need deeply knowledgable champions.

It’s a shame about the orphaned data that didn’t make it to ALA. For easy reference, I’m copying the portion of the newsletter that lays out the struggles.

BowerBird was always offered for free and I have since learnt
that it costs money to offer something for free. The main yearly
cost of BowerBird is hosting it in the Amazon cloud. The
BowerBird programmers argued that it needed two virtual
servers – one for the software and one to store the images.
We initially tried using just one server for both software and
images but the website ran very slowly. So, hiring two virtual
servers is more expensive than hiring one. I have been able to
manage to secure sufficient funds to support delivering
BowerBird. However, the situation changed when software
problems occurred about 12 months ago. The first was the
decision by Google Maps to charge for the use of their maps –
previously Google Maps had been a free service. The second
software problem occurred with data uploads to ALA. Every
night, BowerBird creates a complete data dump for all of its
records. Then weekly on a Sunday afternoon, ALA would
hoover up these records and display the BowerBird dataset on
ALA. By uploading the complete BowerBird dataset meant that
any identification that had changed or been updated took only
one week to be corrected on ALA. The software programmers
created a data dump program that had an expiry time of five
hours. At the time of releasing BowerBird, that seemed like a
reasonable if not excessive time limit. As we have now
discovered, the five hour time limit times out at about 70,000
records but there are now over 120,000 records on BowerBird
and so not all data can be downloaded from BowerBird nightly
to be uploaded to ALA. This has been the case now for over 6
months.
I raised over $50,000 to fix these and other BowerBird software
issues – that was the amount the programmers told me was
needed to fix these issues. I paid the programmers these funds
but when they began working on the problems, they realised
the issues were much bigger than they had initially considered
and would cost at least double what I had paid. I was not able
to raise double these funds and so none of the BowerBird
problems have been resolved.
It took quite some time for me to be told there was insufficient
fund to resolve the BowerBird issues. So, I waited for problems
to be fixed but nothing happened despite assurances from the
programmers. Finally, the time has come to say enough.
I believe that the Google Maps and ALA data upload issues will
not be resolved. To me, a core component of BowerBird is the
data uploads to ALA where the data is permanently stored,
added to millions of other data records and displayed for all to
use.
I am now recommending that if you want your citizen data to
reach ALA, you should stop using BowerBird and start using
the iNaturalist citizen science website
(https://www.inaturalist.org/ ). Many BowerBird users have told
me that they have already made this change.
Finally, I have no idea how much longer the BowerBird website
will continue to be delivered. I believe it will stop sometime in
the next six months but I cannot get an answer from the
programmers on when BowerBird supporting funds with run
out.
This is all somewhat disappointing news for everyone involved
in some way with BowerBird. The website really has made
some amazing discoveries. On ALA for many species,
BowerBird records provide the only distribution points and
species images. Several fascinating magazine stories have
been published about BowerBird records.
We should all be very proud of the legacy that BowerBird
contributors have built and the many lessons that citizen
science has learnt from BowerBird.
Thanks from me to everyone involved with BowerBird. It has
now run its course but there are other websites where new
discoveries can be made and shared with the world. I do hope
that BowerBird users will now engage with other citizen science
websites and continue making valuable contributions about our
natural world.
Thank you all and happy photographing nature. Citizen science
is now a well-accepted and valued part of natural sciences.
BowerBird and your contributions have helped to secure this
status. Best wishes – Ken Walker

I went back a couple of issues in the Bugle to find earlier mention of the technological problems. In October 2018 Ken wrote:

The Google Maps issue
I’m sure you have all seen the recent Google Maps issue. If
you click the OK button you do actually see a map but that’s not
really a fix.
The problem has come about due to a change at Google. They
have provided free access to their Google Maps software for
over a decade and now they have decided to charge for that
service so they changed the code. The charges are very
reasonable as the first 25,000 maps per day on a website are
free. BowerBird currently has over 100,000 records so I am not
sure whether we would be charged each day for all 100,000+
possible maps that could be opened or whether the charge only
applies to maps that are generated when someone creates a
new record or opens and existing record. If the charge is for all
100,000+ possible maps, then the first 25,000 maps per day
are free leaving 75,000 maps per day at $0.50 per map making
the charge $37,500 per day – at least! Yikes – we could not
afford these costs.
I have spoken with the old BowerBird programmers. I employed
these two people for two years when they developed BowerBird
back in 2012-2013. Unfortunately, that funding has long since
been fully expended and sourcing new funding is challenging.
However, funding will not easily solve this problem as one of
the programmers explained to me. First up, the BowerBird
software was compiled and made live back in 2013. Any
changes now to the BowerBird code base would require the
programmers to reuse the 2013 compiler. The Microsoft
compiler code has been upgraded each year since and so the
2013 complier is no longer readily available. The programmers
would have to rebuild a 2013 compiler program on a separate
machine and that will take time … and funding.
The other problem is the so called “knock-on” effect.
Programmers hate going back to old software to make changes
as you can never predict the possible knock-on effects of
changing something that inadvertently affects some other
pieces of the code.
So, we will look into the actual costs which will either be $0 or
at least $37,500 per day and then factor in the “gamble” of
changing the code.
Nothing is ever simple is it? Thanks for your patience.

3 Likes

Part of the issue with the BowerBird software is that they used Microsoft development tools. In my experience with them (C++) it is horrendous to try and import an old project into a new development version (In Delphi, for example, its very easy and often as simple as just opening the project in the new environment). The way around this would have been to develop it in a virtual environment (VM) where you kept the old operating system and development software. My understanding was that a significant part of the cost to fix the bugs was just to get the development environment working again.

There also appear to be misunderstanings about google maps’ costs and API keys. Unfortunately Ken was not able to do anything techical with the system - not even adding to the static species list (of which many groups were missing). Yes, Ken is a great advocate for museums and citizen science and itsa a great pity that his pet project floundered and is about to die.

I can check what my last record is that made it to BowerBird and then submit later observations on iNaturalist but its too complicated to go through all the 4000 records to see which ones were missing ID’s and never made it.

1 Like

@reiner Your sightings on Bowerbird are such a treasure and it would be a tragedy to lose the ones that did not make it to ALA. These are the ones that made it through: https://biocache.ala.org.au/occurrences/search?q=data_resource_uid%3Adr893&fq=collector%3A"Reiner%20Richter"

According to your profile on Bowerbird, you have 4647 sightings submitted. Therefore after Bowerbird goes down, you will have 4647-3703 = 944 orphaned sightings. Nothing uploaded after 17 Dec 2017 made it to ALA, so by viewing all your sightings and moving from the latest to the earliest, you can safely recognise and copy over if you wish everything submitted after 17 Dec 2017.

I agree that finding unidentified records before the cutoff date will be much harder, but I’m doing similar with about 360+ to-be-orphaned records.

This is a bummer to hear especially about the lost observations. Thanks for sharing your experience with the iNaturalist community. I hope iNaturalist proves more durable!

4 Likes

Is this why it seems like there are so many new Australian users on iNat this week?

You can cache Google maps tiles.
And i now observado.org had the same problem and seemed to transfer rather easy to a system which is very popular in Europe and also has better maps than Google and is for free to.

iNat is doubling in my region and also world wide so i guess it is the same in Australia.

https://researchdata.museum.vic.gov.au/BowerBird_Bugle/Edition_60.pdf It was a long read

1 Like

We’re very sad to hear about Bowerbird. Thats good that most of the data has been archived at ALA. We’re also happy if iNaturalist can provide a home for any of the Bowerbird community who’d like to keep being a part of a naturalist community and posting observations into the future as Ken suggested.

Unlike ALA, iNaturalist isn’t a good place to archive orphaned Bowerbird observations. But we’d be happy to help Bowerbird community members get up and running on iNaturalist if that means help moving over their own observations from Bowerbird to iNaturalist. If this is of interest to you or if know someone well who might be interested send/have them send an email to help@inaturalist.org with the name of the iNaturalist account (make one if your don’t already have one), the name of the Bowerbird account, and confirmation that you’d like help moving observations.

Two caveats: (1) please only do so if you have enough observations that it would be difficult to move them manually (lets say >100). (2) please only do so if you haven’t already posted many (lets say >10) of the Bowerbird observations to iNaturalist on your own to avoid duplication issues that would be difficult to clean up manually (unless there’s some super easy way of sorting not-already-posted ones).

I did a quick check of the top Bowerbird users with over 100 observations:
User Name: #Observations
Graeme Cocks: 8003
Ken Harris: 6626
Reiner Richter: 4647
Adam Edmonds: 3716
David Akers: 3674
Anna Lanigan: 3540
Martin Lagerwey: 2726
Rose Robin: 2564
Linda Rogan: 2545
Matthew Connors: 2441
Tamara Leitch: 2067
Daniel Heald: 2033
Judy & Rob Peters: 1633
Geoff Lotton: 1528
Matt Campbell: 1386
David Mules: 1378
Glenda Walter: 1369
Tony Eales: 1302
Ken Walker: 1231
Laurence Sanders: 1154
Jeannie: 1136
John Eichler: 1079
Carol Page: 941
Teresa and John: 863
Jenny Thynne: 862
Ron Greer: 781
Rog Standen: 779
Simon Ong: 766
Lek: 760
Dianne Clarke: 736
Tony D.: 723
Ken Walker: 675
Dacre England: 655
Jenny Holmes: 627
Luis Mata: 589
Joshua Holden: 586
Gippsland Lakes - Mountain to the Sea: 577
Suzanne Jones: 564
Bryan Henshaw: 540
Irene Richardson: 529
Erica Siegel: 512
Ellura Sanctuary: 502
Kim Tarpey: 467
Leuba Ridgway: 458
David Rentz: 456
Jade Craven: 418
Liz O’Donnell: 417
City of Melbourne: 390
Geoff Boyes: 371
Gio Fitzpatrick: 370
Dianne: 343
Gordon Claridge: 341
Zac Billingham: 334
Robert Bender: 328
Mitch Smith: 326
Graham Winterflood: 303
Jean and Fred Hort: 282
Janet Grevillea: 278
Frank Pierce: 273
Frank Prinz: 272
Andrew Allen: 262
Steve Young: 255
Russell Stanley: 254
Kerry Stuart: 241
Mark Ridgway: 237
David Francis: 236
Mark Newton: 234
Beth Shaw: 223
Graham & Maree Goods: 213
Joan Hales: 207
Donna Tomkinson: 206
Hermit: 204
Marc Newman: 199
Tom Nataprawira: 190
Karen Retra: 188
Lily Kumpe: 184
Rosie Benz: 175
Faye Arcaro: 172
Teale Britstra: 169
Michael Bedingfield: 164
Isaac Wilson: 164
John Read: 156
Alison Pearson: 155
Russell Best: 155
Manu Saunders: 152
Sam Weedon: 151
Dane Wimbush: 148
Peter Marriott: 142
Ernst : 142
Geoff Walker: 139
Ethan Beaver: 133
Euan Moore: 132
Julie: 132
Cathy Powers: 131
Nina Franceschi Eason: 130
Val Ryland: 130
Robert Read: 129
Clare McLellan: 125
Nick Porch: 123
Peter Kerr: 116
Torbjorn von Strokirch: 114
UQ volunteers: 113
Gayl Sullivan: 111
Alan Melville: 111
Kristi Ellingsen: 109
Richard HARTLAND: 109
Sabine T Perrone: 109
Robin Corringham: 108
Wilma McNabb: 103

6 Likes

@loarie Have you developed some sort of script to extract records from BowerBird? I don’t see an easy way to do it manually as the only way to access records is to keep scrolling down. I guess the way to automate it would be to access each record by number and get the details that way.

Each observation has a URL that looks like the one below with the latest one being number 126691.
http://www.bowerbird.org.au/observations/119405
Co-ordinates are available in the page source.

Highest resolution images can be extracted by appending “-full1024”.
http://media.bowerbird.org.au/251/250957-full1024.jpg

Only the highest voted ID would need to be extracted.

This way all records could be extracted (with images, location & identification). This could be run against the records that made it to ALA and filtered only on the missing ones.
https://biocache.ala.org.au/occurrences/search?q=data_resource_uid:dr893

Yes, the API is relatively straightforward. It only returns 15 records at a time though so you have to paginate through

I spent a little time this morning and wrote a program to extract the data from any BowerBird observation (currently it doesn’t extract the identifications yet but that is in the source files including the identifiers and “votes”). I got the following from one of my records. The main thing is I can get the media links (and therefore the images). The Filename, Height & Width fields don’t serve much purpose except provide details of the original image (the name could be useful to find the original on someone’s personal backup). Comments could also be collected.

Observer=Reiner Richter
Id=observations/119405
CreatedOn=2018-11-10T07:46:14.9399918Z
ObservedOn=2018-11-10T00:04:50
Latitude=-37.9400861111111
Longitude=145.22260499999993
CommentCount=0
IdentificationCount=1
Title=Asian Paper Wasp in Melbourne
Address=Tirhatuan Wetlands, Rowville VIC 3178, Australia
MediaCount=4
IdentificationComments=Two nests seen in grass beside swamp and also one seen feeding on Leptospermum continentale flowers.

[Media1]
IsPrimaryMedia=false
Filename=DSC02056.JPG
Height=1760
Width=2640
URL=http://media.bowerbird.org.au/251/250955-Full1024.jpg

[Media2]
IsPrimaryMedia=false
Filename=DSC02059.JPG
Height=2624
Width=1024
URL=http://media.bowerbird.org.au/251/250956-Full1024.jpg

[Media3]
IsPrimaryMedia=true
Filename=DSC02058c.JPG
Height=1536
Width=1024
URL=http://media.bowerbird.org.au/251/250957-Full1024.jpg

[Media4]
IsPrimaryMedia=false
Filename=DSC02064.JPG
Height=2624
Width=1024
URL=http://media.bowerbird.org.au/251/250958-Full1024.jpg

I just added an optional user filter and ran my program and got the following log (note most of the time is spent fethcing the data via HTTP sequentially).

12:49:04.406 Starting 119405 to 119405...
12:49:04.828 Observations processed: 1
12:49:04.828 Finished in 422ms
12:49:26.109 Starting 115200 to 115300 for Reiner Richter...
12:49:39.484 Observations processed: 8
12:49:39.500 Finished in 13s
12:53:10.218 Starting 80000 to 79000 for Graeme Cocks...
12:55:29.078 Observations processed: 34
12:55:29.093 Finished in 2min

If anyone is interested, the current code can be downloaded via the link below. Its written in Lazarus and contains a precompiled Win32 executable. I will probably implement the identification extraction and then use it for my records. I don’t need to download the images but if anyone else wants to use it that could be implemented relatively easily. Automatically submitting to inat would be good but I am unfamiliar with the API (I don’t even know how to do the HTTPS & authentication).
http://rnr.id.au/temp/BowerBird%202019-05-20.zip

1 Like

Glad you’ve found a way to download your records from Bowerbird, @reiner.
In case your interested, we’ve gotten the “assisted migration” from Bowerbird to iNat for users who’ve contacted help@inat running, you can track progress here: https://www.inaturalist.org/observations?place_id=any&subview=grid&verifiable=any&field:Bowerbird%20URL=

1 Like

That’s excelelnt and means an editable record exists. I’m still not sure what the best way to go about it is for my few hundred missing records. How much work is involved for you guys? I have downloaded all the BowerBird records from ALA and could then only download the missing ones - can your migration do that too?

I have also realized that my program could generate a CSV database that ALA can import relatively easy (they have imported my 6500 dragonfly records from such a file). I estimate downloading all the observation record data from BowerBird to take about 5 hours (the photos would be extra). This would allow saving the BowerBird data statically but at least all would not be lost. If I use your migration the records could then still be modified.

1 Like