iNaturalist is Down

[anon93074988]

Our co works on AWS, where multiple availability zones are a thing, and very useful to avoid e.g. comet strikes taking out an entire geographical region. Do you have such a strategy? Happy to discuss if it helps.

[kueda]

Azure supports a similar concept. It’s not something we’re using now, clearly, but yes, definitely worth looking into.

[elias105]

I also just found how much I rely on inaturalist - just to know where to go on the weekend. Now it’s Saturday morning and I’m sitting here and have no idea which way to drive to see interesting things… there’s some prairie preserves about 2-3 hours away, but around here all too often those have not a single native plant and are just overgrown with invasives - and no other way to check but inaturalist.

[arborsphere]

@kueda

Microsoft servers…never been as stable as we would like

Has Microsoft ever made anything that could be described otherwise?

[kaden_slone04]

I logged back on the forum because I’m having the same problem on my phone. Whenever I open the app, it says “Service temporarily unavailable. Please try again in -1340 seconds.”

[natemarchessault]

Thanks for your dedicated efforts, iNat staff! Be sure to get some rest when needed. While the servers are down instead of exploring/identifying we are all out observing, so we’ll be making the servers work hard for their decisions

[annkatrinrose]

Thanks for keeping us updated and working on this on a weekend! I had noticed some glitches over the past week with occasional error messages telling me the site was busy (too many connections or something along those lines) and to try again later, and sometimes pages wouldn’t update while I was adding IDs. I’m assuming those may have been connected to the issue?

[annkatrinrose]

@natemarchessault

While the servers are down instead of exploring/identifying we are all out observing

Haha, yes, that’s exactly what I’m planning to do today…

[navin_sasikumar]

As a DevOps person, I feel your pain, iNat staff! Good luck! And sorry you were up late on a Friday night and are stuck working on a weekend.

[carrieseltzer]

@loarie

It was indeed caused by a major outage at Microsoft Azure in the West US 2 region where we rent our servers. Unfortunately, we have to rebuild the search indices which will take a bit of time - and it looks like Azures yet not fully recovered so we’re still partially waiting. We also may need to get some sleep since this happened around midnight EST where our mighty dev-ops team of 1 is located.

Just a minor update to say that although the outage at Azure appears to be resolved (i.e. if you click the link above you’ll see nothing but green checkmarks in our region West US 2), there is still considerable work to do before iNat can get back online.

[charlie]

@kueda

backups

oof, what a mess for you all.

Remember, you can still collect observations on the app as long as you turn auto upload off

[yayemaster]

Is this possibly because of the major solar flares that have been occurring and messing with electrical equipment?

[seeker1000]

Yes, app not working… several uploads waiting in SEEK…

So I just read here what is going on… sigh

@Staff-people: You will do it, you will make it run again, I know!! Thanks for caring!! :)

[loarie]

Hi all -

Thanks for continuing to bear with us. The iNat team is working through the weekend to get iNat back up and running.

As mentioned above, the outage issues that were out of our hands at Microsoft Azure are now resolved. But the slow process of reindexing the search indices remains before we can bring the site back up.

iNaturalist uses these indices in order to quickly find and return information from the databases when you interact with the site. We can start with a week-old version from the weekly snapshots of the index that we make, but in order to bring it up to date with the database, we need to reindex the millions of changes that have occurred over the last week against the current database. This reindexing will take some time.

Thanks to @pleary who worked through the night on this and @kueda. Proud to watch the two of them working through the Saturday to get us back online.

[radbackedsalamander]

Minecraft :)

[ram_k]

Dear iNAT Team,

Thank you,

I know you will get inaturalist back online.

Best wishes and thanking you in advance.

I am sure the tonne of messages you have been getting are a small indication of the level of attachment we have. (addiction would be a better word).

Stay safe, re-hydrate, eat, grab 40 or 80 winks - we folks aren’t going anywhere and can only give our good wishes.

warmly
Ram

[ram_k]

welcome to the forums @navin_sasikumar

[gcwarbler]

@loarie

Thanks to @pleary who worked through the night on this and @kueda

All Hail @pleary and @kueda!!

[michaelpirrello]

So, out of curiosity, occasionally I run into observations that for some reason need to be re-indexed. Does working through the recovery process mean that all of those “need to be re-indexed” observations will now all be fixed?

[kueda]

Nope. We’re re-indexing all obs that were indexed in the last week, b/c we have a backup of the index from a week ago, but there are plenty of other observations like you’re describing that probably need to be indexed but that we’re not going to get to as a part of this recovery process.