Hi Folks. I just wanted give an update on site performance. The City Nature Challenge (CNC) is obviously bringing in a lot of observations this weekend and a lot of traffic to the website. We prepared for that in recent weeks by adding extra capacity in the form of extra web, API, and search servers. But you never know how much traffic you’ll actually get until the event begins, and yesterday morning we found we had more requests coming in than the web servers could process which leads to the “queue full” errors.
Throughout the day we added several more web, API, and search servers as needed, and hopefully things got better as the day went on. Its safe to say not everything is going to work as fast as on a normal week, but hopefully everything does indeed work (i.e. no more error pages).
We are aware that the notification queue is backed up (See https://forum.inaturalist.org/t/notification-issue/2570/3). This is an unfortunate side effect of large events which generate massive amount of notifications. Notifications will very likely be delayed during CNC, but we’re not losing anything - it’s just a matter of churning through them all.
On the bright side - things are working well enough that in the CNC project alone 16k people have have added over 275k observations of 16k species since Friday, and we’re comfortably over 1M observations this month.
Here’s a chart of observations and observation photos added over the last 7 days. It shows that today we’re steadily getting about 15k observations per hour which is 5 times the average peak rate earlier this week and good amount more than even yesterday.
I’m watching site performance very closely this weekend and will continue to add more capacity as needed.