Page MenuHomePhabricator

https://outreachdashboard.wmflabs.org is down 2024-11-12
Closed, ResolvedPublicBUG REPORT

Description

Service is down at the moment of this writing.

We are using it for all our editathons and campaigns and it's a pity this great service is not stable enough to share metrics and activity with external partners.

Sorry if I'm not tagging the issue appropriately.

Thanks in advance.

Event Timeline

Peachey88 renamed this task from https://outreachdashboard.wmflabs.org is down to https://outreachdashboard.wmflabs.org is down 2024-11-12.Nov 12 2024, 9:55 AM

I wonder if this issue should also be reported here: https://meta.wikimedia.org/w/index.php?title=Talk:Programs_%26_Events_Dashboard&action=edit&section=new

That's where the 'Report a problem' link on the Dashboard takes you to.

Either is fine. The system has some database bottlenecks related to running updates on courses/events with large numbers of edits, so when a new high-activity event (or a surge of activity for an ongoing one) hits the system, it sometimes puts enough load on the database that the web service gets backed up and becomes unresponsive. It typically comes back online within a few hours at most (or sooner if I reboot the VM that runs the webserver), but it's down enough that I know it causes problems for things like sharing metrics with external partners.

We're getting close to completion with a major architecture update intended to fix these problems; I'm expecting to switch over to the new statistics update system in December, and by January or February (after working through the bottlenecks of the new system and tuning it appropriately) I expect it to be much more stable and reliable.

(Closing this, as it's been back up since sometime yesterday.)

Ragesoss claimed this task.