Page MenuHomePhabricator

Backups systems for userDBs hosted on replicas
Closed, DeclinedPublic

Description

When one of the replica servers went down on Wednesday[1], one of my userDBs (p50380g50729__grantsbot) went with it. That db and the data in it looks to be gone forever. It sounds like wipe-and-reset is the standard practice when these replicas go down, so I imagine that these dbs will be deleted any time there's a hardware malfuction.

Fortunately for me, there wasn't much data there yet (I'm still migrating). However, it's still lost work.

My bots runs joins on replicas and store data in custom tables, so I can't host it on tools-db. This is a pretty common workflow for bots in general. Therefore, there should be backup systems in place for restoring those userDBs after a failure.

I can and will run daily backups of all my tables from now on, but that doesn't solve the problem: there will still be potential for data loss any time there's a hardware failure. For the many, many other bots that run more frequently than once a day, the data loss could be even more significant and disruptive.

Bots are essential to the maintenance of our projects. Can we get better backups systems for userDBs hosted on replicas?

  • Jonathan

[1]http://lists.wikimedia.org/pipermail/labs-l/2013-September/001668.html


Version: unspecified
Severity: normal

Details

Reference
bz54401

Event Timeline

• bzimport raised the priority of this task from to Needs Triage.Nov 22 2014, 2:06 AM
• bzimport added a project: Cloud-VPS.
• bzimport set Reference to bz54401.

I am not sure if this is something that is desirable. Most user databases there are used for scratch data, and that is the intended usage. There has been never a guarantee of high availability of those databases, while disk space is provided if offline copies are required (each user's responsability). This is the same model than code: you should have copies on a WMF or external repository.

From this, I would say that providing proper backups in general is out of the question, specially having the ability of recovering an arbitrary point of time. There are not resources for that.

I would agree, however, to have some kind of redundancy, like replication- so that in the event of a hardware failure of maintenance they are still available, but that is difficult to setup due to some extreme write queries, that would block replication for everyone else very frequently.

I think something could be figured out, but there should be an architecture redesign of labs first for better redundancy.

• chasemp subscribed.