[[ https://etherpad.wikimedia.org/ | Etherpad ]] , our real-time collaborative editing tool suffered an outage due to what we only know for now is database corruption since 14:27 UTC. This was detected shortly after it happened and we (ops in charge of the service and the database) worked to reestablish the service.
As the service continued crashing despite our efforts, we decided to recover a database backup from 2016-06-22 01:00:01 UTC. The service is now back up and working (since 16:11), but that means that you may have lost a day and a half of edits.
I understand that that may cause a lot of inconveniences, specially for the people at Wikimania. **We are now trying to recover more than that**, but as the crashes could come back, or not all could be recovered, plus the service itself is needed, the plan is the following:
- Keep the current pads as is, not delete any edit done from now on
- If possible, recover the last days of edits on a separate address.
This ticket will be updated for progress.