Page MenuHomePhabricator

Setting up static maintenance page on Foundation servers for Foundation website
Open, MediumPublic

Description

We experienced an outage of the Wikimedia Foundation website on 11 June 2019. The cause was our host and our ability to resolve it was limited. We were able to put up a maintenance page on VIP's servers during the outage. However, the idea came up that we should prepare a similar maintenance page on our servers for the site in case something major happened with our host and we decided we needed to flip the WikimediaFoundation.org domain back to our servers immediately.

To be clear, this outage was the first for our sites with that host in several years. We have no reason to believe it will happen again soon. However, we would like to be prepared just in case.

We have a maintenance page in PHP format that we could flip to HTML easily. What is the best way to get that on the servers in a way that we could easily setup the WikimediaFoundation.org to point to if we had an urgent need to do so?

Event Timeline

Varnent triaged this task as Medium priority.Aug 7 2019, 9:59 PM

However, the idea came up that we should prepare a similar maintenance page on our servers for the site in case something major happened with our host and we decided we needed to flip the WikimediaFoundation.org domain back to our servers immediately.

Unfortunately, DNS won't really work like that... Nothing is guaranteed immediate

Yes - to be fair - immediate outcome is not the expectation, more immediate action on our part. The idea originated with @Bawolff who may have some additional thoughts.

From Brian originally: "Does it make sense to have some sort of emergency mode prepared, where the DNS for the site could be easily redirect back to wmf servers and have it display a pre-prepared static website, in the event something bad happens with the external hoster and we want control of the messaging/error page?"

GDNSD supports HTTP health checks (or custom ones as well). Provided the DNS TTL is no longer than a few minutes, 'automatic failover' is possible.

EDIT: this only seems to work in a pool where both Automattic and WMF servers are pooled, while you only want the WMF servers to be pooled if Automattic is depooled. Pity!

BBlack subscribed.

The swap of Traffic for Traffic-Icebox in this ticket's set of tags was based on a bulk action for all such tickets that haven't been updated in 6 months or more. This does not imply any human judgement about the validity or importance of the task, and is simply the first step in a larger task cleanup effort. Further manual triage and/or requests for updates will happen this month for all such tickets. For more detail, have a look at the extended explanation on the main page of Traffic-Icebox . Thank you!