Page MenuHomePhabricator

Setting up static maintenance page on Foundation servers for Foundation website
Open, NormalPublic

Description

We experienced an outage of the Wikimedia Foundation website on 11 June 2019. The cause was our host and our ability to resolve it was limited. We were able to put up a maintenance page on VIP's servers during the outage. However, the idea came up that we should prepare a similar maintenance page on our servers for the site in case something major happened with our host and we decided we needed to flip the WikimediaFoundation.org domain back to our servers immediately.

To be clear, this outage was the first for our sites with that host in several years. We have no reason to believe it will happen again soon. However, we would like to be prepared just in case.

We have a maintenance page in PHP format that we could flip to HTML easily. What is the best way to get that on the servers in a way that we could easily setup the WikimediaFoundation.org to point to if we had an urgent need to do so?

Event Timeline

Varnent created this task.Wed, Aug 7, 9:59 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptWed, Aug 7, 9:59 PM
Varnent triaged this task as Normal priority.Wed, Aug 7, 9:59 PM
CDanis added a subscriber: CDanis.Wed, Aug 7, 10:00 PM
Varnent updated the task description. (Show Details)Wed, Aug 7, 10:00 PM
Varnent updated the task description. (Show Details)Wed, Aug 7, 10:02 PM
Reedy added a subscriber: Reedy.Wed, Aug 7, 11:16 PM

However, the idea came up that we should prepare a similar maintenance page on our servers for the site in case something major happened with our host and we decided we needed to flip the WikimediaFoundation.org domain back to our servers immediately.

Unfortunately, DNS won't really work like that... Nothing is guaranteed immediate

Yes - to be fair - immediate outcome is not the expectation, more immediate action on our part. The idea originated with @Bawolff who may have some additional thoughts.

From Brian originally: "Does it make sense to have some sort of emergency mode prepared, where the DNS for the site could be easily redirect back to wmf servers and have it display a pre-prepared static website, in the event something bad happens with the external hoster and we want control of the messaging/error page?"

Southparkfan added a subscriber: Southparkfan.EditedThu, Aug 8, 12:34 AM

GDNSD supports HTTP health checks (or custom ones as well). Provided the DNS TTL is no longer than a few minutes, 'automatic failover' is possible.

EDIT: this only seems to work in a pool where both Automattic and WMF servers are pooled, while you only want the WMF servers to be pooled if Automattic is depooled. Pity!

ema moved this task from Triage to General on the Traffic board.Wed, Aug 14, 12:41 PM