We are planning a datacenter switchover which is scheduled tentatively for the beginning of next quarter. In order to be able to do that, there are several thing that are needed as a preparation work:
[] Check MediaWiki for new/old "eqiad-only" dependencies
[] Check all services on scb
[] Install any missing service in codfw
[] Install a secondary etcd cluster in codfw
[] Modify MediaWiki config so that new services in codfw are correctly configured there
[] Add TLS whenever feasible in order to make cross-dc calls encrypted whenever possible
Apart from this, we need to improve on [[ https://wikitech.wikimedia.org/wiki/Switch_Datacenter#Per-service_switchover_instructions | the switchover procedure itself]]. Last time, it required a lot of coordination and long sequence of commands we had to execute in sequence. Another goal we have to pursue is to simplify this procedure, by reducing the number of manual steps needed and the number of code commits needed for the switchover.
[] Deploy the new cluster orchestration tool
[] Create a script based on said tool to automate most of the procedures
[] Make the switchover depend less on code deploys by integrating conftool with the configuration of services
[] Create a real ES-memcached warmup tool that is not just a list of URLs for apache-fast-test, or modify apache-fast-test to work more in parallel.