The other part of this is Redis. @zhuyifei1999 says that we shouldn't use tools' Redis server, but run our own. Can this just be on one of the two new prod servers?

So I talked to @bd808, and while some form of load balancing is possible, it's not particularly easy and we don't even know if we need it yet. If you check Horizon now you'll see I've created xtools-prod01, which is a m1.large with 8 GB of RAM. I really wanted to start off with a xlarge, but we don't have the quota unless we delete both the existing xtools-dev02 (staging) and xtools-live01 (production). So let's get xtools-prod01 set up as-is and do some load testing. If necessary we can request more quota for an xlarge, and if absolutely necessary we can look more into the load balancing. For the latter we'll need some smart sysadmin person (not me) to hold our hand in setting it up. I have some notes from my conversation with Bryan that may help.

In the current Cloud-VPS environment, the only load balancing that we can do is layer 7, meaning at the HTTP server tier. We may be able to use other LB techniques in the future after our nova-network SDN service has been replaced with OpenStack Neutron. This is an ongoing project for the cloud-services-team that will not be complete for many more months.

There is currently no support in dynamic-proxy, the proxy service that Horizon manages for us, for round-robin to multiple backends or other load balancing setups. That might be possible to implement, but it would not be a trivial project. One way to do some load balancing within a VPS project would be to have a single small instance that runs nginx and then configure that nginx instance to reverse proxy to multiple backend servers in other VMs.

Introducing load balancing adds a lot of complexity, so make sure you need it first. Once you are load balanced you need to keep multiple HTTP servers running and trace requests across multiple hosts. This can be challenging even with full-time ops support.

Also the 3 story point estimate here is crazy. ;)

In T169590#3423574, @Samwilson wrote:

We're also wanting to track 404s, and if we're just running on a single server that's easier (are we allowed to install e.g. awstats?).

Sure, do whatever log monitoring you want. You won't get real IPs from behind the common proxy, but that's by design.

The new box is up and running, and is auto-deploying the master branch whenever there's a change (it checks every 10 minutes).

I've created two new databases for XTools to put its things in; these are documented on https://wikitech.wikimedia.org/wiki/Tool:XTools (some more work is needed on that page, I know).

xtools-live01 is turned off now; we can delete it as far as I know.