In theory it should be simple to have horizon be active/active between the two labweb hosts. They should only need to share memcached access for tokens.
|operations/puppet : production||labweb nutcracker: further attempt to pass in memcached_pools correctly|
|operations/puppet : production||labweb: create /var/run/nutcracker|
|operations/puppet : production||labweb nutcracker: add non-functional redis host list|
|operations/puppet : production||labweb nutcracker: re-use profile::mediawiki::nutcracker|
|operations/puppet : production||labweb: install nutcracker|
|operations/puppet : production||labweb: remove specific memcached port|
|operations/puppet : production||horizon/labweb: open firewall to internal IPs for port 80|
|operations/puppet : production||labweb: inclued role::lvs::realserver on labweb hosts|
|operations/puppet : production||horizon memcache: fix an issue with erb var resolution|
|operations/puppet : production||horizon: add a missing arg|
|operations/puppet : production||labweb horizon: share memcached among labwebs|
|operations/puppet : production||labweb: add lvs service in front of labweb1001/1002|
|operations/dns : master||add lvs ip for labweb services|
|Resolved||bd808||T166396 Program 1 Outcome 4: VPS hosting|
|Resolved||Jdforrester-WMF||T172165 Require either PHP 7.0+ or HHVM in MW 1.31|
|Resolved||RobH||T168559 decom silver (was silver has trouble rebooting)|
|Resolved||Jclark-ctr||T189921 decom californium|
|Resolved||Andrew||T168470 Setup wikitech, horizon, and striker on new labweb hardware|
|Resolved||Andrew||T187493 deploy horizon version 'ocata' using scap3|
|Resolved||Andrew||T187506 load-balance horizon between labweb1001 and 1002|
I misunderstood this a bit -- I assumed that the misc-web varnishes would load-balance for us, but it turns out we would need to add an additional layer of load-balancing (and a service ip) in front of the labwebs to have them be active/active. I'm not totally sure this is worth the additional complexity (especially the complexity of debugging future issues) given that we'll have no trouble supporting current traffic loads with one active host.
nutcracker is now running on both labwebs, sharing a memcached pool. In theory it's also configured for redis but redis isn't actually running on either host. We'll see if we turn out to need it.