I uploaded this 3 weeks ago but i'm not sure what the status is even though i wrote "we now have a new instance called relic-stretch to replace it".

https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/455737/

If it was just the Apache part and a regular "web proxy" to click in Horizon i would have done it right now. The apache setup from puppet on the new instance looks fine.

But there is also an email part of this and the special setup in DNS zones in Horizon.

Change 455737 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] openstack::network: update private IP of relic.toolserver instance

https://gerrit.wikimedia.org/r/455737

checked and tested exim config on existing instance and adding a new alias, works. not much going on in exim.log at all

checked apache config and exim config on new instance have been generated by puppet, all looks ok

checked the DNS setup in Horizon, looks like it doesn't need a change and all we need is the Gerrit change above, afaict

should be just that hopefully and then testing https://www.toolserver.org/ works and sending an email to somebody @toolserver.org

The mapping of aliases is in /etc/toolserver.aliases

Change 462004 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] toolserver_legacy: enable ssl module for httpd

https://gerrit.wikimedia.org/r/462004

Change 462004 merged by Dzahn:
[operations/puppet@production] toolserver_legacy: enable ssl module for httpd

https://gerrit.wikimedia.org/r/462004

Change 455737 merged by Dzahn:
[operations/puppet@production] openstack::network: update IPs of relic.toolserver

https://gerrit.wikimedia.org/r/455737

13:58 < mutante> !log toolserver.org and subdomains (wiki.toolserver, status.toolserver, stable.toolserver) legacy URLs have been switched to new stretch backend, away from trusty

13:59 < mutante> !log *.toolserver.org also moved from eqiad to eqiad-r region in cloud vps, which gave it new IP addresses

Change 462012 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] network: add 172.16.0.0/21 (labs-instances2-b-eqiad) to external_networks

https://gerrit.wikimedia.org/r/462012

Change 462012 abandoned by Dzahn:
network: add 172.16.0.0/21 (labs-instances2-b-eqiad) to external_networks

Reason:
< paravoid> the real problem is this: "The production MX servers, mx1001/2001 are the external SMTP for cloud VPS instances." :)

< paravoid> the solution here is to set up separate email relays for WMCS

https://gerrit.wikimedia.org/r/462012

How to migrate relic instance to relic-stretch-eqiad instance:

• create new instance, run puppet
• apply puppet role role::toollabs::legacy, run puppet again

• scp root@relic.toolserver-legacy:/etc/toolserver.aliases .
• scp root@relic.toolserver-legacy:/etc/acme/cert/* .
• scp root@relic.toolserver-legacy:/etc/acme/key/toolserver.key .

• scp toolserver.key root@relic-stretch-eqiad.toolserver-legacy:/etc/acme/key/
• scp toolserver.c* root@relic-stretch-eqiad.toolserver-legacy:/etc/acme/cert/
• scp toolserver.aliases root@relic-stretch-eqiad.toolserver-legacy:/etc/

• run puppet again on new instance, should show no errors

Change 462023 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] openstack::network: update private IP of relic.toolserver

https://gerrit.wikimedia.org/r/462023

17:16 < mutante> !log toolserver-legacy associating floating IP (the existing one in eqiad) with relic-stretch-eqiad

17:19 < mutante> !log toolserver-legacy update DNS zone entries and remove eqiad-r IP, revert to eqiad IP

This is done. Things have switched away from "relic" on trusty and over to "relic-stretch-eqiad" , a stretch instance in eqiad.

switching from eqiad to eqiad-r has been reverted and should be seen as unrelated to removing trusty.

This also means email works fine now, just like it did before. Tested the aliases and it delivered to an external address.

Change 462023 merged by Dzahn:
[operations/puppet@production] openstack::network: update private IP of relic.toolserver

https://gerrit.wikimedia.org/r/462023

Mentioned in SAL (#wikimedia-cloud) [2018-09-21T23:12:10Z] <mutante> migration complete. shutting down trusty instance 'relic'. (T204564)

Mentioned in SAL (#wikimedia-cloud) [2018-09-21T23:25:16Z] <mutante> deleting now unused trusty instance relic (T204564) (/etc is backed up on new instance in /root/ just in case)