Page MenuHomePhabricator

cloudvps: toolserver-legacy project trusty deprecation
Closed, ResolvedPublic


Ubuntu Trusty is no longer available in Cloud VPS since Nov 2017 for new instances. However, the EOL of Trusty is approaching in 2019 and we need to move to Debian Stretch before that date.

All instances in the toolserver-legacy project needs to upgrade as soon as possible.

The list of affected VMs is:

  • relic.toolserver-legacy.eqiad.wmflabs

Listed administrator are:

More info in openstack browser:

Related Objects

Event Timeline

Krenair triaged this task as Medium priority.Sep 17 2018, 4:57 PM
Krenair created this task.

I uploaded this 3 weeks ago but i'm not sure what the status is even though i wrote "we now have a new instance called relic-stretch to replace it".

guess i cookie-licked it i gotta take it ;)

If it was just the Apache part and a regular "web proxy" to click in Horizon i would have done it right now. The apache setup from puppet on the new instance looks fine.

But there is also an email part of this and the special setup in DNS zones in Horizon.

Change 455737 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] openstack::network: update private IP of relic.toolserver instance

checked and tested exim config on existing instance and adding a new alias, works. not much going on in exim.log at all

checked apache config and exim config on new instance have been generated by puppet, all looks ok

checked the DNS setup in Horizon, looks like it doesn't need a change and all we need is the Gerrit change above, afaict

should be just that hopefully and then testing works and sending an email to somebody

The mapping of aliases is in /etc/toolserver.aliases

Change 462004 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] toolserver_legacy: enable ssl module for httpd

Change 462004 merged by Dzahn:
[operations/puppet@production] toolserver_legacy: enable ssl module for httpd

Change 455737 merged by Dzahn:
[operations/puppet@production] openstack::network: update IPs of relic.toolserver


13:58 < mutante> !log and subdomains (wiki.toolserver, status.toolserver, stable.toolserver) legacy URLs have been switched to new stretch backend, away from trusty

13:59 < mutante> !log * also moved from eqiad to eqiad-r region in cloud vps, which gave it new IP addresses

Change 462012 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] network: add (labs-instances2-b-eqiad) to external_networks

Change 462012 abandoned by Dzahn:
network: add (labs-instances2-b-eqiad) to external_networks

< paravoid> the real problem is this: "The production MX servers, mx1001/2001 are the external SMTP for cloud VPS instances." :)

< paravoid> the solution here is to set up separate email relays for WMCS

How to migrate relic instance to relic-stretch-eqiad instance:

  • create new instance, run puppet
  • apply puppet role role::toollabs::legacy, run puppet again
  • scp root@relic.toolserver-legacy:/etc/toolserver.aliases .
  • scp root@relic.toolserver-legacy:/etc/acme/cert/* .
  • scp root@relic.toolserver-legacy:/etc/acme/key/toolserver.key .
  • scp toolserver.key root@relic-stretch-eqiad.toolserver-legacy:/etc/acme/key/
  • scp toolserver.c* root@relic-stretch-eqiad.toolserver-legacy:/etc/acme/cert/
  • scp toolserver.aliases root@relic-stretch-eqiad.toolserver-legacy:/etc/
  • run puppet again on new instance, should show no errors

Change 462023 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] openstack::network: update private IP of relic.toolserver

17:16 < mutante> !log toolserver-legacy associating floating IP (the existing one in eqiad) with relic-stretch-eqiad

17:19 < mutante> !log toolserver-legacy update DNS zone entries and remove eqiad-r IP, revert to eqiad IP

This is done. Things have switched away from "relic" on trusty and over to "relic-stretch-eqiad" , a stretch instance in eqiad.

switching from eqiad to eqiad-r has been reverted and should be seen as unrelated to removing trusty.

This also means email works fine now, just like it did before. Tested the aliases and it delivered to an external address.

Change 462023 merged by Dzahn:
[operations/puppet@production] openstack::network: update private IP of relic.toolserver

Mentioned in SAL (#wikimedia-cloud) [2018-09-21T23:12:10Z] <mutante> migration complete. shutting down trusty instance 'relic'. (T204564)

Mentioned in SAL (#wikimedia-cloud) [2018-09-21T23:25:16Z] <mutante> deleting now unused trusty instance relic (T204564) (/etc is backed up on new instance in /root/ just in case)

Dzahn edited projects, added SRE; removed Patch-For-Review.