Page MenuHomePhabricator

cloudvps: toolserver-legacy project trusty deprecation
Closed, ResolvedPublic

Description

Ubuntu Trusty is no longer available in Cloud VPS since Nov 2017 for new instances. However, the EOL of Trusty is approaching in 2019 and we need to move to Debian Stretch before that date.

All instances in the toolserver-legacy project needs to upgrade as soon as possible.

The list of affected VMs is:

  • relic.toolserver-legacy.eqiad.wmflabs

Listed administrator are:

More info in openstack browser: https://tools.wmflabs.org/openstack-browser/project/toolserver-legacy

Related Objects

Event Timeline

Krenair triaged this task as Medium priority.Sep 17 2018, 4:57 PM
Krenair created this task.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 17 2018, 4:57 PM
Dzahn added a comment.Sep 17 2018, 9:12 PM

I uploaded this 3 weeks ago but i'm not sure what the status is even though i wrote "we now have a new instance called relic-stretch to replace it".

https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/455737/

Dzahn claimed this task.Sep 17 2018, 9:16 PM

guess i cookie-licked it i gotta take it ;)

Dzahn added a comment.EditedSep 17 2018, 11:50 PM

If it was just the Apache part and a regular "web proxy" to click in Horizon i would have done it right now. The apache setup from puppet on the new instance looks fine.

But there is also an email part of this and the special setup in DNS zones in Horizon.

Change 455737 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] openstack::network: update private IP of relic.toolserver instance

https://gerrit.wikimedia.org/r/455737

checked and tested exim config on existing instance and adding a new alias, works. not much going on in exim.log at all

checked apache config and exim config on new instance have been generated by puppet, all looks ok

checked the DNS setup in Horizon, looks like it doesn't need a change and all we need is the Gerrit change above, afaict

should be just that hopefully and then testing https://www.toolserver.org/ works and sending an email to somebody @toolserver.org

The mapping of aliases is in /etc/toolserver.aliases

Change 462004 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] toolserver_legacy: enable ssl module for httpd

https://gerrit.wikimedia.org/r/462004

Change 462004 merged by Dzahn:
[operations/puppet@production] toolserver_legacy: enable ssl module for httpd

https://gerrit.wikimedia.org/r/462004

Change 455737 merged by Dzahn:
[operations/puppet@production] openstack::network: update IPs of relic.toolserver

https://gerrit.wikimedia.org/r/455737

Dzahn added a comment.Sep 21 2018, 7:04 PM

1http://www.toolserver.org
2https://www.toolserver.org
3http://toolserver.org
4https://toolserver.org
5http://stable.toolserver.org
6https:/stable.toolserver.org
7http://wiki.toolserver.org
8https://wiki.toolserver.org/wiki/Foo
9https://toolserver.org/~alexz/pop/requests.php
10http://toolserver.org/~apper/pd
11

Dzahn added a comment.Sep 21 2018, 7:04 PM

13:58 < mutante> !log toolserver.org and subdomains (wiki.toolserver, status.toolserver, stable.toolserver) legacy URLs have been switched to new stretch backend, away from trusty

13:59 < mutante> !log *.toolserver.org also moved from eqiad to eqiad-r region in cloud vps, which gave it new IP addresses

Change 462012 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] network: add 172.16.0.0/21 (labs-instances2-b-eqiad) to external_networks

https://gerrit.wikimedia.org/r/462012

Change 462012 abandoned by Dzahn:
network: add 172.16.0.0/21 (labs-instances2-b-eqiad) to external_networks

Reason:
< paravoid> the real problem is this: "The production MX servers, mx1001/2001 are the external SMTP for cloud VPS instances." :)

< paravoid> the solution here is to set up separate email relays for WMCS

https://gerrit.wikimedia.org/r/462012

Dzahn added a comment.Sep 21 2018, 8:39 PM

How to migrate relic instance to relic-stretch-eqiad instance:

  • create new instance, run puppet
  • apply puppet role role::toollabs::legacy, run puppet again
  • scp root@relic.toolserver-legacy:/etc/toolserver.aliases .
  • scp root@relic.toolserver-legacy:/etc/acme/cert/* .
  • scp root@relic.toolserver-legacy:/etc/acme/key/toolserver.key .
  • scp toolserver.key root@relic-stretch-eqiad.toolserver-legacy:/etc/acme/key/
  • scp toolserver.c* root@relic-stretch-eqiad.toolserver-legacy:/etc/acme/cert/
  • scp toolserver.aliases root@relic-stretch-eqiad.toolserver-legacy:/etc/
  • run puppet again on new instance, should show no errors

Change 462023 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] openstack::network: update private IP of relic.toolserver

https://gerrit.wikimedia.org/r/462023

Dzahn added a comment.EditedSep 21 2018, 9:36 PM

17:16 < mutante> !log toolserver-legacy associating floating IP (the existing one in eqiad) with relic-stretch-eqiad

17:19 < mutante> !log toolserver-legacy update DNS zone entries and remove eqiad-r IP, revert to eqiad IP

This is done. Things have switched away from "relic" on trusty and over to "relic-stretch-eqiad" , a stretch instance in eqiad.

switching from eqiad to eqiad-r has been reverted and should be seen as unrelated to removing trusty.

This also means email works fine now, just like it did before. Tested the aliases and it delivered to an external address.

Change 462023 merged by Dzahn:
[operations/puppet@production] openstack::network: update private IP of relic.toolserver

https://gerrit.wikimedia.org/r/462023

Mentioned in SAL (#wikimedia-cloud) [2018-09-21T23:12:10Z] <mutante> migration complete. shutting down trusty instance 'relic'. (T204564)

Mentioned in SAL (#wikimedia-cloud) [2018-09-21T23:25:16Z] <mutante> deleting now unused trusty instance relic (T204564) (/etc is backed up on new instance in /root/ just in case)

Dzahn closed this task as Resolved.Sep 21 2018, 11:25 PM
Dzahn edited projects, added Operations; removed Patch-For-Review.