Page MenuHomePhabricator
Create Task
Maniphest T253780

Upgrade cloudservices nodes to Debian Buster
Closed, ResolvedPublic
Actions

Description

Designate doesn't store any state on these boxes, and is an HA cluster, so that part should be simple.

PDNS has a local database which will need to be copied over to any newly-built host:

  • On old host: mysqldump pdns > pdns-dump.sql
  • stand up mysql
    1. cd /opt/wmf-mariadb104/
    2. ./scripts/mysql_install_db
  • Import data on new host: mysql> CREATE DATABASE pdns;
    1. mysql -p pdns < pdns-dump.sql
    2. # ^ empty password for this
    3. mysql > create database pdns;
  • bootstrap ip aliaser (this file will get filled in later)
    1. echo '{}' > /var/cache/labsaliaser/labs-ip-aliases.json
  • grants
    1. GRANT ALL PRIVILEGES ON pdns.* TO 'pdns'@'localhost' identified by '<password>
    2. GRANT ALL PRIVILEGES ON pdns.* TO 'pdns'@'ipv4' identified by '<password>
    3. GRANT ALL PRIVILEGES ON pdns.* TO 'pdns'@'ipv6' identified by '<password>

Details

Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

Andrew created this task.May 27 2020, 6:02 PM
Restricted Application removed a project: Patch-For-Review. · View Herald TranscriptMay 27 2020, 6:02 PM
Andrew updated the task description. (Show Details)May 27 2020, 6:12 PM
Andrew updated the task description. (Show Details)May 27 2020, 6:30 PM
gerritbot added a comment.May 27 2020, 10:29 PM

Change 599133 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] codfw1dev: add cloudservices2003-dev to the designate host list

https://gerrit.wikimedia.org/r/599133

gerritbot added a project: Patch-For-Review.May 27 2020, 10:29 PM

Change 599133 merged by Andrew Bogott:
[operations/puppet@production] codfw1dev: add cloudservices2003-dev to the designate host list

https://gerrit.wikimedia.org/r/599133

gerritbot added a comment.May 27 2020, 11:15 PM

Change 599137 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] M5 grants: allow designate access via ipv6

https://gerrit.wikimedia.org/r/599137

Stashbot added a comment.May 28 2020, 12:33 AM

Mentioned in SAL (#wikimedia-cloud) [2020-05-28T00:33:15Z] <andrewbogott> shutting down cloudservices2002-dev to see if we can live without it. This is in anticipation or rebuilding it entirely for T253780

gerritbot added a comment.May 28 2020, 1:30 PM

Change 599324 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Prepare cloudservices2002-dev for debian Buster

https://gerrit.wikimedia.org/r/599324

gerritbot added a comment.May 28 2020, 1:33 PM

Change 599137 merged by Andrew Bogott:
[operations/puppet@production] M5 grants: allow designate access via ipv6

https://gerrit.wikimedia.org/r/599137

gerritbot added a comment.May 28 2020, 1:34 PM

Change 599324 merged by Andrew Bogott:
[operations/puppet@production] Prepare cloudservices2002-dev for debian Buster

https://gerrit.wikimedia.org/r/599324

ops-monitoring-bot added a comment.May 28 2020, 2:01 PM

Script wmf-auto-reimage was launched by andrew on cumin1001.eqiad.wmnet for hosts:

cloudservices2002-dev.wikimedia.org

The log can be found in /var/log/wmf-auto-reimage/202005281401_andrew_62878_cloudservices2002-dev_wikimedia_org.log.

ops-monitoring-bot added a comment.May 28 2020, 2:01 PM

Completed auto-reimage of hosts:

['cloudservices2002-dev.wikimedia.org']

Of which those FAILED:

['cloudservices2002-dev.wikimedia.org']
ops-monitoring-bot added a comment.May 28 2020, 2:04 PM

Script wmf-auto-reimage was launched by andrew on cumin1001.eqiad.wmnet for hosts:

cloudservices2002-dev.wikimedia.org

The log can be found in /var/log/wmf-auto-reimage/202005281404_andrew_65222_cloudservices2002-dev_wikimedia_org.log.

Andrew updated the task description. (Show Details)May 28 2020, 2:54 PM
ops-monitoring-bot added a comment.May 28 2020, 2:58 PM

Completed auto-reimage of hosts:

['cloudservices2002-dev.wikimedia.org']

and were ALL successful.

gerritbot added a comment.May 28 2020, 3:16 PM

Change 599354 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] labs-ip-alias-dump.py: upgrade to python3

https://gerrit.wikimedia.org/r/599354

gerritbot added a comment.May 28 2020, 3:22 PM

Change 599354 merged by Andrew Bogott:
[operations/puppet@production] labs-ip-alias-dump.py: upgrade to python3

https://gerrit.wikimedia.org/r/599354

Andrew updated the task description. (Show Details)May 28 2020, 3:33 PM
gerritbot added a comment.May 28 2020, 9:34 PM

Change 599448 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] designate pools.yaml: Fix some hard-coded eqiad things

https://gerrit.wikimedia.org/r/599448

gerritbot added a comment.May 28 2020, 9:34 PM

Change 599449 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Designate: designate doesn't need to write to the pdns db anymore

https://gerrit.wikimedia.org/r/599449

gerritbot added a comment.May 28 2020, 9:34 PM

Change 599450 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] designate.conf.erb: remove pdns pool config

https://gerrit.wikimedia.org/r/599450

gerritbot added a comment.May 28 2020, 9:48 PM

Change 599448 merged by Andrew Bogott:
[operations/puppet@production] designate pools.yaml: Fix some hard-coded eqiad things

https://gerrit.wikimedia.org/r/599448

gerritbot added a comment.May 28 2020, 9:59 PM

Change 599450 merged by Andrew Bogott:
[operations/puppet@production] designate.conf.erb: remove pdns pool config

https://gerrit.wikimedia.org/r/599450

gerritbot added a comment.May 28 2020, 9:59 PM

Change 599449 merged by Andrew Bogott:
[operations/puppet@production] Designate: designate doesn't need to write to the pdns db anymore

https://gerrit.wikimedia.org/r/599449

gerritbot added a comment.May 29 2020, 10:16 PM

Change 600017 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] designate: allow mdns to listen on ipv6

https://gerrit.wikimedia.org/r/600017

gerritbot added a comment.May 30 2020, 1:59 AM

Change 600095 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Designate: have mdns use tcp rather than udp for axfr

https://gerrit.wikimedia.org/r/600095

gerritbot added a comment.May 30 2020, 2:01 AM

Change 600017 merged by Andrew Bogott:
[operations/puppet@production] designate: allow mdns to listen on ipv6

https://gerrit.wikimedia.org/r/600017

gerritbot added a comment.May 30 2020, 2:08 AM

Change 600095 merged by Andrew Bogott:
[operations/puppet@production] Designate: have mdns use tcp rather than udp for axfr

https://gerrit.wikimedia.org/r/600095

gerritbot added a comment.Jun 2 2020, 3:21 AM

Change 601551 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Rocky/Buster/Designate: a few live hacks to get things working on Buster

https://gerrit.wikimedia.org/r/601551

gerritbot added a comment.Jun 2 2020, 3:32 AM

Change 601551 merged by Andrew Bogott:
[operations/puppet@production] Rocky/Buster/Designate: a few live hacks to get things working on Buster

https://gerrit.wikimedia.org/r/601551

gerritbot added a comment.Jun 2 2020, 12:25 PM

Change 601711 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] wmcs resolv.conf: reduce timeout to 1s

https://gerrit.wikimedia.org/r/601711

gerritbot added a comment.Jun 2 2020, 12:36 PM

Change 601714 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] wmcs vms: stop using ns1 for resolving

https://gerrit.wikimedia.org/r/601714

Andrew triaged this task as Medium priority.Jun 2 2020, 4:15 PM
Andrew moved this task from Inbox to Doing on the cloud-services-team (Kanban) board.
gerritbot added a comment.Jun 9 2020, 12:47 PM

Change 601711 merged by Andrew Bogott:
[operations/puppet@production] wmcs resolv.conf: reduce timeout to 1s

https://gerrit.wikimedia.org/r/601711

gerritbot added a comment.Jun 9 2020, 1:06 PM

Change 601714 merged by Andrew Bogott:
[operations/puppet@production] wmcs vms: stop using ns1 for resolving

https://gerrit.wikimedia.org/r/601714

gerritbot added a comment.Jun 9 2020, 1:34 PM

Change 604010 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Horizon: put into maintenance mode for cloudservices rebuilds

https://gerrit.wikimedia.org/r/604010

Stashbot added a comment.Jun 9 2020, 2:01 PM

Mentioned in SAL (#wikimedia-cloud) [2020-06-09T14:01:21Z] <arturo> icinga downtime everything cloud* lab* for 2h (T253780)

gerritbot added a comment.Jun 9 2020, 2:04 PM

Change 604010 merged by Andrew Bogott:
[operations/puppet@production] Horizon: put into maintenance mode for cloudservices rebuilds

https://gerrit.wikimedia.org/r/604010

Andrew added a comment.Jun 9 2020, 2:07 PM

I'm pretty sure we can get an adequate dump with just

  1. mysqldump pdns > pdns-dump.sql
Stashbot added a comment.Jun 9 2020, 2:09 PM

Mentioned in SAL (#wikimedia-cloud) [2020-06-09T14:09:51Z] <andrewbogott> stopping puppet, all designate services and all pdns services on cloudservices1004 for T253780

gerritbot added a comment.Jun 9 2020, 2:11 PM

Change 604019 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Move cloudservices1003/1004 to Debian Buster

https://gerrit.wikimedia.org/r/604019

gerritbot added a comment.Jun 9 2020, 2:13 PM

Change 604019 merged by Andrew Bogott:
[operations/puppet@production] Move cloudservices1003/1004 to Debian Buster

https://gerrit.wikimedia.org/r/604019

ops-monitoring-bot added a comment.Jun 9 2020, 2:17 PM

Script wmf-auto-reimage was launched by andrew on cumin1001.eqiad.wmnet for hosts:

cloudservices1004.wikimedia.org

The log can be found in /var/log/wmf-auto-reimage/202006091416_andrew_126076_cloudservices1004_wikimedia_org.log.

ops-monitoring-bot added a comment.Jun 9 2020, 3:05 PM

Completed auto-reimage of hosts:

['cloudservices1004.wikimedia.org']

and were ALL successful.

Andrew updated the task description. (Show Details)Jun 9 2020, 3:10 PM
gerritbot added a comment.Jun 9 2020, 3:23 PM

Change 604049 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] cloud-vps resolve.conf: move traffic from recursor0 to recursor1

https://gerrit.wikimedia.org/r/604049

Stashbot added a comment.Jun 9 2020, 3:25 PM

Mentioned in SAL (#wikimedia-cloud) [2020-06-09T15:25:30Z] <arturo> icinga downtime everything cloud* lab* for 2h more (T253780)

gerritbot added a comment.Jun 9 2020, 3:35 PM

Change 604049 merged by Andrew Bogott:
[operations/puppet@production] cloud-vps resolve.conf: move traffic from recursor0 to recursor1

https://gerrit.wikimedia.org/r/604049

ops-monitoring-bot added a comment.Jun 9 2020, 4:17 PM

Script wmf-auto-reimage was launched by andrew on cumin1001.eqiad.wmnet for hosts:

cloudservices1003.wikimedia.org

The log can be found in /var/log/wmf-auto-reimage/202006091617_andrew_222628_cloudservices1003_wikimedia_org.log.

Samwalton9-WMF mentioned this in T254915: Wikipedia Library 500 internal server error.Jun 9 2020, 4:49 PM
ops-monitoring-bot added a comment.Jun 9 2020, 4:59 PM

Completed auto-reimage of hosts:

['cloudservices1003.wikimedia.org']

Of which those FAILED:

['cloudservices1003.wikimedia.org']
gerritbot added a comment.Jun 9 2020, 5:32 PM

Change 604084 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] cloud-vps resolv.conf: restore use of both recursors

https://gerrit.wikimedia.org/r/604084

gerritbot added a comment.Jun 9 2020, 5:33 PM

Change 604084 merged by Andrew Bogott:
[operations/puppet@production] cloud-vps resolv.conf: restore use of both recursors

https://gerrit.wikimedia.org/r/604084

Andrew closed this task as Resolved.Jun 9 2020, 5:53 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL