Page MenuHomePhabricator

Move puppetmaster to Stretch
Closed, ResolvedPublic

Description

Following prod: T184562: Upgrade Puppet Master Infrastructure to Debian Stretch
Already started this by spinning up deployment-puppetmaster03.deployment-prep.eqiad.wmflabs

  • deployment-apertium02
  • deployment-aqs01
  • deployment-aqs02
  • deployment-aqs03
  • deployment-cache-text04
  • deployment-cache-upload04
  • deployment-cassandra3-01
  • deployment-cassandra3-02
  • deployment-certcentral
  • deployment-certcentral-testclient
  • deployment-changeprop
  • deployment-chromium01
  • deployment-conf03
  • deployment-cpjobqueue
  • deployment-cumin
  • deployment-db03
  • deployment-db04
  • deployment-deploy1001 - puppet errors - T192561 - temporarily commented broken resource for migration
  • deployment-dumps-puppetmaster (despite this being a puppetmaster, it is not using itself)
  • deployment-elastic05
  • deployment-elastic06
  • deployment-elastic07
  • deployment-etcd-01
  • deployment-eventlog05 - puppet errors - T191109 - by temporarily removing problematic class
  • deployment-fluorine02
  • deployment-imagescaler01
  • deployment-imagescaler02
  • deployment-ircd
  • deployment-jobrunner03
  • deployment-kafka-jumbo-1
  • deployment-kafka-jumbo-2
  • deployment-kafka-main-1
  • deployment-kafka-main-2
  • deployment-logstash2
  • deployment-maps03
  • deployment-mathoid
  • deployment-mcs01
  • deployment-mediawiki-07
  • deployment-mediawiki-09
  • deployment-mediawiki06
  • deployment-memc04
  • deployment-memc05
  • deployment-memc06
  • deployment-memc07
  • deployment-mira
  • deployment-ms-be03 - puppet errors - T184236 - cherry-picked patch
  • deployment-ms-be04 - puppet errors - T184236 - cherry-picked patch
  • deployment-ms-fe02
  • deployment-mx - puppet errors - T184244 - temporarily removed problematic class
  • deployment-mx02 - puppet errors - related work at T184244, specific error untracked - temporarily removed problematic class
  • deployment-ores01
  • deployment-parsoid09
  • deployment-pdfrender02
  • deployment-poolcounter04
  • deployment-prometheus01
  • deployment-puppetdb01
  • deployment-puppetdb02
  • deployment-puppetmaster02 (probably no point pointing the agent from this to 03)
  • deployment-puppetmaster03
  • deployment-redis01 - puppet errors - T179371#3872776 - temporarily removed problematic class
  • deployment-redis02 - puppet errors - T179371#3872776 - temporarily removed problematic class
  • deployment-redis05
  • deployment-redis06
  • deployment-restbase01
  • deployment-restbase02
  • deployment-sca01
  • deployment-sca02
  • deployment-sca04
  • deployment-sentry01
  • deployment-snapshot01 - special case as this uses deployment-dumps-puppetmaster, am going to need to talk to @ArielGlenn about that
  • deployment-tin
  • deployment-urldownloader
  • deployment-webperf01 - was never set up on old puppetmaster anyway
  • deployment-zookeeper02
  • deployment-zotero01
Process
sudo -i
puppet agent -tv
# make hieradata change in horizon: puppetmaster: deployment-puppetmaster03.deployment-prep.eqiad.wmflabs
puppet agent -tv
cd /var/lib/puppet; mv ssl ssl_old; rm /usr/local/share/ca-certificates/Puppet_Internal_CA.crt; nano /usr/local/share/ca-certificates/Puppet_Internal_CA.crt
# copy in cert from another host already using deployment-puppetmaster03
update-ca-certificates --fresh
puppet agent -tv
# sign cert on deployment-puppetmaster03: puppet cert sign deployment-?.deployment-prep.eqiad.wmflabs
puppet agent -tv
puppet agent -tv

Event Timeline

Krenair created this task.May 26 2018, 9:56 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 26 2018, 9:56 PM
Krenair updated the task description. (Show Details)May 27 2018, 12:04 AM
EddieGP triaged this task as Normal priority.May 27 2018, 8:23 AM
EddieGP moved this task from To Triage to Backlog on the Beta-Cluster-Infrastructure board.
Krenair raised the priority of this task from Normal to Needs Triage.May 27 2018, 2:38 PM
Krenair triaged this task as Normal priority.
Krenair updated the task description. (Show Details)
Krenair updated the task description. (Show Details)May 27 2018, 4:09 PM
Krenair updated the task description. (Show Details)May 27 2018, 4:12 PM
Krenair updated the task description. (Show Details)May 27 2018, 4:45 PM
Krenair updated the task description. (Show Details)May 27 2018, 5:06 PM
Krenair added a subscriber: ArielGlenn.
Krenair updated the task description. (Show Details)May 27 2018, 5:27 PM
Krenair updated the task description. (Show Details)May 27 2018, 5:33 PM
Krenair updated the task description. (Show Details)May 27 2018, 5:40 PM
Krenair updated the task description. (Show Details)
Krenair updated the task description. (Show Details)May 27 2018, 6:21 PM
Krenair updated the task description. (Show Details)May 27 2018, 6:43 PM
Krenair updated the task description. (Show Details)May 27 2018, 6:55 PM
Krenair updated the task description. (Show Details)May 27 2018, 7:00 PM
Krenair updated the task description. (Show Details)May 27 2018, 7:19 PM
Krenair updated the task description. (Show Details)May 27 2018, 7:31 PM
Krenair updated the task description. (Show Details)May 27 2018, 7:36 PM
Krenair updated the task description. (Show Details)May 27 2018, 7:41 PM
Krenair updated the task description. (Show Details)May 27 2018, 7:53 PM
Krenair updated the task description. (Show Details)May 27 2018, 8:04 PM

Have put in temporary overrides for the instances with broken puppet while I sort that out, new default: https://wikitech.wikimedia.org/w/index.php?title=Hiera:Deployment-prep&diff=next&oldid=1792365

Krenair updated the task description. (Show Details)May 27 2018, 8:17 PM
Krenair updated the task description. (Show Details)May 27 2018, 8:44 PM
Krenair updated the task description. (Show Details)May 27 2018, 8:56 PM

Change 435715 had a related patch set uploaded (by Alex Monk; owner: Alex Monk):
[operations/puppet@production] Replace etcd cert after puppetmaster change

https://gerrit.wikimedia.org/r/435715

Krenair updated the task description. (Show Details)May 27 2018, 9:26 PM
Krenair updated the task description. (Show Details)May 27 2018, 9:48 PM
Krenair updated the task description. (Show Details)May 27 2018, 10:07 PM
Krenair updated the task description. (Show Details)May 27 2018, 10:12 PM
Krenair updated the task description. (Show Details)
Krenair updated the task description. (Show Details)May 27 2018, 10:20 PM
krenair@deployment-cumin:~$ sudo cumin '*' 'grep "server = " /etc/puppet/puppet.conf'
75 hosts will be targeted:
deployment-apertium02.deployment-prep.eqiad.wmflabs,deployment-aqs[01-03].deployment-prep.eqiad.wmflabs,deployment-cache-text04.deployment-prep.eqiad.wmflabs,deployment-cache-upload04.deployment-prep.eqiad.wmflabs,deployment-cassandra3-[01-02].deployment-prep.eqiad.wmflabs,deployment-certcentral-testclient.deployment-prep.eqiad.wmflabs,deployment-certcentral.deployment-prep.eqiad.wmflabs,deployment-changeprop.deployment-prep.eqiad.wmflabs,deployment-chromium01.deployment-prep.eqiad.wmflabs,deployment-conf03.deployment-prep.eqiad.wmflabs,deployment-cpjobqueue.deployment-prep.eqiad.wmflabs,deployment-cumin.deployment-prep.eqiad.wmflabs,deployment-db[03-04].deployment-prep.eqiad.wmflabs,deployment-deploy1001.deployment-prep.eqiad.wmflabs,deployment-dumps-puppetmaster.deployment-prep.eqiad.wmflabs,deployment-elastic[05-07].deployment-prep.eqiad.wmflabs,deployment-etcd-01.deployment-prep.eqiad.wmflabs,deployment-eventlog05.deployment-prep.eqiad.wmflabs,deployment-fluorine02.deployment-prep.eqiad.wmflabs,deployment-imagescaler[01-02].deployment-prep.eqiad.wmflabs,deployment-ircd.deployment-prep.eqiad.wmflabs,deployment-jobrunner03.deployment-prep.eqiad.wmflabs,deployment-kafka-jumbo-[1-2].deployment-prep.eqiad.wmflabs,deployment-kafka-main-[1-2].deployment-prep.eqiad.wmflabs,deployment-logstash2.deployment-prep.eqiad.wmflabs,deployment-maps03.deployment-prep.eqiad.wmflabs,deployment-mathoid.deployment-prep.eqiad.wmflabs,deployment-mcs01.deployment-prep.eqiad.wmflabs,deployment-mediawiki06.deployment-prep.eqiad.wmflabs,deployment-mediawiki-[07,09].deployment-prep.eqiad.wmflabs,deployment-memc[04-07].deployment-prep.eqiad.wmflabs,deployment-mira.deployment-prep.eqiad.wmflabs,deployment-ms-be[03-04].deployment-prep.eqiad.wmflabs,deployment-ms-fe02.deployment-prep.eqiad.wmflabs,deployment-mx02.deployment-prep.eqiad.wmflabs,deployment-mx.deployment-prep.eqiad.wmflabs,deployment-ores01.deployment-prep.eqiad.wmflabs,deployment-parsoid09.deployment-prep.eqiad.wmflabs,deployment-pdfrender02.deployment-prep.eqiad.wmflabs,deployment-poolcounter04.deployment-prep.eqiad.wmflabs,deployment-prometheus01.deployment-prep.eqiad.wmflabs,deployment-puppetdb[01-02].deployment-prep.eqiad.wmflabs,deployment-puppetmaster[02-03].deployment-prep.eqiad.wmflabs,deployment-redis[01-02,05-06].deployment-prep.eqiad.wmflabs,deployment-restbase[01-02].deployment-prep.eqiad.wmflabs,deployment-sca[01-02,04].deployment-prep.eqiad.wmflabs,deployment-sentry01.deployment-prep.eqiad.wmflabs,deployment-snapshot01.deployment-prep.eqiad.wmflabs,deployment-tin.deployment-prep.eqiad.wmflabs,deployment-urldownloader.deployment-prep.eqiad.wmflabs,deployment-webperf01.deployment-prep.eqiad.wmflabs,deployment-zookeeper02.deployment-prep.eqiad.wmflabs,deployment-zotero01.deployment-prep.eqiad.wmflabs
Confirm to continue [y/n]? t
Confirm to continue [y/n]? y
===== NODE GROUP =====                                                                                                                                                                                      
(1) deployment-snapshot01.deployment-prep.eqiad.wmflabs                                                                                                                                                     
----- OUTPUT of 'grep "server = "...ppet/puppet.conf' -----                                                                                                                                                 
server = deployment-dumps-puppetmaster.deployment-prep.eqiad.wmflabs                                                                                                                                        
===== NODE GROUP =====                                                                                                                                                                                      
(73) deployment-apertium02.deployment-prep.eqiad.wmflabs,deployment-aqs[01-03].deployment-prep.eqiad.wmflabs,deployment-cache-text04.deployment-prep.eqiad.wmflabs,deployment-cache-upload04.deployment-prep.eqiad.wmflabs,deployment-cassandra3-[01-02].deployment-prep.eqiad.wmflabs,deployment-certcentral-testclient.deployment-prep.eqiad.wmflabs,deployment-certcentral.deployment-prep.eqiad.wmflabs,deployment-changeprop.deployment-prep.eqiad.wmflabs,deployment-chromium01.deployment-prep.eqiad.wmflabs,deployment-conf03.deployment-prep.eqiad.wmflabs,deployment-cpjobqueue.deployment-prep.eqiad.wmflabs,deployment-cumin.deployment-prep.eqiad.wmflabs,deployment-db[03-04].deployment-prep.eqiad.wmflabs,deployment-deploy1001.deployment-prep.eqiad.wmflabs,deployment-dumps-puppetmaster.deployment-prep.eqiad.wmflabs,deployment-elastic[05-07].deployment-prep.eqiad.wmflabs,deployment-etcd-01.deployment-prep.eqiad.wmflabs,deployment-eventlog05.deployment-prep.eqiad.wmflabs,deployment-fluorine02.deployment-prep.eqiad.wmflabs,deployment-imagescaler[01-02].deployment-prep.eqiad.wmflabs,deployment-ircd.deployment-prep.eqiad.wmflabs,deployment-jobrunner03.deployment-prep.eqiad.wmflabs,deployment-kafka-jumbo-[1-2].deployment-prep.eqiad.wmflabs,deployment-kafka-main-[1-2].deployment-prep.eqiad.wmflabs,deployment-logstash2.deployment-prep.eqiad.wmflabs,deployment-maps03.deployment-prep.eqiad.wmflabs,deployment-mathoid.deployment-prep.eqiad.wmflabs,deployment-mcs01.deployment-prep.eqiad.wmflabs,deployment-mediawiki06.deployment-prep.eqiad.wmflabs,deployment-mediawiki-[07,09].deployment-prep.eqiad.wmflabs,deployment-memc[04-07].deployment-prep.eqiad.wmflabs,deployment-mira.deployment-prep.eqiad.wmflabs,deployment-ms-be[03-04].deployment-prep.eqiad.wmflabs,deployment-ms-fe02.deployment-prep.eqiad.wmflabs,deployment-mx02.deployment-prep.eqiad.wmflabs,deployment-mx.deployment-prep.eqiad.wmflabs,deployment-ores01.deployment-prep.eqiad.wmflabs,deployment-parsoid09.deployment-prep.eqiad.wmflabs,deployment-pdfrender02.deployment-prep.eqiad.wmflabs,deployment-poolcounter04.deployment-prep.eqiad.wmflabs,deployment-prometheus01.deployment-prep.eqiad.wmflabs,deployment-puppetdb[01-02].deployment-prep.eqiad.wmflabs,deployment-puppetmaster03.deployment-prep.eqiad.wmflabs,deployment-redis[01-02,05-06].deployment-prep.eqiad.wmflabs,deployment-restbase[01-02].deployment-prep.eqiad.wmflabs,deployment-sca[01-02,04].deployment-prep.eqiad.wmflabs,deployment-sentry01.deployment-prep.eqiad.wmflabs,deployment-tin.deployment-prep.eqiad.wmflabs,deployment-urldownloader.deployment-prep.eqiad.wmflabs,deployment-webperf01.deployment-prep.eqiad.wmflabs,deployment-zookeeper02.deployment-prep.eqiad.wmflabs,deployment-zotero01.deployment-prep.eqiad.wmflabs
----- OUTPUT of 'grep "server = "...ppet/puppet.conf' -----                                                                                                                                                 
server = deployment-puppetmaster03.deployment-prep.eqiad.wmflabs                                                                                                                                            
===== NODE GROUP =====                                                                                                                                                                                      
(1) deployment-puppetmaster02.deployment-prep.eqiad.wmflabs                                                                                                                                                 
----- OUTPUT of 'grep "server = "...ppet/puppet.conf' -----                                                                                                                                                 
server = deployment-puppetmaster02.deployment-prep.eqiad.wmflabs                                                                                                                                            
================                                                                                                                                                                                            
PASS:  |█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100% (75/75) [01:18<00:00,  1.04s/hosts]         
FAIL:  |                                                                                                                                                          |   0% (0/75) [01:18<?, ?hosts/s]         
100.0% (75/75) success ratio (>= 100.0% threshold) for command: 'grep "server = "...ppet/puppet.conf'.
100.0% (75/75) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
Krenair closed this task as Resolved.May 27 2018, 10:44 PM

Request ID Action Start Time User ID Message
req-c1d7511f-d09e-45fd-aef8-a3748c4a3b66 Stop 27 May 2018, 10:33 p.m. krenair -

Will delete the old instance in a few weeks, and will talk to @ArielGlenn about what we're doing with -snapshot01 under the dumps puppetmaster. That might need moving to the -puppetmaster03 CA + PuppetDB at some point, though it would be ideal if we could roll the changes into a cherry-pick on -puppetmaster03.

Change 435715 merged by Dzahn:
[operations/puppet@production] deployment-prep: Replace etcd cert after puppetmaster change

https://gerrit.wikimedia.org/r/435715

Krenair updated the task description. (Show Details)Jun 2 2018, 2:25 PM
Krenair removed a project: Patch-For-Review.

Change 440016 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[labs/private@master] Regenerate all certificates that were signed by the now decommed puppetmaster02

https://gerrit.wikimedia.org/r/440016

Change 440016 merged by Ottomata:
[labs/private@master] Regenerate all certificates that were signed by the now decommed puppetmaster02

https://gerrit.wikimedia.org/r/440016

Vvjjkkii renamed this task from Move puppetmaster to Stretch to l7baaaaaaa.Jul 1 2018, 1:07 AM
Vvjjkkii reopened this task as Open.
Vvjjkkii removed Krenair as the assignee of this task.
Vvjjkkii raised the priority of this task from Normal to High.
Vvjjkkii updated the task description. (Show Details)
Vvjjkkii removed subscribers: gerritbot, Aklapper.
ArielGlenn renamed this task from l7baaaaaaa to Move puppetmaster to Stretch.Jul 1 2018, 8:21 AM
ArielGlenn closed this task as Resolved.
ArielGlenn assigned this task to Krenair.
ArielGlenn lowered the priority of this task from High to Normal.
ArielGlenn updated the task description. (Show Details)