Page MenuHomePhabricator
Create Task
Maniphest T195686

Move puppetmaster to Stretch
Closed, ResolvedPublic
Actions

Description

Following prod: T184562: Upgrade Puppet Master Infrastructure to Debian Stretch
Already started this by spinning up deployment-puppetmaster03.deployment-prep.eqiad.wmflabs

  • deployment-apertium02
  • deployment-aqs01
  • deployment-aqs02
  • deployment-aqs03
  • deployment-cache-text04
  • deployment-cache-upload04
  • deployment-cassandra3-01
  • deployment-cassandra3-02
  • deployment-certcentral
  • deployment-certcentral-testclient
  • deployment-changeprop
  • deployment-chromium01
  • deployment-conf03
  • deployment-cpjobqueue
  • deployment-cumin
  • deployment-db03
  • deployment-db04
  • deployment-deploy1001 - puppet errors - T192561 - temporarily commented broken resource for migration
  • deployment-dumps-puppetmaster (despite this being a puppetmaster, it is not using itself)
  • deployment-elastic05
  • deployment-elastic06
  • deployment-elastic07
  • deployment-etcd-01
  • deployment-eventlog05 - puppet errors - T191109 - by temporarily removing problematic class
  • deployment-fluorine02
  • deployment-imagescaler01
  • deployment-imagescaler02
  • deployment-ircd
  • deployment-jobrunner03
  • deployment-kafka-jumbo-1
  • deployment-kafka-jumbo-2
  • deployment-kafka-main-1
  • deployment-kafka-main-2
  • deployment-logstash2
  • deployment-maps03
  • deployment-mathoid
  • deployment-mcs01
  • deployment-mediawiki-07
  • deployment-mediawiki-09
  • deployment-mediawiki06
  • deployment-memc04
  • deployment-memc05
  • deployment-memc06
  • deployment-memc07
  • deployment-mira
  • deployment-ms-be03 - puppet errors - T184236 - cherry-picked patch
  • deployment-ms-be04 - puppet errors - T184236 - cherry-picked patch
  • deployment-ms-fe02
  • deployment-mx - puppet errors - T184244 - temporarily removed problematic class
  • deployment-mx02 - puppet errors - related work at T184244, specific error untracked - temporarily removed problematic class
  • deployment-ores01
  • deployment-parsoid09
  • deployment-pdfrender02
  • deployment-poolcounter04
  • deployment-prometheus01
  • deployment-puppetdb01
  • deployment-puppetdb02
  • deployment-puppetmaster02 (probably no point pointing the agent from this to 03)
  • deployment-puppetmaster03
  • deployment-redis01 - puppet errors - T179371#3872776 - temporarily removed problematic class
  • deployment-redis02 - puppet errors - T179371#3872776 - temporarily removed problematic class
  • deployment-redis05
  • deployment-redis06
  • deployment-restbase01
  • deployment-restbase02
  • deployment-sca01
  • deployment-sca02
  • deployment-sca04
  • deployment-sentry01
  • deployment-snapshot01 - special case as this uses deployment-dumps-puppetmaster, am going to need to talk to @ArielGlenn about that
  • deployment-tin
  • deployment-urldownloader
  • deployment-webperf01 - was never set up on old puppetmaster anyway
  • deployment-zookeeper02
  • deployment-zotero01
Process
sudo -i
puppet agent -tv
# make hieradata change in horizon: puppetmaster: deployment-puppetmaster03.deployment-prep.eqiad.wmflabs
puppet agent -tv
cd /var/lib/puppet; mv ssl ssl_old; rm /usr/local/share/ca-certificates/Puppet_Internal_CA.crt; nano /usr/local/share/ca-certificates/Puppet_Internal_CA.crt
# copy in cert from another host already using deployment-puppetmaster03
update-ca-certificates --fresh
puppet agent -tv
# sign cert on deployment-puppetmaster03: puppet cert sign deployment-?.deployment-prep.eqiad.wmflabs
puppet agent -tv
puppet agent -tv

Details

SubjectRepoBranchLines +/-
Regenerate all certificates that were signed by the now decommed puppetmaster02labs/privatemaster+256 -255
deployment-prep: Replace etcd cert after puppetmaster changeoperations/puppetproduction+24 -23
Customize query in gerrit

Related Objects

Event Timeline

Krenair created this task.May 26 2018, 9:56 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 26 2018, 9:56 PM
Krenair updated the task description. (Show Details)May 27 2018, 12:04 AM
EddieGP triaged this task as Medium priority.May 27 2018, 8:23 AM
EddieGP moved this task from To Triage to Backlog on the Beta-Cluster-Infrastructure board.
Krenair raised the priority of this task from Medium to Needs Triage.May 27 2018, 2:38 PM
Krenair triaged this task as Medium priority.
Krenair updated the task description. (Show Details)
Krenair updated the task description. (Show Details)May 27 2018, 4:09 PM
Krenair updated the task description. (Show Details)May 27 2018, 4:12 PM
Krenair updated the task description. (Show Details)May 27 2018, 4:45 PM
Krenair updated the task description. (Show Details)May 27 2018, 5:06 PM
Krenair added a subscriber: ArielGlenn.
Krenair updated the task description. (Show Details)May 27 2018, 5:27 PM
Krenair updated the task description. (Show Details)May 27 2018, 5:33 PM
Krenair updated the task description. (Show Details)May 27 2018, 5:40 PM
Krenair updated the task description. (Show Details)
Krenair updated the task description. (Show Details)May 27 2018, 6:21 PM
Krenair updated the task description. (Show Details)May 27 2018, 6:43 PM
Krenair updated the task description. (Show Details)May 27 2018, 6:55 PM
Krenair updated the task description. (Show Details)May 27 2018, 7:00 PM
Krenair updated the task description. (Show Details)May 27 2018, 7:19 PM
Krenair updated the task description. (Show Details)May 27 2018, 7:31 PM
Krenair updated the task description. (Show Details)May 27 2018, 7:36 PM
Krenair updated the task description. (Show Details)May 27 2018, 7:41 PM
Krenair updated the task description. (Show Details)May 27 2018, 7:53 PM
Krenair updated the task description. (Show Details)May 27 2018, 8:04 PM
Krenair added a comment.May 27 2018, 8:14 PM

Have put in temporary overrides for the instances with broken puppet while I sort that out, new default: https://wikitech.wikimedia.org/w/index.php?title=Hiera:Deployment-prep&diff=next&oldid=1792365

Krenair updated the task description. (Show Details)May 27 2018, 8:17 PM
Krenair updated the task description. (Show Details)May 27 2018, 8:44 PM
Krenair updated the task description. (Show Details)May 27 2018, 8:56 PM
gerritbot subscribed.May 27 2018, 9:21 PM

Change 435715 had a related patch set uploaded (by Alex Monk; owner: Alex Monk):
[operations/puppet@production] Replace etcd cert after puppetmaster change

https://gerrit.wikimedia.org/r/435715

gerritbot added a project: Patch-For-Review.May 27 2018, 9:21 PM
Krenair updated the task description. (Show Details)May 27 2018, 9:26 PM
Krenair updated the task description. (Show Details)May 27 2018, 9:48 PM
Krenair updated the task description. (Show Details)May 27 2018, 10:07 PM
Krenair updated the task description. (Show Details)May 27 2018, 10:12 PM
Krenair updated the task description. (Show Details)
Krenair updated the task description. (Show Details)May 27 2018, 10:20 PM
Krenair added a comment.May 27 2018, 10:25 PM
krenair@deployment-cumin:~$ sudo cumin '*' 'grep "server = " /etc/puppet/puppet.conf'
75 hosts will be targeted:
deployment-apertium02.deployment-prep.eqiad.wmflabs,deployment-aqs[01-03].deployment-prep.eqiad.wmflabs,deployment-cache-text04.deployment-prep.eqiad.wmflabs,deployment-cache-upload04.deployment-prep.eqiad.wmflabs,deployment-cassandra3-[01-02].deployment-prep.eqiad.wmflabs,deployment-certcentral-testclient.deployment-prep.eqiad.wmflabs,deployment-certcentral.deployment-prep.eqiad.wmflabs,deployment-changeprop.deployment-prep.eqiad.wmflabs,deployment-chromium01.deployment-prep.eqiad.wmflabs,deployment-conf03.deployment-prep.eqiad.wmflabs,deployment-cpjobqueue.deployment-prep.eqiad.wmflabs,deployment-cumin.deployment-prep.eqiad.wmflabs,deployment-db[03-04].deployment-prep.eqiad.wmflabs,deployment-deploy1001.deployment-prep.eqiad.wmflabs,deployment-dumps-puppetmaster.deployment-prep.eqiad.wmflabs,deployment-elastic[05-07].deployment-prep.eqiad.wmflabs,deployment-etcd-01.deployment-prep.eqiad.wmflabs,deployment-eventlog05.deployment-prep.eqiad.wmflabs,deployment-fluorine02.deployment-prep.eqiad.wmflabs,deployment-imagescaler[01-02].deployment-prep.eqiad.wmflabs,deployment-ircd.deployment-prep.eqiad.wmflabs,deployment-jobrunner03.deployment-prep.eqiad.wmflabs,deployment-kafka-jumbo-[1-2].deployment-prep.eqiad.wmflabs,deployment-kafka-main-[1-2].deployment-prep.eqiad.wmflabs,deployment-logstash2.deployment-prep.eqiad.wmflabs,deployment-maps03.deployment-prep.eqiad.wmflabs,deployment-mathoid.deployment-prep.eqiad.wmflabs,deployment-mcs01.deployment-prep.eqiad.wmflabs,deployment-mediawiki06.deployment-prep.eqiad.wmflabs,deployment-mediawiki-[07,09].deployment-prep.eqiad.wmflabs,deployment-memc[04-07].deployment-prep.eqiad.wmflabs,deployment-mira.deployment-prep.eqiad.wmflabs,deployment-ms-be[03-04].deployment-prep.eqiad.wmflabs,deployment-ms-fe02.deployment-prep.eqiad.wmflabs,deployment-mx02.deployment-prep.eqiad.wmflabs,deployment-mx.deployment-prep.eqiad.wmflabs,deployment-ores01.deployment-prep.eqiad.wmflabs,deployment-parsoid09.deployment-prep.eqiad.wmflabs,deployment-pdfrender02.deployment-prep.eqiad.wmflabs,deployment-poolcounter04.deployment-prep.eqiad.wmflabs,deployment-prometheus01.deployment-prep.eqiad.wmflabs,deployment-puppetdb[01-02].deployment-prep.eqiad.wmflabs,deployment-puppetmaster[02-03].deployment-prep.eqiad.wmflabs,deployment-redis[01-02,05-06].deployment-prep.eqiad.wmflabs,deployment-restbase[01-02].deployment-prep.eqiad.wmflabs,deployment-sca[01-02,04].deployment-prep.eqiad.wmflabs,deployment-sentry01.deployment-prep.eqiad.wmflabs,deployment-snapshot01.deployment-prep.eqiad.wmflabs,deployment-tin.deployment-prep.eqiad.wmflabs,deployment-urldownloader.deployment-prep.eqiad.wmflabs,deployment-webperf01.deployment-prep.eqiad.wmflabs,deployment-zookeeper02.deployment-prep.eqiad.wmflabs,deployment-zotero01.deployment-prep.eqiad.wmflabs
Confirm to continue [y/n]? t
Confirm to continue [y/n]? y
===== NODE GROUP =====                                                                                                                                                                                      
(1) deployment-snapshot01.deployment-prep.eqiad.wmflabs                                                                                                                                                     
----- OUTPUT of 'grep "server = "...ppet/puppet.conf' -----                                                                                                                                                 
server = deployment-dumps-puppetmaster.deployment-prep.eqiad.wmflabs                                                                                                                                        
===== NODE GROUP =====                                                                                                                                                                                      
(73) deployment-apertium02.deployment-prep.eqiad.wmflabs,deployment-aqs[01-03].deployment-prep.eqiad.wmflabs,deployment-cache-text04.deployment-prep.eqiad.wmflabs,deployment-cache-upload04.deployment-prep.eqiad.wmflabs,deployment-cassandra3-[01-02].deployment-prep.eqiad.wmflabs,deployment-certcentral-testclient.deployment-prep.eqiad.wmflabs,deployment-certcentral.deployment-prep.eqiad.wmflabs,deployment-changeprop.deployment-prep.eqiad.wmflabs,deployment-chromium01.deployment-prep.eqiad.wmflabs,deployment-conf03.deployment-prep.eqiad.wmflabs,deployment-cpjobqueue.deployment-prep.eqiad.wmflabs,deployment-cumin.deployment-prep.eqiad.wmflabs,deployment-db[03-04].deployment-prep.eqiad.wmflabs,deployment-deploy1001.deployment-prep.eqiad.wmflabs,deployment-dumps-puppetmaster.deployment-prep.eqiad.wmflabs,deployment-elastic[05-07].deployment-prep.eqiad.wmflabs,deployment-etcd-01.deployment-prep.eqiad.wmflabs,deployment-eventlog05.deployment-prep.eqiad.wmflabs,deployment-fluorine02.deployment-prep.eqiad.wmflabs,deployment-imagescaler[01-02].deployment-prep.eqiad.wmflabs,deployment-ircd.deployment-prep.eqiad.wmflabs,deployment-jobrunner03.deployment-prep.eqiad.wmflabs,deployment-kafka-jumbo-[1-2].deployment-prep.eqiad.wmflabs,deployment-kafka-main-[1-2].deployment-prep.eqiad.wmflabs,deployment-logstash2.deployment-prep.eqiad.wmflabs,deployment-maps03.deployment-prep.eqiad.wmflabs,deployment-mathoid.deployment-prep.eqiad.wmflabs,deployment-mcs01.deployment-prep.eqiad.wmflabs,deployment-mediawiki06.deployment-prep.eqiad.wmflabs,deployment-mediawiki-[07,09].deployment-prep.eqiad.wmflabs,deployment-memc[04-07].deployment-prep.eqiad.wmflabs,deployment-mira.deployment-prep.eqiad.wmflabs,deployment-ms-be[03-04].deployment-prep.eqiad.wmflabs,deployment-ms-fe02.deployment-prep.eqiad.wmflabs,deployment-mx02.deployment-prep.eqiad.wmflabs,deployment-mx.deployment-prep.eqiad.wmflabs,deployment-ores01.deployment-prep.eqiad.wmflabs,deployment-parsoid09.deployment-prep.eqiad.wmflabs,deployment-pdfrender02.deployment-prep.eqiad.wmflabs,deployment-poolcounter04.deployment-prep.eqiad.wmflabs,deployment-prometheus01.deployment-prep.eqiad.wmflabs,deployment-puppetdb[01-02].deployment-prep.eqiad.wmflabs,deployment-puppetmaster03.deployment-prep.eqiad.wmflabs,deployment-redis[01-02,05-06].deployment-prep.eqiad.wmflabs,deployment-restbase[01-02].deployment-prep.eqiad.wmflabs,deployment-sca[01-02,04].deployment-prep.eqiad.wmflabs,deployment-sentry01.deployment-prep.eqiad.wmflabs,deployment-tin.deployment-prep.eqiad.wmflabs,deployment-urldownloader.deployment-prep.eqiad.wmflabs,deployment-webperf01.deployment-prep.eqiad.wmflabs,deployment-zookeeper02.deployment-prep.eqiad.wmflabs,deployment-zotero01.deployment-prep.eqiad.wmflabs
----- OUTPUT of 'grep "server = "...ppet/puppet.conf' -----                                                                                                                                                 
server = deployment-puppetmaster03.deployment-prep.eqiad.wmflabs                                                                                                                                            
===== NODE GROUP =====                                                                                                                                                                                      
(1) deployment-puppetmaster02.deployment-prep.eqiad.wmflabs                                                                                                                                                 
----- OUTPUT of 'grep "server = "...ppet/puppet.conf' -----                                                                                                                                                 
server = deployment-puppetmaster02.deployment-prep.eqiad.wmflabs                                                                                                                                            
================                                                                                                                                                                                            
PASS:  |█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100% (75/75) [01:18<00:00,  1.04s/hosts]         
FAIL:  |                                                                                                                                                          |   0% (0/75) [01:18<?, ?hosts/s]         
100.0% (75/75) success ratio (>= 100.0% threshold) for command: 'grep "server = "...ppet/puppet.conf'.
100.0% (75/75) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
Krenair closed this task as Resolved.May 27 2018, 10:44 PM

Request ID Action Start Time User ID Message
req-c1d7511f-d09e-45fd-aef8-a3748c4a3b66 Stop 27 May 2018, 10:33 p.m. krenair -

Will delete the old instance in a few weeks, and will talk to @ArielGlenn about what we're doing with -snapshot01 under the dumps puppetmaster. That might need moving to the -puppetmaster03 CA + PuppetDB at some point, though it would be ideal if we could roll the changes into a cherry-pick on -puppetmaster03.

gerritbot added a comment.May 29 2018, 6:34 PM

Change 435715 merged by Dzahn:
[operations/puppet@production] deployment-prep: Replace etcd cert after puppetmaster change

https://gerrit.wikimedia.org/r/435715

Krenair updated the task description. (Show Details)Jun 2 2018, 2:25 PM
Krenair removed a project: Patch-For-Review.
gerritbot added a comment.Jun 12 2018, 8:51 PM

Change 440016 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[labs/private@master] Regenerate all certificates that were signed by the now decommed puppetmaster02

https://gerrit.wikimedia.org/r/440016

gerritbot added a project: Patch-For-Review.Jun 12 2018, 8:51 PM

Change 440016 merged by Ottomata:
[labs/private@master] Regenerate all certificates that were signed by the now decommed puppetmaster02

https://gerrit.wikimedia.org/r/440016

• Vvjjkkii renamed this task from Move puppetmaster to Stretch to l7baaaaaaa.Jul 1 2018, 1:07 AM
• Vvjjkkii reopened this task as Open.
• Vvjjkkii removed Krenair as the assignee of this task.
• Vvjjkkii raised the priority of this task from Medium to High.
• Vvjjkkii added projects: CheckUser, Connected-Open-Heritage-Batch-uploads (RAÄ-KMB_1_2017-02), Tamil-Sites, Gamepress, Hashtags, Jade, KartoEditor, Language-2018-Apr-June, New-Editor-Experiences, Mail, TCB-Team (now WMDE-TechWish).
• Vvjjkkii updated the task description. (Show Details)
• Vvjjkkii removed subscribers: gerritbot, Aklapper.
ArielGlenn renamed this task from l7baaaaaaa to Move puppetmaster to Stretch.Jul 1 2018, 8:21 AM
ArielGlenn closed this task as Resolved.
ArielGlenn assigned this task to Krenair.
ArielGlenn lowered the priority of this task from High to Medium.
ArielGlenn removed projects: TCB-Team (now WMDE-TechWish), Mail, New-Editor-Experiences, Language-2018-Apr-June, KartoEditor, Jade, Hashtags, Gamepress, Tamil-Sites, Connected-Open-Heritage-Batch-uploads (RAÄ-KMB_1_2017-02), CheckUser.
ArielGlenn updated the task description. (Show Details)
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL