Page MenuHomePhabricator
Create Task
Maniphest T189891

Failover puppet ca service from eqiad to codfw
Closed, ResolvedPublic
Actions

Description

In order to rebuild puppetmaster1001 with stretch we will need to first failover the puppet ca service to puppetmaster2001. Creating a task to prepare for this.

Puppet CA failover process for review

  1. Disable puppet across the fleet
    1. neodymium:~$ sudo cumin -p 95 -b 100 '*' "disable-puppet 'temporarily disabled for puppet ca relocation - T189891 - godog'"
  2. Ensure rsync/git (ca, private and volatile) destinations are up to date on puppetmaster2001
    1. /var/lib/puppet/server/ssl/ca
    2. /var/lib/puppet/volatile
    3. /srv/private/
  3. Make backup copies of puppetmaster[12]001:/var/lib/puppet to neodymium/sarin
  4. Merge change updating puppetmaster::ca_server: puppetmaster2001.codfw.wmnet in hiera (https://gerrit.wikimedia.org/r/c/420721/) in order to...
    1. Repoint puppet agents ca_server to puppetmaster2001.codfw.wmnet
    2. Repoint apache frontend proxypass entries to puppetmaster2001.codfw.wmnet
    3. Reverse the direction of the puppetmaster rsync to puppetmaster2001 -> puppetmaster1001
  5. Enable and run puppet on puppetmaster1001
  6. Enable and run puppet on puppetmaster2001
  7. Enable and run puppet on a few canary hosts (puppet agents)
  8. Enable and force puppet agent run across fleet
    1. open a screen/tmux on neodymium or sarin and run:
    2. sudo cumin -p 70 -b 15 '*' "run-puppet-agent -q -e 'temporarily disabled for puppet ca relocation - T189891 - godog'"

Details

SubjectRepoBranchLines +/-
puppetmaster: lock commits on /srv/private on non-master hostsoperations/puppetproduction+14 -0
Revert "hieradata: use puppetmaster2001 as ca_server"operations/puppetproduction+1 -1
puppetmaster: install keypair for 'puppet' when running as CAoperations/puppetproduction+45 -0
Remove non-authoritative SRV puppet recordsoperations/dnsmaster+0 -24
hieradata: use puppetmaster2001 as ca_serveroperations/puppetproduction+1 -1
Customize query in gerrit

Related Objects
Search...

Event Timeline

herron triaged this task as Medium priority.Mar 16 2018, 4:59 PM
herron created this task.
This comment was removed by herron.
gerritbot added a comment.Mar 20 2018, 1:52 PM

Change 420705 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] puppetmaster: lock commits on /srv/private on non-master hosts

https://gerrit.wikimedia.org/r/420705

gerritbot added a project: Patch-For-Review.Mar 20 2018, 1:52 PM
gerritbot added a comment.Mar 20 2018, 2:37 PM

Change 420721 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] hieradata: use puppetmaster2001 as ca_server

https://gerrit.wikimedia.org/r/420721

fgiunchedi updated the task description. (Show Details)Mar 20 2018, 5:22 PM
herron updated the task description. (Show Details)Mar 21 2018, 2:43 PM
fgiunchedi mentioned this in T184562: Upgrade Puppet Master Infrastructure to Debian Stretch.Mar 21 2018, 4:16 PM
Volans updated the task description. (Show Details)Mar 22 2018, 11:13 AM
Stashbot added a comment.Mar 22 2018, 12:12 PM

Mentioned in SAL (#wikimedia-operations) [2018-03-22T12:12:56Z] <godog> stopping puppet fleetwide for ca migration - T189891

gerritbot added a comment.Mar 22 2018, 12:17 PM

Change 420721 merged by Filippo Giunchedi:
[operations/puppet@production] hieradata: use puppetmaster2001 as ca_server

https://gerrit.wikimedia.org/r/420721

Stashbot added a comment.Mar 22 2018, 12:20 PM

Mentioned in SAL (#wikimedia-operations) [2018-03-22T12:20:39Z] <godog> running puppet on puppetmaster[21]001 - T189891

Stashbot added a comment.Mar 22 2018, 1:23 PM

Mentioned in SAL (#wikimedia-operations) [2018-03-22T13:23:27Z] <godog> reenabling puppet fleetwide to enable CA switch - T189891

Stashbot added a comment.Mar 22 2018, 3:23 PM

Mentioned in SAL (#wikimedia-operations) [2018-03-22T15:23:26Z] <ottomata> ran puppet-merge on puppetmaster2001, got ssh: connect to host puppetmaster1001.eqiad.wmnet port 22: Connection timed out, hope all is ok. T189891

fgiunchedi added a comment.Mar 22 2018, 4:20 PM

Problems discovered today during the ca switchover:

  1. permissions during rsync for ca/volatile are set based on uids apparently, not user names, thus some files were owned by nagios and puppet-master refused to start
  2. We need to rsync the "puppet" keypair used by the server as well, namely /var/lib/puppet/server/ssl/private_keys/puppet.pem and /var/lib/puppet/server/ssl/certs/puppet.pem for puppet-server to start properly. Said keypair was autogenerated once the master was in ca mode (current theory) and thus wouldn't match what the server had cached from puppetmaster1001
gerritbot added a comment.Mar 26 2018, 9:02 AM

Change 421839 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/dns@master] Remove non-authoritative SRV puppet records

https://gerrit.wikimedia.org/r/421839

gerritbot added a comment.Mar 26 2018, 9:18 AM

Change 421842 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] puppetmaster: install keypair for 'puppet' when running as CA

https://gerrit.wikimedia.org/r/421842

gerritbot added a comment.Mar 26 2018, 1:49 PM

Change 421839 merged by Filippo Giunchedi:
[operations/dns@master] Remove non-authoritative SRV puppet records

https://gerrit.wikimedia.org/r/421839

gerritbot added a comment.Mar 26 2018, 3:07 PM

Change 421917 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] Revert "hieradata: use puppetmaster2001 as ca_server"

https://gerrit.wikimedia.org/r/421917

gerritbot added a comment.Mar 27 2018, 11:13 AM

Change 421842 merged by Filippo Giunchedi:
[operations/puppet@production] puppetmaster: install keypair for 'puppet' when running as CA

https://gerrit.wikimedia.org/r/421842

Stashbot added a comment.Mar 27 2018, 3:10 PM

Mentioned in SAL (#wikimedia-operations) [2018-03-27T15:10:12Z] <godog> stop puppet fleetwide for CA failover - T189891

gerritbot added a comment.Mar 27 2018, 3:15 PM

Change 421917 merged by Filippo Giunchedi:
[operations/puppet@production] Revert "hieradata: use puppetmaster2001 as ca_server"

https://gerrit.wikimedia.org/r/421917

Stashbot added a comment.Mar 27 2018, 3:23 PM

Mentioned in SAL (#wikimedia-operations) [2018-03-27T15:23:29Z] <godog> reenable puppet fleetwide for CA failover - T189891

fgiunchedi closed this task as Resolved.Mar 28 2018, 2:17 PM
fgiunchedi claimed this task.

This is complete. Added documentation to https://wikitech.wikimedia.org/wiki/Puppet#Puppet_CA

gerritbot added a comment.Mar 20 2020, 9:26 AM

Change 420705 abandoned by Filippo Giunchedi:
puppetmaster: lock commits on /srv/private on non-master hosts

Reason:
puppet-merge now locks

https://gerrit.wikimedia.org/r/420705

Maintenance_bot removed a project: Patch-For-Review.Mar 20 2020, 10:10 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits