Page MenuHomePhabricator

Upgrade Puppet Masters and Puppet DB servers
Closed, ResolvedPublic

Description

Main tracking for Q1 FY19-20 Goals

Puppet 5

  • Upgrade Puppetmasters to Puppet 5
    • Rebuild rhodium on buster using puppet5 packages and work out any bugs
    • Rebuild other puppetmasters one by one
  • Upgrade PuppetDB
    • Upgrade PuppetDB in a virtual environment to work out any unforeseen bugs
    • Upgrade production PuppetDB in both data centers

Details

Related Gerrit Patches:
operations/puppet : productionCI rspec: update puppet version used in spec tests
operations/puppet : productionBump CI puppet Gem version to 5.5.10
operations/puppet : productionRemove late-install hack for puppet 4 installation
operations/puppet : productionpuppetmaster1003: move ms-be1016, ms-fe1005 & thumbor1001 to new pmaster
operations/puppet : productionpuppetmaster1003: move rdb1006, restbase1016 & scb1001 to pmaster
operations/puppet : productionpuppetmaster1003: move dubnium, ores1001 & wtp1025 to new pmaster
operations/puppet : productionpuppetmaster1003: move mw appserver, api and lvs server to new pmaster
operations/puppet : productionpuppetmaster1003: move cp1075-77 to new pmaster
operations/puppet : productionpuppetmaster1003: move mw1261 and mwdebug1001 to new pmaster
operations/puppet : productionpuppetmaster::frontend: update web conf to use RewriteRules instead of proxypass
operations/puppet : productionpuppetmaster1003: add canary hosts back and remove debug logging
operations/puppet : productionpuppetmaster1003: offline this puppetmaster as its scheme is incompatible
operations/puppet : productionpuppetmaster: add canary hosts to puppetmaster
operations/puppet : productionpuppetmaster: add canary hosts to puppetmaster
operations/puppet : productionpuppetmaster: add canary hosts to puppetmaster
operations/puppet : productionpuppetmaster: add canary hosts to puppetmaster
operations/puppet : productionpuppetmaster: add canary hosts to puppetmaster
operations/puppet : productionpuppetmaster: add canary hosts to puppetmaster
operations/puppet : productionpuppetmaster: add canary hosts to puppetmaster
operations/puppet : productionpuppetmaster: Fix regression
operations/puppet : productionpuppetmaster: add canary hosts to puppetmaster
operations/puppet : productionpuppetmaster: add canary hosts to puppetmaster

Event Timeline

jbond created this task.Jul 22 2019, 1:05 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 22 2019, 1:05 PM

Change 526104 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/puppet@production] Bump CI puppet Gem version to 5.5.10

https://gerrit.wikimedia.org/r/526104

Change 527056 had a related patch set uploaded (by Jbond; owner: John Bond):
[operations/puppet@production] puppetmaster: add canary hosts to puppetmaster

https://gerrit.wikimedia.org/r/527056

Change 527056 merged by Jbond:
[operations/puppet@production] puppetmaster: add canary hosts to puppetmaster

https://gerrit.wikimedia.org/r/527056

Change 527064 had a related patch set uploaded (by Jbond; owner: John Bond):
[operations/puppet@production] puppetmaster: add canary hosts to puppetmaster

https://gerrit.wikimedia.org/r/527064

Change 527064 merged by Jbond:
[operations/puppet@production] puppetmaster: add canary hosts to puppetmaster

https://gerrit.wikimedia.org/r/527064

Change 527078 had a related patch set uploaded (by Jbond; owner: John Bond):
[operations/puppet@production] puppetmaster: Fix regression

https://gerrit.wikimedia.org/r/527078

Change 527078 merged by Jbond:
[operations/puppet@production] puppetmaster: Fix regression

https://gerrit.wikimedia.org/r/527078

Change 527086 had a related patch set uploaded (by Jbond; owner: John Bond):
[operations/puppet@production] puppetmaster: add canary hosts to puppetmaster

https://gerrit.wikimedia.org/r/527086

Change 527086 merged by Jbond:
[operations/puppet@production] puppetmaster: add canary hosts to puppetmaster

https://gerrit.wikimedia.org/r/527086

Change 527092 had a related patch set uploaded (by Jbond; owner: John Bond):
[operations/puppet@production] puppetmaster: add canary hosts to puppetmaster

https://gerrit.wikimedia.org/r/527092

Change 527092 merged by Jbond:
[operations/puppet@production] puppetmaster: add canary hosts to puppetmaster

https://gerrit.wikimedia.org/r/527092

Change 527095 had a related patch set uploaded (by Jbond; owner: John Bond):
[operations/puppet@production] puppetmaster: add canary hosts to puppetmaster

https://gerrit.wikimedia.org/r/527095

Change 527095 merged by Jbond:
[operations/puppet@production] puppetmaster: add canary hosts to puppetmaster

https://gerrit.wikimedia.org/r/527095

Change 527100 had a related patch set uploaded (by Jbond; owner: John Bond):
[operations/puppet@production] puppetmaster: add canary hosts to puppetmaster

https://gerrit.wikimedia.org/r/527100

Change 527100 merged by Jbond:
[operations/puppet@production] puppetmaster: add canary hosts to puppetmaster

https://gerrit.wikimedia.org/r/527100

Change 527109 had a related patch set uploaded (by Jbond; owner: John Bond):
[operations/puppet@production] puppetmaster: add canary hosts to puppetmaster

https://gerrit.wikimedia.org/r/527109

Change 527109 merged by Jbond:
[operations/puppet@production] puppetmaster: add canary hosts to puppetmaster

https://gerrit.wikimedia.org/r/527109

Change 527129 had a related patch set uploaded (by Jbond; owner: John Bond):
[operations/puppet@production] puppetmaster: add canary hosts to puppetmaster

https://gerrit.wikimedia.org/r/527129

Change 527129 merged by Jbond:
[operations/puppet@production] puppetmaster: add canary hosts to puppetmaster

https://gerrit.wikimedia.org/r/527129

Change 527135 had a related patch set uploaded (by Jbond; owner: John Bond):
[operations/puppet@production] puppetmaster: add canary hosts to puppetmaster

https://gerrit.wikimedia.org/r/527135

Change 527135 merged by Jbond:
[operations/puppet@production] puppetmaster: add canary hosts to puppetmaster

https://gerrit.wikimedia.org/r/527135

Change 528521 had a related patch set uploaded (by Jbond; owner: John Bond):
[operations/puppet@production] puppetmaster::frontend: update web conf to use RewriteRules instead of proxypass

https://gerrit.wikimedia.org/r/528521

jbond triaged this task as Medium priority.Aug 7 2019, 10:23 AM

Change 528744 had a related patch set uploaded (by Jbond; owner: John Bond):
[operations/puppet@production] puppetmaster1003: offline this puppetmaster as its scheme is incompatible

https://gerrit.wikimedia.org/r/528744

Change 528744 merged by Jbond:
[operations/puppet@production] puppetmaster1003: offline this puppetmaster as its scheme is incompatible

https://gerrit.wikimedia.org/r/528744

Change 529095 had a related patch set uploaded (by Jbond; owner: John Bond):
[operations/puppet@production] puppetmaster1003: add canary hosts back and remove debug logging

https://gerrit.wikimedia.org/r/529095

Change 529095 merged by Jbond:
[operations/puppet@production] puppetmaster1003: add canary hosts back and remove debug logging

https://gerrit.wikimedia.org/r/529095

Change 528521 merged by Jbond:
[operations/puppet@production] puppetmaster::frontend: update web conf to use RewriteRules instead of proxypass

https://gerrit.wikimedia.org/r/528521

Change 535564 had a related patch set uploaded (by Jbond; owner: John Bond):
[operations/puppet@production] puppetmaster1003: move mw1261 and mwdebug1001 to new pmaster

https://gerrit.wikimedia.org/r/535564

Change 535564 merged by Jbond:
[operations/puppet@production] puppetmaster1003: move mw1261 and mwdebug1001 to new pmaster

https://gerrit.wikimedia.org/r/535564

Change 535570 merged by Jbond:
[operations/puppet@production] puppetmaster1003: move cp1075-77 to new pmaster

https://gerrit.wikimedia.org/r/535570

Change 535609 had a related patch set uploaded (by Jbond; owner: John Bond):
[operations/puppet@production] puppetmaster1003: move mw appserver, api and lvs server to new pmaster

https://gerrit.wikimedia.org/r/535609

Change 535609 merged by Jbond:
[operations/puppet@production] puppetmaster1003: move mw appserver, api and lvs server to new pmaster

https://gerrit.wikimedia.org/r/535609

Change 535856 had a related patch set uploaded (by Jbond; owner: John Bond):
[operations/puppet@production] puppetmaster1003: move dubnium, ores1001 & wtp1025 to new pmaster

https://gerrit.wikimedia.org/r/535856

Change 535856 merged by Jbond:
[operations/puppet@production] puppetmaster1003: move dubnium, ores1001 & wtp1025 to new pmaster

https://gerrit.wikimedia.org/r/535856

Change 535862 had a related patch set uploaded (by Jbond; owner: John Bond):
[operations/puppet@production] puppetmaster1003: move rdb1006, restbase1016 & scb1001 to pmaster

https://gerrit.wikimedia.org/r/535862

Change 535862 merged by Jbond:
[operations/puppet@production] puppetmaster1003: move rdb1006, restbase1016 & scb1001 to pmaster

https://gerrit.wikimedia.org/r/535862

Change 535873 had a related patch set uploaded (by Jbond; owner: John Bond):
[operations/puppet@production] puppetmaster1003: move ms-be1016, ms-fe1005 & thumbor1001 to new pmaster

https://gerrit.wikimedia.org/r/535873

Change 535873 merged by Jbond:
[operations/puppet@production] puppetmaster1003: move ms-be1016, ms-fe1005 & thumbor1001 to new pmaster

https://gerrit.wikimedia.org/r/535873

Change 543816 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Remove late-install hack for puppet 4 installation

https://gerrit.wikimedia.org/r/543816

Change 543816 merged by Muehlenhoff:
[operations/puppet@production] Remove late-install hack for puppet 4 installation

https://gerrit.wikimedia.org/r/543816

Change 545289 had a related patch set uploaded (by Jbond; owner: John Bond):
[operations/puppet@production] CI rspec: update puppet version used in spec tests

https://gerrit.wikimedia.org/r/545289

jbond moved this task from Unsorted 💣 to Active 🚁 on the User-jbond board.Oct 30 2019, 4:53 PM

Change 545289 merged by Jbond:
[operations/puppet@production] CI rspec: update puppet version used in spec tests

https://gerrit.wikimedia.org/r/545289

cookbooks.sre.hosts.decommission executed by jmm@cumin2001 for hosts: puppetdb2001.codfw.wmnet

  • puppetdb2001.codfw.wmnet (FAIL)
    • Downtimed host on Icinga
    • No management interface found (likely a VM)
    • Wiped bootloaders
    • Shutdown issued. Verify it manually, verification not yet supported
    • Set Netbox status on VM not yet supported: manual intervention required
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB

ERROR: some step on some host failed, check the bolded items above

Mentioned in SAL (#wikimedia-operations) [2019-12-12T15:13:00Z] <moritzm> deleting puppetdb1001 in Ganeti T228657

cookbooks.sre.hosts.decommission executed by jmm@cumin1001 for hosts: puppetdb1001.eqiad.wmnet

  • puppetdb1001.eqiad.wmnet (FAIL)
    • Downtimed host on Icinga
    • No management interface found (likely a VM)
    • Unable to connect to the host, wipe of bootloaders will not be performed: Cumin execution failed (exit_code=2)
    • Failed to shutdown, manual intervention required: Cumin execution failed (exit_code=2)
    • Set Netbox status on VM not yet supported: manual intervention required
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB

ERROR: some step on some host failed, check the bolded items above

jbond closed this task as Resolved.Dec 12 2019, 4:54 PM

This is now complete all ops productions servers are on puppet 5