Page MenuHomePhabricator

Upgrade to puppet 4 (4.8 or newer)
Closed, ResolvedPublic

Description

Parent for tasks related to puppet 4 upgrade.

Testing strategy

Upgrade procedure

  • Point a small but representative sampling of prod hosts at codfw (which is now upgraded to puppet 4)
    • change the hiera "puppetmaster" variable to puppetmaster2001.codfw.wmnet
  • Move back dns for codfw, ulsfo to use puppetmaster2001, check puppet logs on all servers in those datacenters.
  • Disable puppet fleet-wide with cumin
  • upgrade puppet on all the puppetmasters in eqiad to puppet 4.8.2 via jessie-backports package
  • perform dry runs
  • perform actual runs on selected hosts
  • Reenable puppet everywhere via cumin
  • Upgrade puppet to 4.8.2 on all jessie/stretch/trusty hosts.

Cleanup

Related Objects

StatusAssignedTask
Resolvedaborrero
ResolvedNone
ResolvedNone
OpenNone
ResolvedJoe
DeclinedNone
ResolvedJoe
ResolvedNone
Resolvedhashar
ResolvedNone
ResolvedNone
ResolvedJoe
ResolvedAndrew
ResolvedAndrew
OpenNone
Resolvedherron
Resolvedherron
Resolvedherron
Resolvedherron
Resolvedherron
OpenNone
Resolvedherron
Resolvedherron
Resolvedherron
Resolvedherron
Resolvedfgiunchedi
Resolvedfgiunchedi
Resolvedherron
Resolvedherron
Resolvedherron
Resolvedherron
Resolvedherron
ResolvedAndrew
ResolvedBstorm
Resolvedhashar
InvalidNone
ResolvedJoe
Resolvedherron
Resolvedherron
StalledNone
DuplicateNone
Resolvedherron
Resolvedherron
Resolvedherron
Resolvedherron
Resolvedherron
Resolvedherron
OpenNone
DuplicateNone
Resolvedherron
DuplicateNone
DuplicateNone
DuplicateNone
DuplicateNone
DuplicateNone
DuplicateNone
DuplicateNone
Resolvedherron
Resolvedherron
Resolvedherron
Resolvedherron
Resolvedherron
Resolvedherron
Resolvedherron
Resolvedherron
Resolvedherron
Resolvedherron
DuplicateNone
ResolvedJoe
Resolvedherron
OpenNone
Resolvedherron
Resolvedherron
Resolvedherron
Resolvedherron

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Change 391067 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] puppet: fix puppetmaster-passenger package in puppetmaster::passenger

https://gerrit.wikimedia.org/r/391067

Change 391067 merged by Herron:
[operations/puppet@production] puppet: fix puppetmaster-passenger package in puppetmaster::passenger

https://gerrit.wikimedia.org/r/391067

Change 391076 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] puppet: puppetmaster remove puppetmaster-common package ensure

https://gerrit.wikimedia.org/r/391076

Change 391076 merged by Herron:
[operations/puppet@production] puppet: puppetmaster remove puppetmaster-common package ensure

https://gerrit.wikimedia.org/r/391076

Change 391627 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] puppet: point codfw mediawiki::appservers at puppetmaster2001

https://gerrit.wikimedia.org/r/391627

Change 391646 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] puppet: point codfw mediawiki canary appservers at puppetmaster2001

https://gerrit.wikimedia.org/r/391646

Change 391646 merged by Herron:
[operations/puppet@production] puppet: point codfw mediawiki canary appservers at puppetmaster2001

https://gerrit.wikimedia.org/r/391646

Change 391627 abandoned by Herron:
puppet: point codfw mediawiki::appservers at puppetmaster2001

https://gerrit.wikimedia.org/r/391627

Change 391856 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] puppet: point codfw mw systems at puppet 4 master puppetmaster2001

https://gerrit.wikimedia.org/r/391856

Change 391856 merged by Herron:
[operations/puppet@production] puppet: point codfw mw systems at puppet 4 master puppetmaster2001

https://gerrit.wikimedia.org/r/391856

Change 392047 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/puppet@production] Revert "puppet: point codfw mw systems at puppet 4 master puppetmaster2001"

https://gerrit.wikimedia.org/r/392047

Change 392047 merged by Alexandros Kosiaris:
[operations/puppet@production] Revert "puppet: point codfw mw systems at puppet 4 master puppetmaster2001"

https://gerrit.wikimedia.org/r/392047

Change 392406 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/puppet@production] servermon.rb: Parse puppet.conf

https://gerrit.wikimedia.org/r/392406

Change 392406 merged by Alexandros Kosiaris:
[operations/puppet@production] servermon.rb: Parse puppet.conf

https://gerrit.wikimedia.org/r/392406

Change 392416 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/puppet@production] Revert "Revert "puppet: point codfw mw systems at puppet 4 master puppetmaster2001""

https://gerrit.wikimedia.org/r/392416

Change 392416 merged by Herron:
[operations/puppet@production] puppet: point codfw mw systems at puppet 4 master puppetmaster2001

https://gerrit.wikimedia.org/r/392416

Change 392664 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] puppet: point db2* hosts at puppet 4 master puppetmaster2001

https://gerrit.wikimedia.org/r/392664

Change 392676 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] puppet: point codfw cp servers at codfw puppet 4 masters

https://gerrit.wikimedia.org/r/392676

Change 392691 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] puppet: point codfw elasticsearch servers at codfw puppet 4 masters

https://gerrit.wikimedia.org/r/392691

Change 392664 merged by Herron:
[operations/puppet@production] puppet: point db2* hosts at puppet 4 master puppetmaster2001

https://gerrit.wikimedia.org/r/392664

Change 392863 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] puppet: point codfw scb hosts at codfw puppet 4 masters

https://gerrit.wikimedia.org/r/392863

Change 392863 merged by Herron:
[operations/puppet@production] puppet: point codfw scb hosts at codfw puppet 4 masters

https://gerrit.wikimedia.org/r/392863

Change 393602 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] puppet: point codfw prometheus servers at codfw puppetmasters

https://gerrit.wikimedia.org/r/393602

Change 392691 merged by Herron:
[operations/puppet@production] puppet: point codfw elasticsearch servers at codfw puppet 4 masters

https://gerrit.wikimedia.org/r/392691

Change 393602 merged by Herron:
[operations/puppet@production] puppet: point codfw prometheus servers at codfw puppet 4 masters

https://gerrit.wikimedia.org/r/393602

Change 393670 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] puppet: point codfw lvs servers at codfw puppet 4 masters

https://gerrit.wikimedia.org/r/393670

Change 392676 merged by Herron:
[operations/puppet@production] puppet: point codfw misc and canary cp hosts at codfw puppet4 masters

https://gerrit.wikimedia.org/r/392676

Change 394084 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] puppet: point codfw cp servers at codfw puppet 4 masters

https://gerrit.wikimedia.org/r/394084

Change 394085 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] puppet: initial cut over of codfw cp text/upload to puppet 4 masters

https://gerrit.wikimedia.org/r/394085

Change 394085 merged by Herron:
[operations/puppet@production] puppet: initial cut over of codfw cp text/upload to puppet 4 masters

https://gerrit.wikimedia.org/r/394085

Change 394084 merged by Herron:
[operations/puppet@production] puppet: point codfw cp servers at codfw puppet 4 masters

https://gerrit.wikimedia.org/r/394084

Change 393670 merged by Herron:
[operations/puppet@production] puppet: point codfw lvs servers at codfw puppet 4 masters

https://gerrit.wikimedia.org/r/393670

Change 394110 had a related patch set uploaded (by Herron; owner: Herron):
[operations/dns@master] dns: restore puppet.codfw.wmnet CNAME puppetmaster2001.eqiad.wmnet

https://gerrit.wikimedia.org/r/394110

Change 394119 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] puppet: cut over eqiad scb hosts to codfw puppet 4 masters

https://gerrit.wikimedia.org/r/394119

Change 394119 merged by Herron:
[operations/puppet@production] puppet: cut over eqiad scb hosts to codfw puppet 4 masters

https://gerrit.wikimedia.org/r/394119

Change 394110 merged by Herron:
[operations/dns@master] dns: restore puppet.codfw.wmnet CNAME puppetmaster2001.codfw.wmnet

https://gerrit.wikimedia.org/r/394110

Change 394301 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] puppet: fix cqlsh.erb template to parse under puppet 4

https://gerrit.wikimedia.org/r/394301

Change 394301 merged by Herron:
[operations/puppet@production] puppet: fix cqlsh.erb template to parse under puppet 4

https://gerrit.wikimedia.org/r/394301

Change 394307 had a related patch set uploaded (by Herron; owner: Herron):
[operations/dns@master] dns: set puppet.ulsfo.wmnet CNAME puppetmaster2001.codfw.wmnet

https://gerrit.wikimedia.org/r/394307

Change 394308 had a related patch set uploaded (by Herron; owner: Herron):
[operations/dns@master] dns: set puppet.esams.wmnet CNAME puppetmaster2001.codfw.wmnet

https://gerrit.wikimedia.org/r/394308

Change 394307 merged by Herron:
[operations/dns@master] dns: set puppet.ulsfo.wmnet CNAME puppetmaster2001.codfw.wmnet

https://gerrit.wikimedia.org/r/394307

Change 394308 merged by Herron:
[operations/dns@master] dns: set puppet.esams.wmnet CNAME puppetmaster2001.codfw.wmnet

https://gerrit.wikimedia.org/r/394308

Change 394556 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/puppet@production] puppet-compiler: switch to puppet 4

https://gerrit.wikimedia.org/r/394556

Change 394556 merged by Giuseppe Lavagetto:
[operations/puppet@production] puppet-compiler: switch to puppet 4

https://gerrit.wikimedia.org/r/394556

Change 394570 had a related patch set uploaded (by Herron; owner: Herron):
[operations/dns@master] puppet: cut over all puppet service records to codfw puppet 4 masters

https://gerrit.wikimedia.org/r/394570

Change 394570 merged by Herron:
[operations/dns@master] puppet: cut over all puppet service records to codfw puppet 4 masters

https://gerrit.wikimedia.org/r/394570

Change 394577 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] puppet: fix rsyncd.conf.media template to parse under puppet 4

https://gerrit.wikimedia.org/r/394577

Change 394577 merged by Herron:
[operations/puppet@production] puppet: fix rsyncd.conf.media template to parse under puppet 4

https://gerrit.wikimedia.org/r/394577

Change 394613 had a related patch set uploaded (by Herron; owner: Herron):
[operations/dns@master] puppet: revert puppet agents back to eqiad puppet 3 masters

https://gerrit.wikimedia.org/r/394613

Change 394613 merged by Herron:
[operations/dns@master] puppet: revert puppet agents back to eqiad puppet 3 masters

https://gerrit.wikimedia.org/r/394613

Change 395003 had a related patch set uploaded (by Herron; owner: Herron):
[operations/dns@master] puppet: cut over all puppet service records to codfw puppet 4 masters

https://gerrit.wikimedia.org/r/395003

Change 395003 merged by Herron:
[operations/dns@master] puppet: cut over all puppet service records to codfw puppet 4 masters

https://gerrit.wikimedia.org/r/395003

Change 395028 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] puppet: change codfw puppet masters to use eqiad puppetdb server

https://gerrit.wikimedia.org/r/395028

Change 395791 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] puppet: set rhodium puppet_major_version to 4

https://gerrit.wikimedia.org/r/395791

Change 395791 merged by Herron:
[operations/puppet@production] puppet: set rhodium puppet_major_version to 4

https://gerrit.wikimedia.org/r/395791

Change 395832 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] puppetmaster: add proxypassmatch rules for puppet 4 url variants

https://gerrit.wikimedia.org/r/395832

Change 396036 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] puppetmaster: change puppetmaster1002 puppet major version to 4

https://gerrit.wikimedia.org/r/396036

Change 396036 merged by Herron:
[operations/puppet@production] puppetmaster: change puppetmaster1002 puppet major version to 4

https://gerrit.wikimedia.org/r/396036

Change 395832 merged by Herron:
[operations/puppet@production] puppetmaster: add proxypassmatch rules for puppet 4 url variants

https://gerrit.wikimedia.org/r/395832

Change 396058 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] puppetmaster: change puppetmaster1001 puppet major version to 4

https://gerrit.wikimedia.org/r/396058

Change 396058 merged by Herron:
[operations/puppet@production] puppetmaster: change puppetmaster1001 puppet major version to 4

https://gerrit.wikimedia.org/r/396058

Change 396070 had a related patch set uploaded (by Herron; owner: Herron):
[operations/dns@master] puppet: re-pool puppetmaster1001 as puppet.ulsfo.wmnet

https://gerrit.wikimedia.org/r/396070

Change 396070 merged by Herron:
[operations/dns@master] puppet: re-pool puppetmaster1001 as puppet.ulsfo.wmnet

https://gerrit.wikimedia.org/r/396070

Change 396078 had a related patch set uploaded (by Herron; owner: Herron):
[operations/dns@master] re-pool puppetmaster1001 as puppet.(eqiad.wmnet|wikimedia.org)

https://gerrit.wikimedia.org/r/396078

Change 396079 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] puppet: remove (cleanup) hiera regex used for puppet 4 validation

https://gerrit.wikimedia.org/r/396079

Change 396078 merged by Herron:
[operations/dns@master] re-pool puppetmaster1001 as puppet.(eqiad.wmnet|wikimedia.org)

https://gerrit.wikimedia.org/r/396078

herron updated the task description. (Show Details)Dec 7 2017, 8:23 PM

Change 396079 merged by Herron:
[operations/puppet@production] puppet: remove (cleanup) hiera regex used for puppet 4 validation

https://gerrit.wikimedia.org/r/396079

herron updated the task description. (Show Details)Dec 7 2017, 9:22 PM

Change 396132 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] puppetdb: remove (cleanup) ferm allow for puppetcompiler1001

https://gerrit.wikimedia.org/r/396132

Change 396132 merged by Herron:
[operations/puppet@production] puppetdb: remove (cleanup) ferm allow for puppetcompiler1001

https://gerrit.wikimedia.org/r/396132

Change 396423 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] puppet: set puppetcompiler1001 puppet major version 4

https://gerrit.wikimedia.org/r/396423

Change 396423 merged by Herron:
[operations/puppet@production] puppet: set puppetcompiler1001 puppet major version 4

https://gerrit.wikimedia.org/r/396423

Change 396438 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] puppet: fix puppet/facter package pinning in stretch

https://gerrit.wikimedia.org/r/396438

Change 396438 merged by Herron:
[operations/puppet@production] puppet: fix puppet/facter package pinning in stretch

https://gerrit.wikimedia.org/r/396438

Change 397553 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] puppet: change puppet major version to 4 in codfw

https://gerrit.wikimedia.org/r/397553

Change 397553 merged by Herron:
[operations/puppet@production] puppet: change puppet major version to 4 in codfw

https://gerrit.wikimedia.org/r/397553

Change 397624 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] puppet: change location of environment setting from [main] to [agent]

https://gerrit.wikimedia.org/r/397624

Change 397624 merged by Volans:
[operations/puppet@production] puppet: change location of environment setting from [main] to [agent]

https://gerrit.wikimedia.org/r/397624

Change 404517 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] remove ganeti VM puppetcompiler1001.eqiad.wmnet

https://gerrit.wikimedia.org/r/404517

Change 404518 had a related patch set uploaded (by Herron; owner: Herron):
[operations/dns@master] remove dns records for retired ganeti VM puppetcompiler1001

https://gerrit.wikimedia.org/r/404518

Change 404517 merged by Herron:
[operations/puppet@production] remove ganeti VM puppetcompiler1001.eqiad.wmnet

https://gerrit.wikimedia.org/r/404517

Change 404518 merged by Herron:
[operations/dns@master] remove dns records for retired ganeti VM puppetcompiler1001

https://gerrit.wikimedia.org/r/404518

herron closed this task as Resolved.Jan 16 2018, 6:52 PM
herron updated the task description. (Show Details)
herron removed a project: Patch-For-Review.

Change 395028 abandoned by Herron:
puppet: change codfw puppet masters to use eqiad puppetdb server

Reason:
not needed

https://gerrit.wikimedia.org/r/395028

Change 385976 abandoned by Herron:
puppet: depool (via firewall) codfw puppetmaster for upgrade

Reason:
not needed

https://gerrit.wikimedia.org/r/385976