Page MenuHomePhabricator

Upgrade wmcs instances and masters to puppet 4.8
Closed, ResolvedPublic

Description

I'm pretty well convinced that there are no significant differences in catalogs, so getting clients to work properly should be fine.

Since clients aren't backwards-compatible and our puppetmasters are also clients, this needs to proceed top to bottom:

  • Prod puppet masters

then

  • labpuppetmaster1001/1002

then

  • in-project puppetmasters (e.g. tools-puppetmaster-01.tools.eqiad.wmflabs)

then

  • instances

Related Objects

StatusAssignedTask
Resolvedaborrero
ResolvedNone
ResolvedNone
OpenNone
ResolvedJoe
DeclinedNone
ResolvedJoe
ResolvedNone
Resolvedhashar
ResolvedNone
ResolvedNone
ResolvedJoe
ResolvedAndrew
ResolvedAndrew
OpenNone
Resolvedherron
Resolvedherron
Resolvedherron
Resolvedherron
Resolvedherron
OpenNone
Resolvedherron
Resolvedherron
Resolvedherron
Resolvedherron
Resolvedfgiunchedi
Resolvedfgiunchedi
Resolvedherron
Resolvedherron
Resolvedherron
Resolvedherron
Resolvedherron
ResolvedAndrew
ResolvedBstorm
Resolvedhashar
InvalidNone
ResolvedJoe
Resolvedherron
Resolvedherron
StalledNone
DuplicateNone
Resolvedherron
Resolvedherron
Resolvedherron
Resolvedherron
Resolvedherron
Resolvedherron
OpenNone
DuplicateNone
Resolvedherron
DuplicateNone
DuplicateNone
DuplicateNone
DuplicateNone
DuplicateNone
DuplicateNone
DuplicateNone
Resolvedherron
Resolvedherron
Resolvedherron
Resolvedherron
Resolvedherron
Resolvedherron
Resolvedherron
Resolvedherron
Resolvedherron
Resolvedherron
DuplicateNone
ResolvedJoe
Resolvedherron
OpenNone
Resolvedherron
Resolvedherron
Resolvedherron
Resolvedherron

Event Timeline

Andrew created this task.Oct 20 2017, 7:12 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 20 2017, 7:12 PM

if you want you can use puppet-paladox3.git.eqiad.wmflabs.org as a test for puppet 4.8 :). Though i just need to know when you would do it :).

bd808 triaged this task as Normal priority.Oct 31 2017, 11:39 PM
Andrew claimed this task.Nov 30 2017, 5:53 PM

Change 394353 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Move labtestpuppetmaster2001 to puppet 4

https://gerrit.wikimedia.org/r/394353

Change 394353 merged by Andrew Bogott:
[operations/puppet@production] Move labtestpuppetmaster2001 to puppet 4

https://gerrit.wikimedia.org/r/394353

Change 397575 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] puppetmaster: include @extra_auth_rules in v4 auth.conf

https://gerrit.wikimedia.org/r/397575

Change 397575 merged by Andrew Bogott:
[operations/puppet@production] puppetmaster: include @extra_auth_rules in v4 auth.conf

https://gerrit.wikimedia.org/r/397575

Change 397579 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] WMCS puppetmaster: modify auth.conf extra rules to account for url changes

https://gerrit.wikimedia.org/r/397579

Change 397579 merged by Andrew Bogott:
[operations/puppet@production] WMCS puppetmaster: modify auth.conf extra rules to account for url changes

https://gerrit.wikimedia.org/r/397579

Change 397584 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] profile::puppetmaster::backend: fix extra_auth_rules for backend masters

https://gerrit.wikimedia.org/r/397584

Change 397584 merged by Andrew Bogott:
[operations/puppet@production] profile::puppetmaster::backend: fix extra_auth_rules for backend masters

https://gerrit.wikimedia.org/r/397584

Andrew updated the task description. (Show Details)Dec 12 2017, 1:53 PM

Change 397711 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] WMCS: set puppet_major_version to 4

https://gerrit.wikimedia.org/r/397711

Change 397711 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] WMCS: set puppet_major_version to 4

https://gerrit.wikimedia.org/r/397711

Change 397821 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] labs-bootstrapvz-jessie: include puppet pinning in base images

https://gerrit.wikimedia.org/r/397821

Change 397821 merged by Andrew Bogott:
[operations/puppet@production] labs-bootstrapvz-jessie: include puppet pinning in base images

https://gerrit.wikimedia.org/r/397821

Change 397850 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] bootstrap-vz: include debian backports in sources.list

https://gerrit.wikimedia.org/r/397850

Change 397850 merged by Andrew Bogott:
[operations/puppet@production] bootstrap-vz: include debian backports in sources.list

https://gerrit.wikimedia.org/r/397850

Change 397966 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] bootstrapvz: allow default (v4) puppet packages on stretch base images.

https://gerrit.wikimedia.org/r/397966

Change 397966 merged by Andrew Bogott:
[operations/puppet@production] bootstrapvz: allow default (v4) puppet packages on stretch base images.

https://gerrit.wikimedia.org/r/397966

Change 397711 merged by Andrew Bogott:
[operations/puppet@production] WMCS: set puppet_major_version to 4

https://gerrit.wikimedia.org/r/397711

Andrew added a subscriber: herron.Dec 13 2017, 5:59 PM

All masters are upgraded, and Debian base images now include puppet 4.

All that remains is upgrading agents, like so:

$ sudo cumin "*" "disable-puppet 'puppet upgrades - andrew'"
$ sudo cumin "*" "apt-get update"
sudo cumin "*" 'DEBIAN_FRONTEND=noninteractive apt-get -q -y --assume-no -o DPkg::Options::="--force-confdef" -o DPkg::Options::="--force-confold" install puppet'
$ sudo cumin "*" "enable-puppet 'puppet upgrades - andrew'"

I'm not doing that right now because currently there are no puppet 4 candidate packages for trusty. @herron is investigating that now; soon we'll decide whether to backport a package or just leave Trusty with v3 agents.

This is done everywhere except for Trusty VMs in Tools. @aborrero is on top of that.

Andrew reassigned this task from Andrew to aborrero.Jan 17 2018, 4:00 PM

I'm trying to accomplish this following the workflow we are developing in T181647

The upgrade script in that workflow crash if new package are being installed due to dependencies:

aborrero@tools-webgrid-generic-1402:~$ sudo apt-upgrade trusty-wikimedia -sv
Updating cache ...
puppet-common 3.8.5-2~bpo8trusty+2 --> 4.8.2-5~trusty1
Traceback (most recent call last):
  File "/usr/local/sbin/apt-upgrade", line 127, in <module>
    main()
  File "/usr/local/sbin/apt-upgrade", line 123, in main
    run(args.source, args.s, args.v)
  File "/usr/local/sbin/apt-upgrade", line 101, in run
    print_verbose_pkg(verbose, pkg)
  File "/usr/local/sbin/apt-upgrade", line 34, in print_verbose_pkg
    orig = pkg.installed.version
AttributeError: 'NoneType' object has no attribute 'version'

It should instead show something like this:

aborrero@tools-webgrid-generic-1402:~$ sudo apt-upgrade trusty-wikimedia -sv
Updating cache ...
puppet-common 3.8.5-2~bpo8trusty+2 --> 4.8.2-5~trusty1
python-ecdsa unknown --> 0.10-2
puppet 3.8.5-2~bpo8trusty+2 --> 4.8.2-5~trusty1
ruby-deep-merge unknown --> 1.1.1-1~trusty1
python-paramiko 1.10.1-1git1build1 --> 1.15.2-1
Simulate, not performing changes

I'm patching this first before going on.

BTW my notes are here: https://etherpad.wikimedia.org/p/arturo-toolforge-upgrades

Change 404963 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] apt: apt-upgrades: dont fail if new packages are being installed

https://gerrit.wikimedia.org/r/404963

Change 404963 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] apt: apt-upgrades: dont fail if new packages are being installed

https://gerrit.wikimedia.org/r/404963

Mentioned in SAL (#wikimedia-cloud) [2018-01-18T11:42:00Z] <arturo> T178717 aborrero@tools-clushmaster-01:~$ clush -w @all 'sudo puppet agent --test'

Mentioned in SAL (#wikimedia-cloud) [2018-01-18T12:11:06Z] <arturo> T178717 aborrero@tools-webgrid-generic-1402:~$ sudo apt-upgrade trusty-wikimedia -v

Mentioned in SAL (#wikimedia-cloud) [2018-01-18T12:24:17Z] <arturo> T178717 aborrero@tools-exec-1401:~$ sudo apt-upgrade trusty-wikimedia -v

Mentioned in SAL (#wikimedia-cloud) [2018-01-18T13:52:31Z] <arturo> T178717 aborrero@tools-clushmaster-01:~$ clush -f 1 -w @all 'sudo facter | grep lsbdistcodename | grep trusty && sudo apt-upgrade trusty-wikimedia -v'

Mentioned in SAL (#wikimedia-cloud) [2018-01-18T15:42:32Z] <arturo> T178717 aborrero@tools-clushmaster-01:~$ clush -w @all 'sudo puppet agent -t -v'

aborrero closed this task as Resolved.Jan 18 2018, 4:29 PM

All instances should now be running puppet 4.

aborrero updated the task description. (Show Details)Jan 18 2018, 4:29 PM