Page MenuHomePhabricator

Prepare our base system layer for Debian buster
Closed, ResolvedPublic

Description

This task tracks the preparation of our base system services for Debian 10/buster.

Event Timeline

Change 483694 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Add support for buster-wikimedia to our internal repository

https://gerrit.wikimedia.org/r/483694

herron triaged this task as Medium priority.Jan 11 2019, 7:47 PM

Change 483694 merged by Muehlenhoff:
[operations/puppet@production] Add support for buster-wikimedia to our internal repository

https://gerrit.wikimedia.org/r/483694

Mentioned in SAL (#wikimedia-operations) [2019-01-14T15:02:26Z] <moritzm> imported debdeploy 0.0.99.6-1+deb10u1 for buster-wikimedia (T213527)

Mentioned in SAL (#wikimedia-operations) [2019-01-18T12:50:14Z] <moritzm> uploaded ferm 2.4-1+wmf1 to buster-wikimedia (T213527)

Change 489171 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Extend d-i config for buster

https://gerrit.wikimedia.org/r/489171

Change 489171 merged by Muehlenhoff:
[operations/puppet@production] Extend d-i config for buster

https://gerrit.wikimedia.org/r/489171

Mentioned in SAL (#wikimedia-operations) [2019-02-08T15:31:46Z] <moritzm> imported debmonitor 0.1.5-1+deb10u1 to buster-wikimedia (T213527)

Still some rough edges to sort out, but bare metal installations are working now:

$ ssh graphite2002.codfw.wmnet
Linux graphite2002 4.19.0-2-amd64 #1 SMP Debian 4.19.16-1 (2019-01-17) x86_64
Debian GNU/Linux buster/sid
graphite2002 is a Unused spare system (spare::system)
The last Puppet run was at Fri Feb  8 15:45:45 UTC 2019 (2 minutes ago).

Mentioned in SAL (#wikimedia-operations) [2019-02-08T16:08:10Z] <moritzm> imported git-fat 0.1.3-2+deb10u1 to buster-wikimedia (T213527)

Mentioned in SAL (#wikimedia-operations) [2019-02-08T18:14:55Z] <gtirloni> T213527 graphite2002 disabled puppet and commented prometheus_puppet_agent_stats cronjob due to cronspam

Change 490353 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Update timedatectl Icinga check for buster

https://gerrit.wikimedia.org/r/490353

Change 490353 merged by Muehlenhoff:
[operations/puppet@production] Update timedatectl Icinga check for buster

https://gerrit.wikimedia.org/r/490353

I've encountered an issue re-imaging cloudstore1008/9 with Buster where the megaraid_sas driver seems to be missing (or is it mpt2sas?), so no disks are detected.

Screenshot from 2019-03-04 04-33-38.png (432×720 px, 43 KB)

@GTirloni: That's a temporary installer issue, the kernel modules on the last installer images provided use a different kernel ABI than the current kernel in the archive. Once installed the driver is fully functional. That said, the NFS servers are currently not a suitable candidate for early d-i testing I'd say, it seems better to resume with Stretch anyway.

Change 494681 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Stop using transitional package names for Icinga plugins

https://gerrit.wikimedia.org/r/494681

Installations in Ganeti are currently blocked for a long time waiting for entropy in the d-i step which generates an SSH host key. This is resolved once 4.9.20-1 is migrated to testing; it enables the kernel to gather entropy from the CPU-internal RNG on the virtualisation server (all our Ganeti servers support that and the rdrand instruction is already passed down to Ganeti VMs.

Change 494718 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Install ack instead of ack-grep

https://gerrit.wikimedia.org/r/494718

Mentioned in SAL (#wikimedia-operations) [2019-03-06T16:05:20Z] <moritzm> imported scap for buster-wikimedia (T213527)

After the buster upgrade, what appears to be the debmonitor hook fails on apt update, upgrade at db1114 with:

su: warning: cannot change directory to /nonexistent: No such file or directory

Not sure if buster or just the partially aborted upgrade (this wasn't a clean reinstall).

@jcrespo that's T216832 and we were thinking to just create a home for the user (cc @MoritzMuehlenhoff )

Change 494718 merged by Muehlenhoff:
[operations/puppet@production] Install ack instead of ack-grep

https://gerrit.wikimedia.org/r/494718

Mentioned in SAL (#wikimedia-operations) [2019-03-12T16:29:42Z] <moritzm> upgraded buster installation image to daily build from 12th of March (T213527)

Change 498045 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Add curl to standard packages

https://gerrit.wikimedia.org/r/498045

Change 498045 merged by Muehlenhoff:
[operations/puppet@production] Add curl to standard packages

https://gerrit.wikimedia.org/r/498045

Change 503027 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Pull in buster udebs from unstable

https://gerrit.wikimedia.org/r/503027

Change 503027 merged by Muehlenhoff:
[operations/puppet@production] Pull in buster udebs from unstable

https://gerrit.wikimedia.org/r/503027

Change 494681 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Stop using transitional package names for Icinga plugins

https://gerrit.wikimedia.org/r/494681

Change 511654 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Add gnupg to d-i base installer for buster

https://gerrit.wikimedia.org/r/511654

Change 511654 merged by Muehlenhoff:
[operations/puppet@production] Add gnupg to d-i base installer for buster

https://gerrit.wikimedia.org/r/511654

MoritzMuehlenhoff claimed this task.

We have a number of buster hosts running in production and fresh installs of Buster are now working fine, I'm closing this task. All further refinements can happen via separate tasks/patches.

Change 494681 merged by Muehlenhoff:
[operations/puppet@production] Stop using transitional package names for Icinga plugins

https://gerrit.wikimedia.org/r/494681

Change 517875 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Update check_timedatectl to latest version from DSA repository

https://gerrit.wikimedia.org/r/517875

Change 517875 merged by Muehlenhoff:
[operations/puppet@production] Update check_timedatectl to latest version from DSA repository

https://gerrit.wikimedia.org/r/517875