This task tracks the preparation of our base system services for Debian 10/buster.
Description
Details
Status | Subtype | Assigned | Task | ||
---|---|---|---|---|---|
Resolved | MoritzMuehlenhoff | T213527 Prepare our base system layer for Debian buster | |||
Resolved | Marostegui | T193224 Evaluate and decide the future of relational datastore at WMF after the upgrade of MariaDB 10.1 is finished | |||
Resolved | • jcrespo | T193226 Test MySQL 8.0 with production data and evaluate its fit for WMF databases | |||
Resolved | Marostegui | T243081 Make Percona 8 report the basics on tendril | |||
Resolved | • jcrespo | T243265 Check why compare.py doesn't work with Percona 8.0 | |||
Resolved | jbond | T213546 Prepare puppet infrastructure for Debian buster | |||
Resolved | fgiunchedi | T218118 prometheus-node-exporter makes puppet fail because requiring a version that no longer exists on buster |
Event Timeline
Change 483694 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Add support for buster-wikimedia to our internal repository
Change 483694 merged by Muehlenhoff:
[operations/puppet@production] Add support for buster-wikimedia to our internal repository
Mentioned in SAL (#wikimedia-operations) [2019-01-14T15:02:26Z] <moritzm> imported debdeploy 0.0.99.6-1+deb10u1 for buster-wikimedia (T213527)
Mentioned in SAL (#wikimedia-operations) [2019-01-18T12:50:14Z] <moritzm> uploaded ferm 2.4-1+wmf1 to buster-wikimedia (T213527)
Change 489171 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Extend d-i config for buster
Change 489171 merged by Muehlenhoff:
[operations/puppet@production] Extend d-i config for buster
Mentioned in SAL (#wikimedia-operations) [2019-02-08T15:31:46Z] <moritzm> imported debmonitor 0.1.5-1+deb10u1 to buster-wikimedia (T213527)
Still some rough edges to sort out, but bare metal installations are working now:
$ ssh graphite2002.codfw.wmnet Linux graphite2002 4.19.0-2-amd64 #1 SMP Debian 4.19.16-1 (2019-01-17) x86_64 Debian GNU/Linux buster/sid graphite2002 is a Unused spare system (spare::system) The last Puppet run was at Fri Feb 8 15:45:45 UTC 2019 (2 minutes ago).
Mentioned in SAL (#wikimedia-operations) [2019-02-08T16:08:10Z] <moritzm> imported git-fat 0.1.3-2+deb10u1 to buster-wikimedia (T213527)
Mentioned in SAL (#wikimedia-operations) [2019-02-08T18:14:55Z] <gtirloni> T213527 graphite2002 disabled puppet and commented prometheus_puppet_agent_stats cronjob due to cronspam
Change 490353 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Update timedatectl Icinga check for buster
Change 490353 merged by Muehlenhoff:
[operations/puppet@production] Update timedatectl Icinga check for buster
I've encountered an issue re-imaging cloudstore1008/9 with Buster where the megaraid_sas driver seems to be missing (or is it mpt2sas?), so no disks are detected.
@GTirloni: That's a temporary installer issue, the kernel modules on the last installer images provided use a different kernel ABI than the current kernel in the archive. Once installed the driver is fully functional. That said, the NFS servers are currently not a suitable candidate for early d-i testing I'd say, it seems better to resume with Stretch anyway.
Change 494681 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Stop using transitional package names for Icinga plugins
Installations in Ganeti are currently blocked for a long time waiting for entropy in the d-i step which generates an SSH host key. This is resolved once 4.9.20-1 is migrated to testing; it enables the kernel to gather entropy from the CPU-internal RNG on the virtualisation server (all our Ganeti servers support that and the rdrand instruction is already passed down to Ganeti VMs.
Change 494718 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Install ack instead of ack-grep
Mentioned in SAL (#wikimedia-operations) [2019-03-06T16:05:20Z] <moritzm> imported scap for buster-wikimedia (T213527)
After the buster upgrade, what appears to be the debmonitor hook fails on apt update, upgrade at db1114 with:
su: warning: cannot change directory to /nonexistent: No such file or directory
Not sure if buster or just the partially aborted upgrade (this wasn't a clean reinstall).
@jcrespo that's T216832 and we were thinking to just create a home for the user (cc @MoritzMuehlenhoff )
Change 494718 merged by Muehlenhoff:
[operations/puppet@production] Install ack instead of ack-grep
Mentioned in SAL (#wikimedia-operations) [2019-03-12T16:29:42Z] <moritzm> upgraded buster installation image to daily build from 12th of March (T213527)
Change 498045 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Add curl to standard packages
Change 498045 merged by Muehlenhoff:
[operations/puppet@production] Add curl to standard packages
Change 503027 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Pull in buster udebs from unstable
Change 503027 merged by Muehlenhoff:
[operations/puppet@production] Pull in buster udebs from unstable
Change 494681 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Stop using transitional package names for Icinga plugins
Change 511654 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Add gnupg to d-i base installer for buster
Change 511654 merged by Muehlenhoff:
[operations/puppet@production] Add gnupg to d-i base installer for buster
We have a number of buster hosts running in production and fresh installs of Buster are now working fine, I'm closing this task. All further refinements can happen via separate tasks/patches.
Change 494681 merged by Muehlenhoff:
[operations/puppet@production] Stop using transitional package names for Icinga plugins
Change 517875 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Update check_timedatectl to latest version from DSA repository
Change 517875 merged by Muehlenhoff:
[operations/puppet@production] Update check_timedatectl to latest version from DSA repository