This task tracks the preparation of our base system services for Debian 10/buster.
|operations/puppet : production||Update check_timedatectl to latest version from DSA repository|
|operations/puppet : production||Stop using transitional package names for Icinga plugins|
|operations/puppet : production||Add gnupg to d-i base installer for buster|
|operations/puppet : production||Pull in buster udebs from unstable|
|operations/puppet : production||Add curl to standard packages|
|operations/puppet : production||Install ack instead of ack-grep|
|operations/puppet : production||Update timedatectl Icinga check for buster|
|operations/puppet : production||Extend d-i config for buster|
|operations/puppet : production||Add support for buster-wikimedia to our internal repository|
|Resolved||MoritzMuehlenhoff||T213527 Prepare our base system layer for Debian buster|
|Open||None||T193224 Evaluate and decide the future of relational datastore at WMF after the upgrade of MariaDB 10.1 is finished|
|Stalled||None||T193226 Test MySQL 8.0 with production data and evaluate its fit for WMF databases|
|Resolved||jbond||T213546 Prepare puppet infrastructure for Debian buster|
|Resolved||fgiunchedi||T218118 prometheus-node-exporter makes puppet fail because requiring a version that no longer exists on buster|
Still some rough edges to sort out, but bare metal installations are working now:
$ ssh graphite2002.codfw.wmnet Linux graphite2002 4.19.0-2-amd64 #1 SMP Debian 4.19.16-1 (2019-01-17) x86_64 Debian GNU/Linux buster/sid graphite2002 is a Unused spare system (spare::system) The last Puppet run was at Fri Feb 8 15:45:45 UTC 2019 (2 minutes ago).
@GTirloni: That's a temporary installer issue, the kernel modules on the last installer images provided use a different kernel ABI than the current kernel in the archive. Once installed the driver is fully functional. That said, the NFS servers are currently not a suitable candidate for early d-i testing I'd say, it seems better to resume with Stretch anyway.
Installations in Ganeti are currently blocked for a long time waiting for entropy in the d-i step which generates an SSH host key. This is resolved once 4.9.20-1 is migrated to testing; it enables the kernel to gather entropy from the CPU-internal RNG on the virtualisation server (all our Ganeti servers support that and the rdrand instruction is already passed down to Ganeti VMs.
After the buster upgrade, what appears to be the debmonitor hook fails on apt update, upgrade at db1114 with:
su: warning: cannot change directory to /nonexistent: No such file or directory
Not sure if buster or just the partially aborted upgrade (this wasn't a clean reinstall).
We have a number of buster hosts running in production and fresh installs of Buster are now working fine, I'm closing this task. All further refinements can happen via separate tasks/patches.