The APT package was already upgraded by unattended-upgrades to 10.6.21, but the running process is still running 10.6.20. Unattended-upgrades should not have upgraded this package, but there was an issue with the APT config, that was fixed in T385885: [toolsdb] Remove apt pinning and upgrade to latest version.
We need to run systemctl restart mariadb both in tools-db-4 and tools-db-5 for them to pick up the new binary.
This hasn't caused problems for 2 months, so I'm gonna schedule a proper maintenance window for this restart. I will restart both servers on Monday April, 28th at 13:00 UTC.
Side note: the reason we need a maintenance window is that restarting the primary can cause a few minutes of downtime, and failing over from primary to replica also creates some downtime because it requires a DNS change. Yes, we should think of a way to have zero-downtime failovers, maybe using keepalived.
Details of version mismatch
root@tools-db-5:~# zcat /var/log/unattended-upgrades/unattended-upgrades-dpkg.log.1.gz |head -4 Log started: 2025-03-01 06:05:52 (Reading database ... 47892 files and directories currently installed.) Preparing to unpack .../wmf-mariadb106_10.6.21+deb12u1_amd64.deb ... Unpacking wmf-mariadb106 (10.6.21+deb12u1) over (10.6.20+deb12u1) ...
root@tools-db-4:~# mariadb -e "SELECT VERSION()" +---------------------+ | VERSION() | +---------------------+ | 10.6.20-MariaDB-log | +---------------------+
root@tools-db-5:~# mariadb -e "SELECT VERSION()" +---------------------+ | VERSION() | +---------------------+ | 10.6.20-MariaDB-log | +---------------------+
