We have two Toolforge-specific differences to how everything else in Cloud VPS does package updates:
- No unattended updates for distro-wikimedia or distro-tools repos
- Pinned versions for Kernel and sssd (profile::toolforge::apt_pinning)
Both of them are causing our systems to run on outdated software versions.
Lack of unattended updates for our own repos alone has caused a large amount of outdated packages (list is outdated versions present on at least one VM). T181647 says the clinic duty person should be manually updating things.. but I don't think anyone is.
taavi@tools-clushmaster-02:~ $ clush -w @all -N 'sudo apt-upgrade -un report' | sort -k3 | uniq buster-wikimedia/main: prometheus-rsyslog-exporter 0.0.0+git20201008-1 --> 0.0.0+git20201008-3 stretch-wikimedia/main: prometheus-rsyslog-exporter 0.0.0+git20201008-1 --> 0.0.0+git20201008-3 stretch-wikimedia/main: python3-prometheus-client 0.0.18-1 --> 0.6.0-1~wmf9u1 stretch-wikimedia/main: python-prometheus-client 0.0.18-1 --> 0.6.0-1~wmf9u1 buster-wikimedia/main: python3-wmflib 0.0.6-1+deb10u1 --> 0.0.9-1+deb10u1 buster-wikimedia/main: python3-wmflib 0.0.7-1+deb10u1 --> 0.0.9-1+deb10u1 buster-wikimedia/main: python3-wmflib 0.0.8-1+deb10u1 --> 0.0.9-1+deb10u1 stretch-wikimedia/main: cloud-init 0.7.9-2+deb9u1 --> 20.2-2~bpo10+1 buster-wikimedia/thirdparty/kubeadm-k8s-1-19: kubectl 1.17.13-00 --> 1.19.13-00 buster-wikimedia/thirdparty/kubeadm-k8s-1-19: kubectl 1.17.17-00 --> 1.19.13-00 buster-wikimedia/thirdparty/kubeadm-k8s-1-19: containerd.io 1.2.13-2 --> 1.4.8-1 buster-tools/main: jobutils 1.41 --> 1.42 buster-tools/main: misctools 1.41 --> 1.42 stretch-tools/main: jobutils 1.41 --> 1.42 buster-wikimedia/thirdparty/kubeadm-k8s-1-19: docker-ce-cli 5:18.09.9~3-0~debian-stretch --> 5:20.10.7~3-0~debian-buster buster-wikimedia/thirdparty/kubeadm-k8s-1-19: docker-ce-cli 5:19.03.14~3-0~debian-stretch --> 5:20.10.7~3-0~debian-buster buster-wikimedia/thirdparty/kubeadm-k8s-1-19: docker-ce 5:19.03.5~3-0~debian-stretch --> 5:20.10.7~3-0~debian-buster buster-wikimedia/thirdparty/kubeadm-k8s-1-19: docker-ce-cli 5:19.03.5~3-0~debian-stretch --> 5:20.10.7~3-0~debian-buster buster-wikimedia/thirdparty/elastic74: elasticsearch-curator 5.2.0-1 --> 5.8.1 oldstable/main: python-elasticsearch-curator 5.2.0-1 --> [remove] stretch-wikimedia/main: puppet 5.5.10-2~deb9u2 --> 5.5.22-1+deb9u1 buster-wikimedia/main: puppet 5.5.10-4 --> 5.5.22-1 buster-wikimedia/component/jdk8: openjdk-8-jdk 8u242-b08-1~deb10u1 --> 8u302-b08-1~deb10u1 buster-wikimedia/component/jdk8: openjdk-8-jdk-headless 8u242-b08-1~deb10u1 --> 8u302-b08-1~deb10u1 buster-wikimedia/component/jdk8: openjdk-8-jre 8u242-b08-1~deb10u1 --> 8u302-b08-1~deb10u1 buster-wikimedia/component/jdk8: openjdk-8-jre-headless 8u242-b08-1~deb10u1 --> 8u302-b08-1~deb10u1
I'm not sure about the origins of Kernel and sssd pinning, but I can't really think of any reasons why we want them - I'd much rather take the security and bugfix updates that are released to (old)stable than take the additional stability coming from not updating.