I noticed that on cloudvirt1015 (a stretch host using the openstack-mitaka-jessie component), systemd-sysv was flagged for downgrade (from 232-25+deb9u11 to 230-7+bpo8+2 from the mitaka component), the other installed systemd packages are not flagged for downgrade.
systemd should be removed from the component, having different systemd packages installed on a system will lead to all kind of subtle issues.
Mentioned in SAL (#wikimedia-operations) [2019-09-24T12:22:37Z] <arturo> remove systemd-sysv from jessie-wikimedia/openstack-mitaka-jessie in install1002 (T231793)
Once the bpo package is dropped from the component, after an "apt-get update", please also align the following systems to use systemd-sysv=232-25+deb9u12:
loudcontrol1003.wikimedia.org cloudcontrol1004.wikimedia.org cloudservices2002-dev.wikimedia.org cloudvirt1001.eqiad.wmnet cloudvirt1002.eqiad.wmnet cloudvirt1003.eqiad.wmnet cloudvirt1005.eqiad.wmnet cloudvirt1024.eqiad.wmnet labpuppetmaster1001.wikimedia.org labpuppetmaster1002.wikimedia.org labstore1004.eqiad.wmnet labstore1005.eqiad.wmnet labtestpuppetmaster2001.wikimedia.org
@JHedden I hijacked this part of the task. This is what I did for the record:
I confirmed we had the right setting here: modules/aptrepo/files/reprepro-update-filter-wmcs-openstack-jessie-bpo.sh (no references to systemd), and then a couple of commands
aborrero@install1002:~ $ sudo -i reprepro -C openstack-mitaka-jessie remove jessie-wikimedia systemd-sysv Exporting indices... Deleting files no longer referenced...
aborrero@cumin1001:~ $ sudo cumin A:cloud-eqiad1 "apt-get update" [...]
aborrero@cumin1001:~ $ sudo cumin A:cloud-eqiad1 "apt-cache policy systemd-sysv"
34 hosts will be targeted:
cloudcontrol[1003-1004].wikimedia.org,cloudnet[1003-1004].eqiad.wmnet,cloudservices[1003-1004].wikimedia.org,cloudvirt[1001-1009,1012-1030].eqiad.wmnet
Confirm to continue [y/n]? y
===== NODE GROUP =====
(6) cloudcontrol[1003-1004].wikimedia.org,cloudvirt[1001-1003,1005].eqiad.wmnet
----- OUTPUT of 'apt-cache policy systemd-sysv' -----
systemd-sysv:
Installed: 230-7~bpo8+2
Candidate: 232-25+deb9u12
Version table:
241-5~bpo9+1 100
100 http://mirrors.wikimedia.org/debian stretch-backports/main amd64 Packages
232-25+deb9u12 500
500 http://mirrors.wikimedia.org/debian stretch/main amd64 Packages
232-25+deb9u11 500
500 http://security.debian.org/debian-security stretch/updates/main amd64 Packages
*** 230-7~bpo8+2 100
100 /var/lib/dpkg/status
===== NODE GROUP =====
(24) cloudnet[1003-1004].eqiad.wmnet,cloudvirt[1004,1006-1009,1012-1014,1016-1023,1025-1030].eqiad.wmnet
----- OUTPUT of 'apt-cache policy systemd-sysv' -----
systemd-sysv:
Installed: 232-25+deb9u11
Candidate: 232-25+deb9u12
Version table:
241-5~bpo9+1 100
100 http://mirrors.wikimedia.org/debian stretch-backports/main amd64 Packages
232-25+deb9u12 500
500 http://mirrors.wikimedia.org/debian stretch/main amd64 Packages
*** 232-25+deb9u11 500
500 http://security.debian.org/debian-security stretch/updates/main amd64 Packages
100 /var/lib/dpkg/status
===== NODE GROUP =====
(2) cloudservices[1003-1004].wikimedia.org
----- OUTPUT of 'apt-cache policy systemd-sysv' -----
systemd-sysv:
Installed: 232-25+deb9u11
Candidate: 232-25+deb9u12
Version table:
241-5~bpo9+1 100
100 http://mirrors.wikimedia.org/debian stretch-backports/main amd64 Packages
232-25+deb9u12 500
500 http://mirrors.wikimedia.org/debian stretch/main amd64 Packages
*** 232-25+deb9u11 500
500 http://security.debian.org/debian-security stretch/updates/main amd64 Packages
100 /var/lib/dpkg/status
215-17+deb8u7 500
500 http://archive.debian.org/debian-archive/debian jessie/main amd64 Packages
[...]But the servers will remain at Stretch, so it's all the more important that they use the supported distro version before upgrading the OpenStack bits.
But Newton is the distro version of OpenStack. We used Mitaka for the upgrade path from Jessie. If we can get Newton working on stretch, then the downgrade is perhaps not as likely to be triggered? It seems to me that the two issues are likely to be linked, however, it may be helpful to keep this open until it is validated.
After the Newton upgrade the Stretch hosts areall running the distro version: systemd-sysv 232-25+deb9u11
cloudvirt1015 is down for maintenance and will be updated when it comes back online.
The jessie hosts are using the backported version 230-7+bpo8+2, but it doesn't look like there's an upgrade or flagged downgrade.
labstore1004:~$ apt-cache policy systemd-sysv
systemd-sysv:
Installed: 230-7~bpo8+2
Candidate: 230-7~bpo8+2
Version table:
*** 230-7~bpo8+2 0
100 /var/lib/dpkg/status
215-17+deb8u13 0
500 http://security.debian.org/ jessie/updates/main amd64 Packages
215-17+deb8u7 0
500 http://mirrors.wikimedia.org/debian/ jessie/main amd64 Packages