Page MenuHomePhabricator

Remove systemd from openstack-mitaka
Open, NormalPublic

Description

I noticed that on cloudvirt1015 (a stretch host using the openstack-mitaka-jessie component), systemd-sysv was flagged for downgrade (from 232-25+deb9u11 to 230-7+bpo8+2 from the mitaka component), the other installed systemd packages are not flagged for downgrade.

systemd should be removed from the component, having different systemd packages installed on a system will lead to all kind of subtle issues.

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 2 2019, 9:55 AM
MoritzMuehlenhoff triaged this task as Normal priority.Sep 2 2019, 9:55 AM
JHedden claimed this task.Sep 11 2019, 3:04 PM
JHedden moved this task from Needs discussion to Doing on the cloud-services-team (Kanban) board.

Mentioned in SAL (#wikimedia-operations) [2019-09-24T12:22:37Z] <arturo> remove systemd-sysv from jessie-wikimedia/openstack-mitaka-jessie in install1002 (T231793)

Once the bpo package is dropped from the component, after an "apt-get update", please also align the following systems to use systemd-sysv=232-25+deb9u12:

loudcontrol1003.wikimedia.org 	
cloudcontrol1004.wikimedia.org 	
cloudservices2002-dev.wikimedia.org
cloudvirt1001.eqiad.wmnet 	
cloudvirt1002.eqiad.wmnet 	
cloudvirt1003.eqiad.wmnet 	
cloudvirt1005.eqiad.wmnet 	
cloudvirt1024.eqiad.wmnet 	
labpuppetmaster1001.wikimedia.org 	
labpuppetmaster1002.wikimedia.org 	
labstore1004.eqiad.wmnet 	
labstore1005.eqiad.wmnet 	
labtestpuppetmaster2001.wikimedia.org

@JHedden I hijacked this part of the task. This is what I did for the record:

I confirmed we had the right setting here: modules/aptrepo/files/reprepro-update-filter-wmcs-openstack-jessie-bpo.sh (no references to systemd), and then a couple of commands

aborrero@install1002:~ $ sudo -i reprepro -C openstack-mitaka-jessie remove jessie-wikimedia systemd-sysv
Exporting indices...
Deleting files no longer referenced...
aborrero@cumin1001:~ $ sudo cumin A:cloud-eqiad1 "apt-get update"
[...]
aborrero@cumin1001:~ $ sudo cumin A:cloud-eqiad1 "apt-cache policy systemd-sysv"
34 hosts will be targeted:
cloudcontrol[1003-1004].wikimedia.org,cloudnet[1003-1004].eqiad.wmnet,cloudservices[1003-1004].wikimedia.org,cloudvirt[1001-1009,1012-1030].eqiad.wmnet
Confirm to continue [y/n]? y
===== NODE GROUP =====                                                                                                                                                                                                                                                                    
(6) cloudcontrol[1003-1004].wikimedia.org,cloudvirt[1001-1003,1005].eqiad.wmnet                                                                                                                                                                                                           
----- OUTPUT of 'apt-cache policy systemd-sysv' -----                                                                                                                                                                                                                                     
systemd-sysv:                                                                                                                                                                                                                                                                             
  Installed: 230-7~bpo8+2                                                                                                                                                                                                                                                                 
  Candidate: 232-25+deb9u12
  Version table:
     241-5~bpo9+1 100
        100 http://mirrors.wikimedia.org/debian stretch-backports/main amd64 Packages
     232-25+deb9u12 500
        500 http://mirrors.wikimedia.org/debian stretch/main amd64 Packages
     232-25+deb9u11 500
        500 http://security.debian.org/debian-security stretch/updates/main amd64 Packages
 *** 230-7~bpo8+2 100
        100 /var/lib/dpkg/status
===== NODE GROUP =====                                                                                                                                                                                                                                                                    
(24) cloudnet[1003-1004].eqiad.wmnet,cloudvirt[1004,1006-1009,1012-1014,1016-1023,1025-1030].eqiad.wmnet                                                                                                                                                                                  
----- OUTPUT of 'apt-cache policy systemd-sysv' -----                                                                                                                                                                                                                                     
systemd-sysv:                                                                                                                                                                                                                                                                             
  Installed: 232-25+deb9u11                                                                                                                                                                                                                                                               
  Candidate: 232-25+deb9u12
  Version table:
     241-5~bpo9+1 100
        100 http://mirrors.wikimedia.org/debian stretch-backports/main amd64 Packages
     232-25+deb9u12 500
        500 http://mirrors.wikimedia.org/debian stretch/main amd64 Packages
 *** 232-25+deb9u11 500
        500 http://security.debian.org/debian-security stretch/updates/main amd64 Packages
        100 /var/lib/dpkg/status
===== NODE GROUP =====                                                                                                                                                                                                                                                                    
(2) cloudservices[1003-1004].wikimedia.org                                                                                                                                                                                                                                                
----- OUTPUT of 'apt-cache policy systemd-sysv' -----                                                                                                                                                                                                                                     
systemd-sysv:                                                                                                                                                                                                                                                                             
  Installed: 232-25+deb9u11                                                                                                                                                                                                                                                               
  Candidate: 232-25+deb9u12
  Version table:
     241-5~bpo9+1 100
        100 http://mirrors.wikimedia.org/debian stretch-backports/main amd64 Packages
     232-25+deb9u12 500
        500 http://mirrors.wikimedia.org/debian stretch/main amd64 Packages
 *** 232-25+deb9u11 500
        500 http://security.debian.org/debian-security stretch/updates/main amd64 Packages
        100 /var/lib/dpkg/status
     215-17+deb8u7 500
        500 http://archive.debian.org/debian-archive/debian jessie/main amd64 Packages
[...]

Once the bpo package is dropped from the component, after an "apt-get update", please also align the following systems to use systemd-sysv=232-25+deb9u12:

(or 232-25+deb9u11 depending on which version is currently installed)

We're precariously close to upgrading to Newton, so maybe this is moot?

We're precariously close to upgrading to Newton, so maybe this is moot?

But the servers will remain at Stretch, so it's all the more important that they use the supported distro version before upgrading the OpenStack bits.

Bstorm added a subscriber: Bstorm.Tue, Sep 24, 3:10 PM

We're precariously close to upgrading to Newton, so maybe this is moot?

But the servers will remain at Stretch, so it's all the more important that they use the supported distro version before upgrading the OpenStack bits.

But Newton is the distro version of OpenStack. We used Mitaka for the upgrade path from Jessie. If we can get Newton working on stretch, then the downgrade is perhaps not as likely to be triggered? It seems to me that the two issues are likely to be linked, however, it may be helpful to keep this open until it is validated.

After the Newton upgrade the Stretch hosts areall running the distro version: systemd-sysv 232-25+deb9u11

  • cloudcontrol[1003-1004].wikimedia.org
  • cloudnet[1003-1004].eqiad.wmnet
  • cloudvirt[1001-1009,1012-1014,1016-1030].eqiad.wmnet

cloudvirt1015 is down for maintenance and will be updated when it comes back online.


The jessie hosts are using the backported version 230-7+bpo8+2, but it doesn't look like there's an upgrade or flagged downgrade.

  • labpuppetmaster[1001-1002].wikimedia.org
  • labstore[1004-1005].eqiad.wmnet
  • labtestpuppetmaster2001.wikimedia.org
labstore1004:~$ apt-cache policy systemd-sysv
systemd-sysv:
  Installed: 230-7~bpo8+2
  Candidate: 230-7~bpo8+2
  Version table:
 *** 230-7~bpo8+2 0
        100 /var/lib/dpkg/status
     215-17+deb8u13 0
        500 http://security.debian.org/ jessie/updates/main amd64 Packages
     215-17+deb8u7 0
        500 http://mirrors.wikimedia.org/debian/ jessie/main amd64 Packages