We are running mixed swift versions (1.3 - 2.2) and distributions (trusty / jessie) at the moment, in T162348: swift-object-server 1.13.1: Wrong Content-Type returned on 304 Not Modified responses a 1.3-specific bug has been uncovered. We should aim at running the same swift version across the fleet, ideally the same distro as well.
Current situation:
eqiad
| hosts | distro | swift version | notes |
|---|---|---|---|
| ms-be[1001-1012] | trusty | 1.13.1 | decom |
| ms-be[1013-1021] | stretch | 2.10.1-3 | |
| ms-be[1022-1039] | jessie | 2.10.1-2~bpo8+1 | |
| ms-fe[1005-1008] | jessie | 2.10.1-2~bpo8+1 | |
codfw
| hosts | distro | swift version | notes |
|---|---|---|---|
| ms-be[2001-2012] | trusty | 1.13.1 | decom |
| ms-be[2013-2021] | stretch | 2.10.1-3 | |
| ms-be[2022-2039] | jessie | 2.10.1-2~bpo8+1 | |
| ms-fe[2005-2008] | jessie | 2.10.1-2~bpo8+1 | |
esams
Hardware to be decom, not in production.
In terms of swift versions in Debian:
| 2.2.0-1+deb8u1 | jessie |
| 2.7.0-10~bpo8+1 | jessie-backports |
| 2.10.1-2 | stretch |
| 2.10.1-2 | sid |
The end state would be swift >= 2.2 and distro >= jessie everywhere, with priority on the swift version. I've listed below possible solutions in increasing order of complexity and time requirements.
a) Backport swift 2.2 to trusty, upgrade trusty hw to 2.2
This is easy because jessie's swift 2.2 builds as-is on trusty and we'll run a single version of swift.
Swift 2.10 (from stretch) on trusty seems trickier because it requires already quite a few python dependencies: (ditto for swift 2.7 from jessie-backports)
pbuilder-satisfydepends-dummy : Depends: debhelper (>= 10) but 9.20131227ubuntu1 is to be installed.
Depends: openstack-pkg-tools (>= 48~) but it is not going to be installed.
Depends: python-setuptools (>= 20.6.8) but 3.3-1ubuntu1 is to be installed.
Depends: python-cryptography (>= 1.0) which is a virtual package.
Depends: python-dnspython (>= 1.14.0) but it is not going to be installed.
Depends: python-eventlet (>= 0.17.4) but 0.13.0-1ubuntu2 is to be installed.
Depends: python-keystoneclient (>= 1:1.3.0) but 1:0.7.1-ubuntu1 is to be installed.
Depends: python-netifaces (> 0.10.1) but it is not going to be installed.
Depends: python-openstackdocstheme (>= 1.0.3) which is a virtual package.
Depends: python-os-api-ref which is a virtual package.
Depends: python-os-testr (>= 0.4.1) which is a virtual package.
Depends: python-pyeclib (>= 1.2.0) which is a virtual package.b) Reimage trusty machines with jessie (16x reimages total, assuming old hardware decom'd)
This is doable with minimal disruption during reimage (only SSDs are wiped) but more time consuming than the upgrade above due to add the right uid/gid for swift before the first puppet run (cfr T123918). Note that this solution assumes we have already finished decommissioning old ms-be hardware (30x machines) and therefore those won't need a reimage.
c) Reimage trusty machines with stretch and upgrade to swift 2.10
Stretch ships with swift 2.10, we could reimage to stretch and upgrade jessie machines to swift 2.10 (stretch's swift package backports cleanly to jessie). The same disclaimer re: decom applies, plus testing swift 2.10 for backend and frontend.
I think we'll have to do this in steps, namely:
- Complete (a) first, run swift 2.2 everywhere and a mix of trusty/jessie
- (Also while the above is in progress) Finish decommissioning old hardware
- Backport swift 2.10 to jessie, upgrade and test existing jessie machines (running a mix of 2.2 and 2.10 while the step below is in progress). This includes upgrading ms-fe to 2.10 first and then ms-be.
- Complete (b) by reinstalling trusty hw with stretch and running swift 2.10 everywhere
- (Nice to have) reimage jessie systems with stretch. ms-fe is easy to do and we should, ms-be more time consuming but doable as well.