Steps to reproduce
Actual results
Expected results
This seems like a fix https://review.openstack.org/#/c/116775/ which is included from pbr release 0.11.0+
Setting to unbreak since the problem just happened and it could happen anytime in production causing ci to go down.
upstream zuul uses pbr 1.1.0 https://phabricator.wikimedia.org/diffusion/CIZU/browse/debian%252Fjessie-wikimedia/requirements.txt;f28830b04f75ed7400d058e557516cabf468cde4$1 we soften that to 0.5 with https://phabricator.wikimedia.org/diffusion/CIZU/browse/debian%252Fjessie-wikimedia/debian/patches/0010-WMF-soften-pbr-requirement.patch
Change 328538 had a related patch set (by Paladox) published:
Update python-pbr to 1.10.0
The failures are spreading, now https://integration.wikimedia.org/ci/job/debian-glue-non-voting/475/console fails.
All tests that do zuul --version will fail on Jessie.
strange nodepool unaffected but instances are.
though instances are controlled by puppet whereas nodepool images need to be built.
Yep, but strange as I did an update on one of my test instances and zuul --version works.
Preparing to unpack .../python-pbr_1.10.0-1_all.deb ...
Unpacking python-pbr (1.10.0-1) over (0.8.2-1) ...
Preparing to unpack .../python-urllib3_1.19.1-1_all.deb ...
Unpacking python-urllib3 (1.19.1-1) over (1.9.1-3) ...
Preparing to unpack .../python-requests_2.12.3-1_all.deb ...
Unpacking python-requests (2.12.3-1) over (2.4.3-6) ...
Setting up python-pbr (1.10.0-1) ...
update-alternatives: using /usr/bin/python2-pbr to provide /usr/bin/pbr (pbr) in auto mode
Setting up python-urllib3 (1.19.1-1) ...
Setting up python-requests (2.12.3-1) ...
root@gerrit-test:/home/paladox# zuul --version
Zuul version: 2.5.0-8-gcbc7f62-wmf4jessie1
Um, I managed to reproduce on a test slave.
root@jenkins-slave-01:/home/paladox# zuul --version
Traceback (most recent call last):
File "/usr/bin/zuul", line 10, in <module> sys.exit(main()) File "/usr/share/python/zuul/local/lib/python2.7/site-packages/zuul/cmd/client.py", line 299, in main client.main() File "/usr/share/python/zuul/local/lib/python2.7/site-packages/zuul/cmd/client.py", line 113, in main self.parse_arguments() File "/usr/share/python/zuul/local/lib/python2.7/site-packages/zuul/cmd/client.py", line 41, in parse_arguments version=self._get_version(), File "/usr/share/python/zuul/local/lib/python2.7/site-packages/zuul/cmd/__init__.py", line 66, in _get_version return "Zuul version: %s" % zuul_version_info.release_string() File "/usr/lib/python2.7/dist-packages/pbr/version.py", line 455, in release_string return self.semantic_version().release_string() File "/usr/lib/python2.7/dist-packages/pbr/version.py", line 460, in semantic_version self._semantic = self._get_version_from_pkg_resources() File "/usr/lib/python2.7/dist-packages/pbr/version.py", line 448, in _get_version_from_pkg_resources return SemanticVersion.from_pip_string(result_string) File "/usr/lib/python2.7/dist-packages/pbr/version.py", line 147, in from_pip_string return klass._from_pip_string_unsafe(version_string) File "/usr/lib/python2.7/dist-packages/pbr/version.py", line 223, in _from_pip_string_unsafe % (remainder, version_string))
ValueError: Unknown remainder ['-8-gcbc7f62-wmf4jessie1'] in '2.5.0-8-gcbc7f62-wmf4jessie1'
Why do we have two files like this
-rw-r--r-- 1 root root 26708 Sep 21 14:14 util.py
-rw-r--r-- 1 root root 17802 Dec 21 06:44 util.pyc
-rw-r--r-- 1 root root 18809 Sep 21 14:14 version.py
-rw-r--r-- 1 root root 16948 Dec 21 06:44 version.pyc
the .pyc seems to be the only ones that have been updated / created. shoulden we be using .py?
@Andrew pbr 1.10.0 is broken it fails to recognize a version such as the Zuul one 2.5.0-8-gcbc7f62-wmf4jessie1.
For the context, Zuul is installed in its own environment to fulfill some package dependencies that are outdated on apt.wikimedia.org, but otherwise depends on Debian system package.
Repro on integration-slave-jessie-1001.integration.eqiad.wmflabs
Actual version of Zuul:
$ /usr/share/python/zuul/bin/pip freeze|grep zuul zuul==2.5.0-8-gcbc7f62-wmf4jessie1
python-pbr system package
$ apt-cache policy python-pbr python-pbr: Installed: 1.10.0-1 Candidate: 1.10.0-1 Version table: *** 1.10.0-1 0 1001 http://apt.wikimedia.org/wikimedia/ jessie-wikimedia/backports amd64 Packages 100 /var/lib/dpkg/status 1.8.0-4.1~bpo8+1 0 100 http://mirrors.wikimedia.org/debian/ jessie-backports/main amd64 Packages 0.8.2-1 0 500 http://mirrors.wikimedia.org/debian/ jessie/main amd64 Packages
It got upgraded on the CI slaves because we have unattended-upgrades enabled.
Repro in Zuul itself:
$ /usr/share/python/zuul/bin/python Python 2.7.9 (default, Jun 29 2016, 13:08:31) [GCC 4.9.2] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import pbr.version >>> pbr.version.VersionInfo('zuul') Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/lib/python2.7/dist-packages/pbr/version.py", line 429, in __repr__ self.package, self.version_string()) File "/usr/lib/python2.7/dist-packages/pbr/version.py", line 465, in version_string return self.semantic_version().brief_string() File "/usr/lib/python2.7/dist-packages/pbr/version.py", line 460, in semantic_version self._semantic = self._get_version_from_pkg_resources() File "/usr/lib/python2.7/dist-packages/pbr/version.py", line 448, in _get_version_from_pkg_resources return SemanticVersion.from_pip_string(result_string) File "/usr/lib/python2.7/dist-packages/pbr/version.py", line 147, in from_pip_string return klass._from_pip_string_unsafe(version_string) File "/usr/lib/python2.7/dist-packages/pbr/version.py", line 223, in _from_pip_string_unsafe % (remainder, version_string)) ValueError: Unknown remainder ['-8-gcbc7f62-wmf4jessie1'] in '2.5.0-8-gcbc7f62-wmf4jessie1' >>>
@hashar but it works on another test instance for me.
gerrit-test.
root@gerrit-test:/usr/lib/python2.7/dist-packages/pbr# /usr/share/python/zuul/bin/pip freeze|grep zuul
zuul==2.5.0-8-gcbc7f62-wmf4jessie1
root@gerrit-test:/usr/lib/python2.7/dist-packages/pbr# apt-cache policy python-pbr
python-pbr:
Installed: 1.10.0-1 Candidate: 1.10.0-1 Version table:
root@gerrit-test:/usr/lib/python2.7/dist-packages/pbr# /usr/share/python/zuul/bin/python
Python 2.7.9 (default, Jun 29 2016, 13:08:31)
[GCC 4.9.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
KeyboardInterrupt
import pbr.version
pbr.version.VersionInfo('zuul')
pbr.version.VersionInfo(zuul:2.5.0-8-gcbc7f62-wmf4jessie1)
I manually ran apt-get update then apt-get upgrade then did n, then I copy and pasted the packages that were shown and did apt-get install <package>
so seems that it is broken since puppet installs it but doing it your self works.
We had version 0.8.2
The CI instance have an unattended-upgrade for repositories *-jessie which thus match jessie-wikimedia/backports and upgraded us to 1.10.
Version 0.11.0 of pbr honors SemanticVersion with https://review.openstack.org/#/c/94108/ and 2.5.0-8-gcbc7f62-wmf4jessie1 is not semantic (has too many dashes) hence the failure....
Change 328555 had a related patch set uploaded (by Andrew Bogott):
Pin python-pbr to an old version for Zuul
Cherry picked https://gerrit.wikimedia.org/r/328555 on the CI puppet master which adds an apt pin. Following recommendations from Andrew, I have purged zuul/python-pbr from integration-slave-jessie-1001 and integration-slave-jessie-1002 and ran puppet. They are now using 0.8.2.
The root cause is python-pbr 0.11 introduced Semantic Versioning which does not work with the Zuul version string we use in the Debian package.
Gotta check the Nodepool images now.
Ran puppet on contint1001 / contint2001 and the package pinning is fine:
contint1001:~# apt-cache policy python-pbr
python-pbr:
Installed: 0.8.2-1
Candidate: 0.8.2-1
Package pin: 0.8.2-1
Version table:
1.10.0-1 1002
1001 http://apt.wikimedia.org/wikimedia/ jessie-wikimedia/backports amd64 Packages
1.8.0-4.1~bpo8+1 1002
100 http://mirrors.wikimedia.org/debian/ jessie-backports/main amd64 Packages
*** 0.8.2-1 1002
500 http://mirrors.wikimedia.org/debian/ jessie/main amd64 Packages
100 /var/lib/dpkg/statusThank you Andrew!
Change 328554 had a related patch set uploaded (by Paladox):
Make zuul semver compatible with pbr 1.10.0
Change 330114 had a related patch set uploaded (by Hashar):
dib: install zuul via puppet class
Mentioned in SAL (#wikimedia-releng) [2017-01-02T11:17:21Z] <hashar> Jessie images have the wrong python-pbr version ( T153877 ) causing zuul-cloner to fail. Refreshing image
Mentioned in SAL (#wikimedia-releng) [2017-06-16T14:40:50Z] <hashar> integration-slave-jessie-1001 apt-get upgrade to downgrade python-pbr to 0.8.2 as pinned since T153877. /usr/bin/unattended-upgrade magically upgraded it for some reason
Just ran into this again while trying to do "apt-get install zuul" on the base debian wikimedia docker image.
It would be great to get this rebuilt rather than this hack :/
Is there a ticket for the actual issue @hashar ?
Yup you need python-pbr 0.8 until I manage to find the time to rebuild our Zuul package and come up with a Semver version :(
2.5.0-8-gcbc7f62-wmf4jessie1 comes from the Debian changelog iirc and is then extracted with something like:
pbr.version.VersionInfo('zuul').release_string()