Zuul has started failing on some repo's in gerrit.wikimedia.org
Closed, ResolvedPublic

Description

Steps to reproduce

Actual results

  • it is failing with 08:48:23 ValueError: Unknown remainder ['-8-gcbc7f62-wmf4jessie1'] in '2.5.0-8-gcbc7f62-wmf4jessie1'
  • See failures

Expected results

  • That test should pass without that error.

This seems like a fix https://review.openstack.org/#/c/116775/ which is included from pbr release 0.11.0+

Paladox created this task.Dec 21 2016, 4:38 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 21 2016, 4:38 PM
Paladox triaged this task as "Unbreak Now!" priority.Dec 21 2016, 4:38 PM

Setting to unbreak since the problem just happened and it could happen anytime in production causing ci to go down.

Restricted Application added subscribers: Jay8g, TerraCodes. · View Herald TranscriptDec 21 2016, 4:38 PM

Change 328538 had a related patch set (by Paladox) published:
Update python-pbr to 1.10.0

https://gerrit.wikimedia.org/r/328538

Paladox added a comment.EditedDec 21 2016, 4:53 PM

The failures are spreading, now https://integration.wikimedia.org/ci/job/debian-glue-non-voting/475/console fails.

All tests that do zuul --version will fail on Jessie.

Paladox added a comment.EditedDec 21 2016, 4:55 PM

strange nodepool unaffected but instances are.

though instances are controlled by puppet whereas nodepool images need to be built.

greg added subscribers: Andrew, greg.Dec 21 2016, 4:59 PM

@Andrew this is probably due to the python package upgrades, fyi

Paladox added a comment.EditedDec 21 2016, 4:59 PM

Yep, but strange as I did an update on one of my test instances and zuul --version works.

Preparing to unpack .../python-pbr_1.10.0-1_all.deb ...
Unpacking python-pbr (1.10.0-1) over (0.8.2-1) ...
Preparing to unpack .../python-urllib3_1.19.1-1_all.deb ...
Unpacking python-urllib3 (1.19.1-1) over (1.9.1-3) ...
Preparing to unpack .../python-requests_2.12.3-1_all.deb ...
Unpacking python-requests (2.12.3-1) over (2.4.3-6) ...
Setting up python-pbr (1.10.0-1) ...
update-alternatives: using /usr/bin/python2-pbr to provide /usr/bin/pbr (pbr) in auto mode
Setting up python-urllib3 (1.19.1-1) ...
Setting up python-requests (2.12.3-1) ...
root@gerrit-test:/home/paladox# zuul --version
Zuul version: 2.5.0-8-gcbc7f62-wmf4jessie1

Paladox changed the title from "Zuul has started failing on the integration/config repo" to "Zuul has started failing on some repo's in gerrit.wikimedia.org".Dec 21 2016, 5:02 PM

Um, I managed to reproduce on a test slave.

root@jenkins-slave-01:/home/paladox# zuul --version
Traceback (most recent call last):

File "/usr/bin/zuul", line 10, in <module>
  sys.exit(main())
File "/usr/share/python/zuul/local/lib/python2.7/site-packages/zuul/cmd/client.py", line 299, in main
  client.main()
File "/usr/share/python/zuul/local/lib/python2.7/site-packages/zuul/cmd/client.py", line 113, in main
  self.parse_arguments()
File "/usr/share/python/zuul/local/lib/python2.7/site-packages/zuul/cmd/client.py", line 41, in parse_arguments
  version=self._get_version(),
File "/usr/share/python/zuul/local/lib/python2.7/site-packages/zuul/cmd/__init__.py", line 66, in _get_version
  return "Zuul version: %s" % zuul_version_info.release_string()
File "/usr/lib/python2.7/dist-packages/pbr/version.py", line 455, in release_string
  return self.semantic_version().release_string()
File "/usr/lib/python2.7/dist-packages/pbr/version.py", line 460, in semantic_version
  self._semantic = self._get_version_from_pkg_resources()
File "/usr/lib/python2.7/dist-packages/pbr/version.py", line 448, in _get_version_from_pkg_resources
  return SemanticVersion.from_pip_string(result_string)
File "/usr/lib/python2.7/dist-packages/pbr/version.py", line 147, in from_pip_string
  return klass._from_pip_string_unsafe(version_string)
File "/usr/lib/python2.7/dist-packages/pbr/version.py", line 223, in _from_pip_string_unsafe
  % (remainder, version_string))

ValueError: Unknown remainder ['-8-gcbc7f62-wmf4jessie1'] in '2.5.0-8-gcbc7f62-wmf4jessie1'

root@jenkins-slave-01:/home/paladox# pbr -v
1.10.0

Why do we have two files like this

-rw-r--r-- 1 root root 26708 Sep 21 14:14 util.py
-rw-r--r-- 1 root root 17802 Dec 21 06:44 util.pyc
-rw-r--r-- 1 root root 18809 Sep 21 14:14 version.py
-rw-r--r-- 1 root root 16948 Dec 21 06:44 version.pyc

the .pyc seems to be the only ones that have been updated / created. shoulden we be using .py?

Paladox added a comment.EditedDec 21 2016, 5:11 PM

at 06:44 utc + 0 am this update was installed for pbr.

hashar edited projects, added Operations; removed Patch-For-Review.Dec 21 2016, 5:11 PM

@Andrew pbr 1.10.0 is broken it fails to recognize a version such as the Zuul one 2.5.0-8-gcbc7f62-wmf4jessie1.

For the context, Zuul is installed in its own environment to fulfill some package dependencies that are outdated on apt.wikimedia.org, but otherwise depends on Debian system package.

Repro on integration-slave-jessie-1001.integration.eqiad.wmflabs

Actual version of Zuul:

$ /usr/share/python/zuul/bin/pip freeze|grep zuul
zuul==2.5.0-8-gcbc7f62-wmf4jessie1

python-pbr system package

$ apt-cache policy python-pbr
python-pbr:
  Installed: 1.10.0-1
  Candidate: 1.10.0-1
  Version table:
 *** 1.10.0-1 0
       1001 http://apt.wikimedia.org/wikimedia/ jessie-wikimedia/backports amd64 Packages
        100 /var/lib/dpkg/status
     1.8.0-4.1~bpo8+1 0
        100 http://mirrors.wikimedia.org/debian/ jessie-backports/main amd64 Packages
     0.8.2-1 0
        500 http://mirrors.wikimedia.org/debian/ jessie/main amd64 Packages

It got upgraded on the CI slaves because we have unattended-upgrades enabled.

Repro in Zuul itself:

$ /usr/share/python/zuul/bin/python
Python 2.7.9 (default, Jun 29 2016, 13:08:31) 
[GCC 4.9.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import pbr.version
>>> pbr.version.VersionInfo('zuul')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.7/dist-packages/pbr/version.py", line 429, in __repr__
    self.package, self.version_string())
  File "/usr/lib/python2.7/dist-packages/pbr/version.py", line 465, in version_string
    return self.semantic_version().brief_string()
  File "/usr/lib/python2.7/dist-packages/pbr/version.py", line 460, in semantic_version
    self._semantic = self._get_version_from_pkg_resources()
  File "/usr/lib/python2.7/dist-packages/pbr/version.py", line 448, in _get_version_from_pkg_resources
    return SemanticVersion.from_pip_string(result_string)
  File "/usr/lib/python2.7/dist-packages/pbr/version.py", line 147, in from_pip_string
    return klass._from_pip_string_unsafe(version_string)
  File "/usr/lib/python2.7/dist-packages/pbr/version.py", line 223, in _from_pip_string_unsafe
    % (remainder, version_string))
ValueError: Unknown remainder ['-8-gcbc7f62-wmf4jessie1'] in '2.5.0-8-gcbc7f62-wmf4jessie1'
>>> 
Paladox added a comment.EditedDec 21 2016, 5:13 PM

@hashar but it works on another test instance for me.

gerrit-test.

root@gerrit-test:/usr/lib/python2.7/dist-packages/pbr# /usr/share/python/zuul/bin/pip freeze|grep zuul
zuul==2.5.0-8-gcbc7f62-wmf4jessie1

root@gerrit-test:/usr/lib/python2.7/dist-packages/pbr# apt-cache policy python-pbr
python-pbr:

Installed: 1.10.0-1
Candidate: 1.10.0-1
Version table:

root@gerrit-test:/usr/lib/python2.7/dist-packages/pbr# /usr/share/python/zuul/bin/python
Python 2.7.9 (default, Jun 29 2016, 13:08:31)
[GCC 4.9.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.

KeyboardInterrupt

import pbr.version
pbr.version.VersionInfo('zuul')

pbr.version.VersionInfo(zuul:2.5.0-8-gcbc7f62-wmf4jessie1)

I manually ran apt-get update then apt-get upgrade then did n, then I copy and pasted the packages that were shown and did apt-get install <package>

so seems that it is broken since puppet installs it but doing it your self works.

Change 328538 abandoned by Paladox:
Update python-pbr to 1.10.0

https://gerrit.wikimedia.org/r/328538

We had version 0.8.2

The CI instance have an unattended-upgrade for repositories *-jessie which thus match jessie-wikimedia/backports and upgraded us to 1.10.

Version 0.11.0 of pbr honors SemanticVersion with https://review.openstack.org/#/c/94108/ and 2.5.0-8-gcbc7f62-wmf4jessie1 is not semantic (has too many dashes) hence the failure....

Change 328555 had a related patch set uploaded (by Andrew Bogott):
Pin python-pbr to an old version for Zuul

https://gerrit.wikimedia.org/r/328555

Cherry picked https://gerrit.wikimedia.org/r/328555 on the CI puppet master which adds an apt pin. Following recommendations from Andrew, I have purged zuul/python-pbr from integration-slave-jessie-1001 and integration-slave-jessie-1002 and ran puppet. They are now using 0.8.2.

The root cause is python-pbr 0.11 introduced Semantic Versioning which does not work with the Zuul version string we use in the Debian package.

Gotta check the Nodepool images now.

Change 328555 merged by Andrew Bogott:
Pin python-pbr to an old version for Zuul

https://gerrit.wikimedia.org/r/328555

hashar closed this task as "Resolved".Dec 21 2016, 7:25 PM
hashar assigned this task to Andrew.

Ran puppet on contint1001 / contint2001 and the package pinning is fine:

contint1001:~# apt-cache policy python-pbr
python-pbr:
  Installed: 0.8.2-1
  Candidate: 0.8.2-1
  Package pin: 0.8.2-1
  Version table:
     1.10.0-1 1002
       1001 http://apt.wikimedia.org/wikimedia/ jessie-wikimedia/backports amd64 Packages
     1.8.0-4.1~bpo8+1 1002
        100 http://mirrors.wikimedia.org/debian/ jessie-backports/main amd64 Packages
 *** 0.8.2-1 1002
        500 http://mirrors.wikimedia.org/debian/ jessie/main amd64 Packages
        100 /var/lib/dpkg/status

Thank you Andrew!

Change 328554 had a related patch set uploaded (by Paladox):
Make zuul semver compatible with pbr 1.10.0

https://gerrit.wikimedia.org/r/328554

Change 330114 had a related patch set uploaded (by Hashar):
dib: install zuul via puppet class

https://gerrit.wikimedia.org/r/330114

Change 330114 merged by Hashar:
dib: install zuul via puppet class

https://gerrit.wikimedia.org/r/330114

Mentioned in SAL (#wikimedia-releng) [2017-01-02T11:17:21Z] <hashar> Jessie images have the wrong python-pbr version ( T153877 ) causing zuul-cloner to fail. Refreshing image

Change 328554 abandoned by Paladox:
Make zuul semver compatible with pbr 1.10.0

https://gerrit.wikimedia.org/r/328554

Change 328554 restored by Paladox:
Make zuul semver compatible with pbr 1.10.0

https://gerrit.wikimedia.org/r/328554