Page MenuHomePhabricator

Fix tests around the toollabs repo and packaging
Closed, ResolvedPublic

Description

In labs/toollabs updates to jsub will tend to fail tests in Jenkins. However, this appears due to the particular build environment on Jenkins it is using and no fault of its own when it tries to build the packages.

See https://gerrit.wikimedia.org/r/c/labs/toollabs/+/476092

Event Timeline

Bstorm created this task.Nov 28 2018, 8:52 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 28 2018, 8:52 PM

The last know build was on Gerrit #474873 which triggered both tox and debian-glue.

Then we had:

jsub: Make release a deprecated noop Gerrit 475103

Since no file got changed in the ./debian directory, the debian-glue job has not been triggered. Only the tox one.

jsub: add the changelog for stretch changes Gerrit 476092

Which touched debian/changelog and hence triggered the debian glue job. It also altered the tests/testsuite.at so it is hard to know whether this change or the previous broke the debian-glue tests. We can send a revert of that change and then:

  • if the revert pass, the change has broken the build
  • if it fails, the parent change Gerrit 475103 broke it but managed to get merged since the testsuite did not run.

Side note: it seems tests/testsuite.at are integration tests, they can probably be ported to python and hence run via tox :)

Change 476461 had a related patch set uploaded (by Hashar; owner: Hashar):
[labs/toollabs@master] Revert "jsub: add the changelog for stretch changes"

https://gerrit.wikimedia.org/r/476461

Change 476462 had a related patch set uploaded (by Hashar; owner: Hashar):
[labs/toollabs@master] Revert "jsub: Make release a deprecated noop"

https://gerrit.wikimedia.org/r/476462

Change 476462 abandoned by Hashar:
Revert "jsub: Make release a deprecated noop"

https://gerrit.wikimedia.org/r/476462

Change 476461 abandoned by Hashar:
Revert "jsub: add the changelog for stretch changes"

https://gerrit.wikimedia.org/r/476461

Fun, if I revert both patches (via Gerrit #476461 and 476462). The failure indicates:

Please send `tests/testsuite.log' and all information you think might help

But the file is not captured :(


I have manually rebuild the last known build (from Gerrit #474873) and it fails as well (build 1344) though a couple tests pass.

debian/changelog points at unstable so it could be a package got updated there that suddenly broke the build.

I crafted a change on tip of the master branch which points the distribution to stretch and it pass https://gerrit.wikimedia.org/r/#/c/labs/toollabs/+/476464/

TLDR: something changed in Debian unstable which break the build. The debian/changelog should point to stretch to fix it (eg: https://gerrit.wikimedia.org/r/#/c/labs/toollabs/+/476464/ )

Change 476464 had a related patch set uploaded (by Hashar; owner: Hashar):
[labs/toollabs@master] Build for stretch

https://gerrit.wikimedia.org/r/476464

Comparing the build log. It seems unstable uses python 3.7 now which might be the reason for ./jobutils/bin/jsub tests to start failing.

The issue might be caught when running tox with python3.7. Unfortunately CI only supports a single version of python 3 (3.5 iirc). Until we implement T191764: CI: run tests with multiple Python3 versions

-Selecting previously unselected package libpython3.6-minimal:amd64.^M
-Preparing to unpack .../libpython3.6-minimal_3.6.7-1_amd64.deb ...^M
-Unpacking libpython3.6-minimal:amd64 (3.6.7-1) ...^M
+Selecting previously unselected package libpython3.7-minimal:amd64.^M
+Preparing to unpack .../libpython3.7-minimal_3.7.1-1_amd64.deb ...^M
+Unpacking libpython3.7-minimal:amd64 (3.7.1-1) ...^M

 ## ------------------------- ##
 ## toollabs 1.33 test suite. ##
 ## ------------------------- ##
-  1: Normal call                                     ok
-  2: Quiet call                                      ok
-  3: -o points to a non-existing file                ok
-  4: -o points to a existing file                    ok
-  5: -o points to a non-existing file and -umask is used ok
-  6: -o points to a existing file and -umask is used ok
-  7: -o points to a existing directory               ok
-  8: .jsubrc is honoured                             ok
-  9: .jsubrc options are overwritten by command line arguments ok
- 10: -l is exploded                                  ok
- 11: -l h_vmem is processed                          ok
- 12: -l largest wins (virtual_free)                  ok
- 13: -l largest wins (h_vmem)                        ok
- 14: -l largest wins (default)                       ok
+  1: Normal call                                     FAILED (testsuite.at:64)
+  2: Quiet call                                      FAILED (testsuite.at:68)
+  3: -o points to a non-existing file                FAILED (testsuite.at:74)
+  4: -o points to a existing file                    FAILED (testsuite.at:84)
+  5: -o points to a non-existing file and -umask is used FAILED (testsuite.at:92)
+  6: -o points to a existing file and -umask is used FAILED (testsuite.at:102)
+  7: -o points to a existing directory               FAILED (testsuite.at:111)
+  8: .jsubrc is honoured                             FAILED (testsuite.at:120)
+  9: .jsubrc options are overwritten by command line arguments FAILED (testsuite.at:133)
+ 10: -l is exploded                                  FAILED (testsuite.at:144)
+ 11: -l h_vmem is processed                          FAILED (testsuite.at:148)
+ 12: -l largest wins (virtual_free)                  FAILED (testsuite.at:152)
+ 13: -l largest wins (h_vmem)                        FAILED (testsuite.at:156)
+ 14: -l largest wins (default)                       FAILED (testsuite.at:160)
  15: -l release=precise fails                        ok
  16: -release precise fails                          ok
- 17: -l release=trusty                               ok
- 18: -release trusty                                 ok
+ 17: -l release=trusty                               FAILED (testsuite.at:172)
+ 18: -release trusty                                 FAILED (testsuite.at:176)

The python code had issues on python 3.7 in general when I messed with it locally. If a main part of the problem is that it isn't compatible with 3.7, I'll work on making it compatible (I find python that doesn't work across versions icky).

That said, we probably do want to start pointing it at stretch for now!

Is there any way to get the log file available after the build? If a setting in the debian stuff is removing it, I can fix that.

Change 476572 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[labs/toollabs@master] jsub: Correct generator issue and silence warnings when not on tty

https://gerrit.wikimedia.org/r/476572

There! I made it work with python 3.7. I didn't want to change the release because this is still built for trusty as well.

Change 476572 merged by Bstorm:
[labs/toollabs@master] jsub: Correct generator issue and silence warnings when not on tty

https://gerrit.wikimedia.org/r/476572

Change 476464 abandoned by Hashar:
Build for stretch

Reason:
Broke because of python 3.7 that entered unstable.

Fixed by

https://gerrit.wikimedia.org/r/476572

https://gerrit.wikimedia.org/r/476464