Page MenuHomePhabricator

labs/toollabs testsuite fails when building the Debian package
Open, MediumPublic

Description

labs/toollabs.git has CI jobs that attempt to build the Debian package against several distributions. Since migrating the CI build hosts to Buster, the suite fails:

$ DIST=buster gbp buildpackage
...
/bin/bash ../build-aux/missing --run autom4te --language=autotest -I '.' -o testsuite.tmp testsuite.at
mv testsuite.tmp testsuite
/bin/bash './testsuite' AUTOTEST_PATH=jobutils/bin:tests \
  
## ------------------------------ ##
## toollabs 1.38+nmu1 test suite. ##
## ------------------------------ ##
  1: Normal call                                     FAILED (testsuite.at:64)
  2: Quiet call                                      FAILED (testsuite.at:68)
  3: -o points to a non-existing file                FAILED (testsuite.at:74)
  4: -o points to a existing file                    FAILED (testsuite.at:84)
  5: -o points to a non-existing file and -umask is used FAILED (testsuite.at:92)
  6: -o points to a existing file and -umask is used FAILED (testsuite.at:102)
  7: -o points to a existing directory               FAILED (testsuite.at:111)
  8: .jsubrc is honoured                             FAILED (testsuite.at:120)
  9: .jsubrc options are overwritten by command line arguments FAILED (testsuite.at:133)
 10: -l is exploded                                  FAILED (testsuite.at:144)
 11: -l h_vmem is processed                          FAILED (testsuite.at:148)
 12: -l largest wins (virtual_free)                  FAILED (testsuite.at:152)
 13: -l largest wins (h_vmem)                        FAILED (testsuite.at:156)
 14: -l largest wins (default)                       FAILED (testsuite.at:160)

## ------------- ##
## Test results. ##
## ------------- ##

ERROR: All 14 tests were run,
14 failed unexpectedly.
## -------------------------- ##
## testsuite.log was created. ##
## -------------------------- ##

Please send `tests/testsuite.log' and all information you think might help:

   To: <mpelletier@wikimedia.org>
   Subject: [toollabs 1.38+nmu1] testsuite: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 failed

You may investigate any problem if you feel able to do so, in which
case the test suite provides a good starting point.  Its output may
be found below `tests/testsuite.dir'.

Using package_builder hooks and setting SHELL_ON_FAILURE=yes I drop in the build environment to inspect the files.

tests/testsuite.dir/01/testsuite.log
#                             -*- compilation -*-
1. testsuite.at:64: testing Normal call ...
./testsuite.at:64: JOBUTILS_JSUBRC=jsubrc JOBUTILS_QSUB=dummy.sh jsub true
--- /dev/null	2019-05-24 07:31:47.327554098 +0000
+++ /build/toollabs-1.38+nmu1/tests/testsuite.dir/at-groups/1/stderr	2019-12-05 14:04:20.867258032 +0000
@@ -0,0 +1 @@
+[Thu Dec  5 14:04:20 2019] Failed to touch '/nonexistent/true.out': [Errno 2] No such file or directory: '/nonexistent/true.out'
--- -	2019-12-05 14:04:20.876010132 +0000
+++ /build/toollabs-1.38+nmu1/tests/testsuite.dir/at-groups/1/stdout	2019-12-05 14:04:20.871257988 +0000
@@ -1,2 +1 @@
-Your job 501377 ("true") has been submitted
 
./testsuite.at:64: exit code was 1, expected 0
1. testsuite.at:64: 1. Normal call (testsuite.at:64): FAILED (testsuite.at:64)

Looks like pbuilder user no more has a home directory created but tests/testsuite.at refers to $HOME.

Details

Related Gerrit Patches:

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 5 2019, 2:07 PM

I have switched the CI hosts from Jessie to Buster (T224943) thus we have pbuilder updated from 0.215 to 0.230.4. In the changelog:

pbuilder (0.220) unstable; urgency=medium
...
  * configuration:
    + new config: BUILD_HOME: set the value of HOME while building.
      Default it to /nonexistent to prevent builds from writing to /home.
      This reverts what was done for #170762.  Closes: #441052
  * buildpackage:
    + only override HOME during dpkg-buildpackage invocation.

https://bugs.debian.org/441052 pbuilder: make $HOME non-writable, like in sbuild

So that is on purpose!

The issue comes from jobutils/bin/jsub

def homedir():
    """Get the homedir of the real user."""
    return pwd.getpwuid(os.getuid())[5]

pwd retrieves informations from the password database. So we can not override it by setting $HOME.

Change 558722 had a related patch set uploaded (by Hashar; owner: Hashar):
[labs/toollabs@master] jsub: recognize HOME environment variable

https://gerrit.wikimedia.org/r/558722

hashar reassigned this task from hashar to Bstorm.Sat, Jan 11, 9:34 PM

Arturo reviewed the change https://gerrit.wikimedia.org/r/#/c/labs/toollabs/+/558722/ and indicated the final strike is for @Bstorm :)

hashar triaged this task as Medium priority.Sat, Jan 11, 9:34 PM