Page MenuHomePhabricator

sbuild isn't behaving well in tools
Closed, ResolvedPublic

Description

Today, we noticed that the sbuild toolchain on tools-package-builder-02 seems to have developed some kind of bug. It cannot chown the chroot to <user>:sbuild. This is strange because the only place that doesn't work when done directly by hand is on an NFS dir, and the build dir should be on the instance disk.

For testing and comparison, I built a new tools-package-builder-03 using buster (and no NFS at all--so changing at least 10 variables in terms of build toolchain), and that seems to be happier except that the autotest/m4-based testsuite attached to toollabs completely fails. The latter may have been something silly I did as well, but you never know. I have not tried rebuilding the stretch version, partly because it was showing some odd issues like the pins no longer being valid (partly because the upstream module pins everything to stretch-backports).

Event Timeline

Bstorm triaged this task as Medium priority.Feb 5 2021, 12:56 AM
Bstorm created this task.

I've deployed the new version of toollabs (jobutils/miscutils) to toolsbeta using the debuild whatnot for now.

The newer sbuild deployment (buster based) works well, but still shows the following:

  • when building a package via our script, it eventually fails in the final build step (dpkg-deb call) with a permissions error
  • I contacted some debian people and they think this is a bug. The schroot used by sbuild 'embeds' the uid/gid from the base system, but the base system uses LDAP, so there is something weird there
  • I was asked to submit a bug report, but when I was about to do it I noticed there are newer versions of sbuild in the debian archive (testing bullseye and also sid).

So next step would be for me to test the newer stack from testing bullseye (or even sid) and see if that solves the issue. If not, then submit a formal bug report to debian.

aborrero raised the priority of this task from Medium to High.
aborrero moved this task from Soon! to Doing on the cloud-services-team (Kanban) board.

Mentioned in SAL (#wikimedia-cloud) [2021-09-10T07:54:17Z] <arturo> created bullseye VM tools-package-builder-04 (T273942)

The sbuild version in bullseye shows the same error:

dpkg-deb: building package 'toollabs-webservice' in '../toollabs-webservice_0.76_all.deb'.
dpkg-deb: error: unable to create '../toollabs-webservice_0.76_all.deb': Permission denied

Change 720284 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] toolforge: wmcs-package-build.py: use sudo in the sbuild call

https://gerrit.wikimedia.org/r/720284

Change 720284 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] toolforge: wmcs-package-build.py: use sudo in the sbuild call

https://gerrit.wikimedia.org/r/720284

With this patch I can build a package!

arturo@endurance:~/git/wmf/operations/puppet$ modules/toolforge/files/wmcs-package-build.py --git-repo https://gerrit.wikimedia.org/r/operations/software/tools-webservice --build-dist stretch
[..]
+------------------------------------------------------------------------------+
| Summary                                                                      |
+------------------------------------------------------------------------------+

Build Architecture: amd64
Build Type: binary
Build-Space: 1288
Build-Time: 6
Distribution: stretch
Host Architecture: amd64
Install-Time: 45
Job: /tmp/wmcs-package-build-rjfpw/toollabs-webservice_0.76.dsc
Lintian: warn
Machine Architecture: amd64
Package: toollabs-webservice
Package-Time: 55
Source-Version: 0.76
Space: 1288
Status: successful
Version: 0.76
--------------------------------------------------------------------------------
Finished at 2021-09-10T11:49:46Z
Build needed 00:00:55, 1288k disk space
INFO: wmcs-package-build.py: Not copying .deb files because --aptly-dist was not set
INFO: wmcs-package-build.py: Not running aptly stage because --aptly-dist was not set
INFO: wmcs-package-build.py: Not running backup stage because --aptly-dist was not set

And in the newer VM:

arturo@endurance:~/git/wmf/operations/puppet$ modules/toolforge/files/wmcs-package-build.py --git-repo https://gerrit.wikimedia.org/r/operations/software/tools-webservice --build-dist stretch --build-host tools-package-builder-04.tools.eqiad1.wikimedia.cloud
[..]
+------------------------------------------------------------------------------+
| Summary                                                                      |
+------------------------------------------------------------------------------+

Build Architecture: amd64
Build Type: binary
Build-Space: 1300
Build-Time: 5
Distribution: stretch
Host Architecture: amd64
Install-Time: 42
Job: /tmp/wmcs-package-build-veqpg/toollabs-webservice_0.76.dsc
Lintian: warn
Machine Architecture: amd64
Package: toollabs-webservice
Package-Time: 49
Source-Version: 0.76
Space: 1300
Status: successful
Version: 0.76
--------------------------------------------------------------------------------
Finished at 2021-09-10T11:42:02Z
Build needed 00:00:49, 1300k disk space
INFO: wmcs-package-build.py: Not copying .deb files because --aptly-dist was not set
INFO: wmcs-package-build.py: Not running aptly stage because --aptly-dist was not set
INFO: wmcs-package-build.py: Not running backup stage because --aptly-dist was not set

Change 720284 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] toolforge: wmcs-package-build.py: use sudo in the sbuild call

https://gerrit.wikimedia.org/r/720284

I think we can live with our small sudo patch until they find a proper solution upstream. Closing this now (finally!!)