Systemd timer ran the following command:
/usr/bin/scap stage-train -Dfull_image_build:True --yes auto
Its return value was 70 and emitted the following output:
03:00:14 Initializing stage-train auto mode
03:00:14 Retrieving train information...
03:00:16 Using version 1.41.0-wmf.12
03:00:16 ----------------------------
1. Starting: prep
03:00:16 Started scap prep 1.41.0-wmf.12
03:00:16 Copying patches from /srv/patches/1.41.0-wmf.11 to /srv/patches/1.41.0-wmf.12
03:00:16 Finished scap prep 1.41.0-wmf.12 (duration: 00m 00s)
03:00:16 Unhandled error:
Traceback (most recent call last):
File "/var/lib/scap/scap/lib/python3.7/site-packages/scap/plugins/prep.py", line 166, in main
self._prep_mw_branch(self.arguments.branch, logger)
File "/var/lib/scap/scap/lib/python3.7/site-packages/scap/plugins/prep.py", line 191, in _prep_mw_branch
self._setup_patches(branch)
File "/var/lib/scap/scap/lib/python3.7/site-packages/scap/plugins/prep.py", line 291, in _setup_patches
git.add_all(patch_base_dir, message='Scap prep for "{}"'.format(version))
File "/var/lib/scap/scap/lib/python3.7/site-packages/scap/git.py", line 238, in add_all
gitcmd("add", "--all", cwd=location)
File "/var/lib/scap/scap/lib/python3.7/site-packages/scap/runcmd.py", line 91, in gitcmd
return _runcmd(["git", subcommand] + list(args), **kwargs)
File "/var/lib/scap/scap/lib/python3.7/site-packages/scap/runcmd.py", line 78, in _runcmd
raise FailedCommand(argv, p.returncode, stdout, stderr)
scap.runcmd.FailedCommand: Command 'git add --all' failed with exit code 128;stdout:
stderr:
fatal: Unable to create 'srv/patches.git/index.lock': Permission denied
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/scap/scap/lib/python3.7/site-packages/scap/cli.py", line 530, in run
exit_status = app.main(app.extra_arguments)
File "/var/lib/scap/scap/lib/python3.7/site-packages/scap/plugins/prep.py", line 170, in main
history.log(self.new_history, self.config["history_log"])
File "/var/lib/scap/scap/lib/python3.7/site-packages/scap/history.py", line 54, in log
with utils.open_with_lock(path, 'a+') as f:
File "/usr/lib/python3.7/contextlib.py", line 112, in __enter__
return next(self.gen)
File "/var/lib/scap/scap/lib/python3.7/site-packages/scap/utils.py", line 995, in open_with_lock
with open(path, mode, *args, **kwargs) as f:
PermissionError: [Errno 13] Permission denied: '/srv/mediawiki-staging/scap/log/history.log'03:00:16 prep failed: <PermissionError> [Errno 13] Permission denied: '/srv/mediawiki-staging/scap/log/history.log' 03:00:16 stage-train failed: <CalledProcessError> Command '['/usr/bin/scap', 'prep', '1.41.0-wmf.12', '-D', 'full_image_build:True']' returned non-zero exit status 70.
Followup actions:
- fix-staging-perms hardcodes the deployment unix group, that should be feed by Puppet. That would prevent forgetting to update one of the occurrences of the group name
- https://gerrit.wikimedia.org/r/c/operations/puppet/+/927674 fix-staging-perms: set group name from Puppet
- The deployment_group variable in the Puppet classes default to wikidev which sounds potentially misleading. I'd got to remove the default and only rely on the deployment_group: deployment hiera variable. Beta-Cluster-Infrastructure will need something similar
- https://gerrit.wikimedia.org/r/c/operations/puppet/+/927675 scap3: stop defaulting deployment_group to 'wikidev'
- If /srv/patches has the set-group-id flag set (g+s), then newly created directory/files would all be owned by the deployment group which should reduce the need to use the fix-staging-perms script.
- https://gerrit.wikimedia.org/r/c/operations/puppet/+/927676 fix-staging-perms: set set-group-id on /srv/patches subdirs
- Files in /srv/patches should be owned by some other users than by uid of former employees
- This last step hasn't been accomplished. Maybe later we can change the nameless uid to nobody.