scap fails to bootstrap itself
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Dzahn
	Nov 28 2023, 9:58 PM

Description

On a new machine in cloud VPS that uses scap (via scap::target in the phabricator role), scap does not get installed.

`-bash: scap: command not found
`

Notice: /Stage[main]/Scap/Exec[bootstrap-scap-target]/returns: rsync: change_dir "/scap-wheels" (in scap-install-staging) failed: No such file or directory (2)
Error: '/usr/local/bin/bootstrap-scap-target.sh deploy-1004.devtools.eqiad1.wikimedia.cloud /var/lib/scap' returned 23 instead of one of [0]
Notice: /Stage[main]/Scap/Exec[bootstrap-scap-target]/returns: rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1819) [generator=3.2.3]
Error: /Stage[main]/Scap/Exec[bootstrap-scap-target]/returns: change from 'notrun' to ['0'] failed: '/usr/local/bin/bootstrap-scap-target.sh deploy-1004.devtools.eqiad1.wikimedia.cloud /var/lib/scap' returned 23 instead of one of [0] (corrective)
Notice: /Stage[main]/Scap/File[/usr/bin/scap]: Dependency Exec[bootstrap-scap-target] has failures: true
Warning: /Stage[main]/Scap/File[/usr/bin/scap]: Skipping because of failed dependencies
Error: /Stage[main]/Phabricator::Config/Scap::Target[phabricator/deployment]/Package[phabricator/deployment]: Provider scap3 is not functional on this host

Related Objects

Mentioned In: T334519: upgrade phab (phorge) hosts to bullseye

Event Timeline

Dzahn created this task.Nov 28 2023, 9:58 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 28 2023, 9:58 PM

Dzahn added a project: Scap.Nov 28 2023, 9:59 PM

scap::target does an "include scap" and that has "enable_bootstrapping = true".

The /etc/scap.cfg and /usr/local/bin/bootstrap-scap-target.sh gets installed.

Inside the bootstrap script there is an rsync command to rsync from the deployment server.

This fails because "scap-wheels" does not exist there.

root@deploy-1004:/var/lib/scap# file scap-wheels
scap-wheels: cannot open `scap-wheels' (No such file or directory)

Dzahn mentioned this in T334519: upgrade phab (phorge) hosts to bullseye.Nov 28 2023, 10:07 PM

The scap-wheels directory exists on a production deployment server but was only created on September 27. Does not exist in deployment server in cloud project.

So seems like things were changed on that date but only in production.

dancy added a subscriber: jnuche.Nov 28 2023, 10:33 PM

dancy subscribed.

Should I simply copy the scap-wheels directory from production deployment server over to cloud deployment server?

Or do we need to create a new deployment server from scratch and then it would appear?

I hate that you don't get a notification when you get subscribed to something. I didn't get a message until @Dzahn posted his last comment.

My guess is that the scap::master puppet class is not getting applied to your deployment server and that's causing scap to not get initialized there. That should get fixed if you applied the deployment server profile to the host.

Another option can be to checkout the scap code on the deployment server and then run the bootstrap script yourself. That will install scap locally and create the missing scap-wheels dir.

Thank you @jnuche!

The role::deployment_server is applied on the deployment server. (just like production deployment servers).

This includes profile::mediawiki::deployment::server and that uses class { '::scap::master':.

So it should work but somehow doesn't.

I will now try the other option you mentioned.

I noticed puppet is also broken on that deployment_server:

Function lookup() did not find a value for the name 'deployment_group' (file: /etc/puppet/modules/profile/manifests/mediawiki/deployment/server.pp, line: 7)

But the last puppet run was only (359 minutes ago) so that looks an unrelated additional problem that just happened.

The puppet breakage seems to come from https://gerrit.wikimedia.org/r/c/operations/puppet/+/927675

In scap::master I found the code that should run the bootstrap script:

exec { 'bootstrap_scap_master':
  command => "${scap_source_path}/bin/bootstrap_scap_master.sh scap ${scap_source_path}",
  creates => "${scap_home}/scap/bin/scap",

scap_source_path defaults to '/srv/deployment/scap'. So I manually ran:

/srv/deployment/scap/bin/bootstrap_scap_master.sh scap /srv/deployment/scap/

This is what happened:

remote: Enumerating objects: 1259, done.
remote: Counting objects: 100% (280/280), done.
remote: Compressing objects: 100% (173/173), done.
remote: Total 1259 (delta 143), reused 157 (delta 105), pack-reused 979
Receiving objects: 100% (1259/1259), 765.52 KiB | 4.81 MiB/s, done.
Resolving deltas: 100% (665/665), completed with 14 local objects.
From https://gitlab.wikimedia.org/repos/releng/scap
 + 83dda29...75ac24e master                    -> origin/master  (forced update)
 * [new branch]      abandoned-depends-on      -> origin/abandoned-depends-on
 * [new branch]      backport-tests            -> origin/backport-tests
 * [new branch]      option-a                  -> origin/option-a
 * [new branch]      option-b                  -> origin/option-b
 * [new branch]      review/dancy/refactor-publish-docs -> origin/review/dancy/refactor-publish-docs
 * [new branch]      review/dancy/refactor-supported-distros -> origin/review/dancy/refactor-supported-distros
 * [new branch]      review/dancy/test-kokkuri -> origin/review/dancy/test-kokkuri
 * [new branch]      review/dancy/test-staging -> origin/review/dancy/test-staging
 * [new branch]      review/dancy/testing      -> origin/review/dancy/testing
 * [new branch]      test-staging              -> origin/test-staging
 * [new branch]      timeout                   -> origin/timeout
 * [new tag]         4.64.0                    -> 4.64.0
 * [new tag]         4.28.0                    -> 4.28.0
 * [new tag]         4.28.1                    -> 4.28.1
 * [new tag]         4.28.2                    -> 4.28.2
 * [new tag]         4.29.0                    -> 4.29.0
 * [new tag]         4.29.1                    -> 4.29.1
 * [new tag]         4.29.2                    -> 4.29.2
 * [new tag]         4.29.3                    -> 4.29.3
 * [new tag]         4.30.0                    -> 4.30.0
 * [new tag]         4.30.1                    -> 4.30.1
 * [new tag]         4.30.2                    -> 4.30.2
 * [new tag]         4.30.3                    -> 4.30.3
 * [new tag]         4.30.3-1                  -> 4.30.3-1
 * [new tag]         4.31.0                    -> 4.31.0
 * [new tag]         4.31.1                    -> 4.31.1
 * [new tag]         4.32.0                    -> 4.32.0
 * [new tag]         4.33.0                    -> 4.33.0
 * [new tag]         4.33.1                    -> 4.33.1
 * [new tag]         4.34.0                    -> 4.34.0
 * [new tag]         4.35.0                    -> 4.35.0
 * [new tag]         4.36.0                    -> 4.36.0
 * [new tag]         4.37.0                    -> 4.37.0
 * [new tag]         4.38.0                    -> 4.38.0
 * [new tag]         4.39.0                    -> 4.39.0
 * [new tag]         4.40.0                    -> 4.40.0
 * [new tag]         4.41.0                    -> 4.41.0
 * [new tag]         4.42.0                    -> 4.42.0
 * [new tag]         4.43.0                    -> 4.43.0
 * [new tag]         4.44.0                    -> 4.44.0
 * [new tag]         4.45.0                    -> 4.45.0
 * [new tag]         4.46.0                    -> 4.46.0
 * [new tag]         4.47.0                    -> 4.47.0
 * [new tag]         4.47.1                    -> 4.47.1
 * [new tag]         4.48.0                    -> 4.48.0
 * [new tag]         4.49.0                    -> 4.49.0
 * [new tag]         4.50.0                    -> 4.50.0
 * [new tag]         4.51.0                    -> 4.51.0
 * [new tag]         4.52.0                    -> 4.52.0
 * [new tag]         4.52.1                    -> 4.52.1
 * [new tag]         4.52.2                    -> 4.52.2
 * [new tag]         4.52.3                    -> 4.52.3
 * [new tag]         4.53.0                    -> 4.53.0
 * [new tag]         4.54.0                    -> 4.54.0
 * [new tag]         4.55.0                    -> 4.55.0
 * [new tag]         4.56.0                    -> 4.56.0
 * [new tag]         4.57.0                    -> 4.57.0
 * [new tag]         4.58.0                    -> 4.58.0
 * [new tag]         4.59.0                    -> 4.59.0
 * [new tag]         4.60.0                    -> 4.60.0
 * [new tag]         4.61.0                    -> 4.61.0
 * [new tag]         4.61.1                    -> 4.61.1
 * [new tag]         4.62.0                    -> 4.62.0
 * [new tag]         4.63.0                    -> 4.63.0
ERROR: Specified source path "." does not seem to be a scap repository. Aborting


dzahn@deploy-1004:~$ /srv/deployment/scap/bin/bootstrap_scap_master.sh scap /srv/deployment/scap
ERROR: Specified source path "." does not seem to be a scap repository. Aborting

$scap_home = '/var/lib/scap'
..
 creates => "${scap_home}/scap/bin/scap",

file /var/lib/scap/scap/bin/scap
/var/lib/scap/scap/bin/scap: Python script, UTF-8 Unicode text executable

 file /var/lib/scap/scap-wheels
/var/lib/scap/scap-wheels: cannot open `/var/lib/scap/scap-wheels' (No such file or directory)

dzahn@deploy-1004:/srv/deployment/scap$ git status
On branch master
Your branch and 'origin/master' have diverged,
and have 16 and 221 different commits each, respectively.
  (use "git pull" to merge the remote branch into yours)

Untracked files:
  (use "git add <file>..." to include in what will be committed)

	scap/log/

nothing added to commit but untracked files present (use "git add" to track)
(reverse-i-search)`': ^C
dzahn@deploy-1004:/srv/deployment/scap$ git pull
Auto-merging tox.ini
Auto-merging tests/scap/test_utils.py
Auto-merging tests/scap/test_tasks.py
Auto-merging tests/scap/test_targets.py
Auto-merging tests/scap/test_main.py
Auto-merging tests/scap/test_log.py
Auto-merging tests/scap/test_lock.py
Auto-merging tests/scap/test_lint.py
Auto-merging tests/scap/test_config.py
Auto-merging tests/scap/test_cli.py
Auto-merging tests/scap/test_checks.py
Auto-merging test-requirements.txt
CONFLICT (content): Merge conflict in test-requirements.txt
Auto-merging scap/version.py
CONFLICT (content): Merge conflict in scap/version.py
Auto-merging scap/utils.py
Auto-merging scap/terminal.py
Auto-merging scap/tasks.py
Auto-merging scap/targets.py
Auto-merging scap/ssh.py
Auto-merging scap/plugins/test.py
Auto-merging scap/plugins/say.py
Auto-merging scap/plugins/patch.py
Auto-merging scap/plugins/gerrit.py
Auto-merging scap/plugins/backport.py
CONFLICT (content): Merge conflict in scap/plugins/backport.py
Auto-merging scap/plugins/__init__.py
Auto-merging scap/php_fpm.py
Auto-merging scap/main.py
Auto-merging scap/log.py
Auto-merging scap/lock.py
CONFLICT (content): Merge conflict in scap/lock.py
Auto-merging scap/lint.py
Auto-merging scap/git.py
Auto-merging scap/deploy.py
CONFLICT (content): Merge conflict in scap/deploy.py
Auto-merging scap/config.py
CONFLICT (content): Merge conflict in scap/config.py
Auto-merging scap/cli.py
Auto-merging scap/checks.py
CONFLICT (content): Merge conflict in scap/checks.py
Auto-merging scap/cdblib.py
Auto-merging scap/arg.py
Auto-merging scap/__init__.py
Auto-merging release-scripts/perform-release
CONFLICT (content): Merge conflict in release-scripts/perform-release
Auto-merging changelog
CONFLICT (content): Merge conflict in changelog
Auto-merging bin/install_local_version.sh
CONFLICT (content): Merge conflict in bin/install_local_version.sh
Removing .pipeline/config.yaml
Auto-merging .pipeline/blubber.yaml
CONFLICT (content): Merge conflict in .pipeline/blubber.yaml
Automatic merge failed; fix conflicts and then commit the result.

dzahn@deploy-1004:/srv/deployment/scap$ sudo /srv/deployment/scap/bin/bootstrap_scap_master.sh scap /srv/deployment/scap
.pipeline/blubber.yaml: needs merge
bin/install_local_version.sh: needs merge
changelog: needs merge
release-scripts/perform-release: needs merge
scap/checks.py: needs merge
scap/config.py: needs merge
scap/deploy.py: needs merge
scap/lock.py: needs merge
scap/plugins/backport.py: needs merge
scap/version.py: needs merge
test-requirements.txt: needs merge
error: you need to resolve your current index first

git reset origin
..

git pull
...

sudo /srv/deployment/scap/bin/bootstrap_scap_master.sh scap /srv/deployment/scap
M	.pipeline/blubber.yaml
M	bin/install_local_version.sh
M	changelog
M	release-scripts/perform-release
M	scap/checks.py
M	scap/config.py
M	scap/deploy.py
M	scap/lock.py
M	scap/plugins/backport.py
M	scap/plugins/gerrit.py
M	scap/version.py
M	test-requirements.txt
HEAD is now at 75ac24e Release 4.64.0-1
bin/install_local_version.sh: line 152: syntax error near unexpected token `<<<'
Switched to branch 'master'

bin/install_local_version.sh: line 152: syntax error near unexpected token `<<<'

git reset origin master
..
git reset --hard origin/master
..
git checkout master
..
sudo /srv/deployment/scap/bin/bootstrap_scap_master.sh scap /srv/deployment/scap
HEAD is now at 75ac24e Release 4.64.0-1
INFO: Scap distribution successfully extracted at /var/lib/scap/scap-wheels/buster
INFO: Scap distribution successfully extracted at /var/lib/scap/scap-wheels/bullseye
INFO: Scap distribution successfully extracted at /var/lib/scap/scap-wheels/bookworm
Processing /var/lib/scap/scap-wheels/buster/certifi-2022.9.24-py3-none-any.whl
Processing /var/lib/scap/scap-wheels/buster/chardet-5.2.0-py3-none-any.whl
Processing /var/lib/scap/scap-wheels/buster/charset_normalizer-2.1.1-py3-none-any.whl
Processing /var/lib/scap/scap-wheels/buster/idna-2.8-py2.py3-none-any.whl
Processing /var/lib/scap/scap-wheels/buster/importlib_metadata-6.7.0-py3-none-any.whl
Processing /var/lib/scap/scap-wheels/buster/Jinja2-2.10.2-py2.py3-none-any.whl
Processing /var/lib/scap/scap-wheels/buster/MarkupSafe-1.1.0-cp37-cp37m-manylinux1_x86_64.whl
Processing /var/lib/scap/scap-wheels/buster/packaging-23.1-py3-none-any.whl
Processing /var/lib/scap/scap-wheels/buster/prettytable-3.7.0-py3-none-any.whl
Processing /var/lib/scap/scap-wheels/buster/Pygments-2.3.1-py2.py3-none-any.whl
Processing /var/lib/scap/scap-wheels/buster/pyparsing-3.0.9-py3-none-any.whl
Processing /var/lib/scap/scap-wheels/buster/PyYAML-5.1-cp37-cp37m-linux_x86_64.whl
Processing /var/lib/scap/scap-wheels/buster/requests-2.31.0-py3-none-any.whl
Processing /var/lib/scap/scap-wheels/buster/Scap-4.64.0.post1-py37-none-any.whl
Processing /var/lib/scap/scap-wheels/buster/setuptools-68.0.0-py3-none-any.whl
Processing /var/lib/scap/scap-wheels/buster/typing_extensions-4.7.1-py3-none-any.whl
Processing /var/lib/scap/scap-wheels/buster/urllib3-2.0.4-py3-none-any.whl
Processing /var/lib/scap/scap-wheels/buster/wcwidth-0.2.5-py2.py3-none-any.whl
Processing /var/lib/scap/scap-wheels/buster/zipp-3.15.0-py3-none-any.whl
Installing collected packages: certifi, chardet, charset-normalizer, idna, importlib-metadata, Jinja2, MarkupSafe, packaging, prettytable, Pygments, pyparsing, PyYAML, requests, Scap, setuptools, typing-extensions, urllib3, wcwidth, zipp
  Found existing installation: setuptools 40.8.0
    Uninstalling setuptools-40.8.0:
      Successfully uninstalled setuptools-40.8.0
Successfully installed Jinja2-2.10.2 MarkupSafe-1.1.0 PyYAML-5.1 Pygments-2.3.1 Scap-4.64.0.post1 certifi-2022.9.24 chardet-5.2.0 charset-normalizer-2.1.1 idna-2.8 importlib-metadata-6.7.0 packaging-23.1 prettytable-3.7.0 pyparsing-3.0.9 requests-2.31.0 setuptools-68.0.0 typing-extensions-4.7.1 urllib3-2.0.4 wcwidth-0.2.5 zipp-3.15.0

INFO: Scap "4.64.0" for buster successfully installed at /var/lib/scap/scap
Switched to branch 'master'
..

file /var/lib/scap/scap-wheels
/var/lib/scap/scap-wheels: directory

^ :)

Dzahn added a project: collaboration-services.Nov 29 2023, 8:12 PM

Dzahn closed this task as Resolved.Nov 29 2023, 8:15 PM

Dzahn claimed this task.

summary: The scap deployment git repo was in a messy state. It is unknown why specifically it was in that state but:

/srv/deployment/scap$ git status
On branch master
Your branch and 'origin/master' have diverged,
and have 16 and 221 different commits each, respectively.

Doing a git reset --hard and git pulling, then manually running the scap bootstrap script, then running puppet on the host to be deployed to fixed things afaict.

That's odd, I wonder what those 16 local commits were. At any rate, glad you got it fixed!

scap fails to bootstrap itselfClosed, ResolvedPublicActions

Description

Related Objects

Event Timeline

scap fails to bootstrap itself
Closed, ResolvedPublic
Actions