Page MenuHomePhabricator
Create Task
Maniphest T417767

scap error on Gerrit first setup
Closed, ResolvedPublic
Actions

Description

upon reimaging gerrit1003 (T417246: Reimage gerrit1003), I consistently got the following scap error:

Error: Execution of '/usr/bin/scap deploy-local --repo gerrit/gerrit -D log_json:False' returned 1: 11:53:30 Fetch from: http://deploy2002.codfw.wmnet/gerrit/gerrit/.git
11:53:30 Unhandled error:
Traceback (most recent call last):
  File "/var/lib/scap/scap/lib/python3.11/site-packages/scap/cli.py", line 827, in run
    exit_status = app.main(app.extra_arguments)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/scap/scap/lib/python3.11/site-packages/scap/deploy.py", line 174, in main
    getattr(self, stage)()
  File "/var/lib/scap/scap/lib/python3.11/site-packages/scap/deploy.py", line 352, in fetch
    git.lfs_install()
  File "/var/lib/scap/scap/lib/python3.11/site-packages/scap/git.py", line 180, in lfs_install
    gitcmd("lfs", *lfsargs)
  File "/var/lib/scap/scap/lib/python3.11/site-packages/scap/runcmd.py", line 88, in gitcmd
    return _runcmd(["git", subcommand] + list(args), **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/scap/scap/lib/python3.11/site-packages/scap/runcmd.py", line 75, in _runcmd
    raise FailedCommand(argv, p.returncode, stdout, stderr)
scap.runcmd.FailedCommand: Command 'git lfs install' failed with exit code 2;
stdout:
warning: error running /usr/lib/git-core/git 'config' '--includes' '--global' '--replace-all' 'filter.lfs.clean' 'git-lfs clean -- %f': 'fatal: $HOME not set' 'exit status 128'
Run `git lfs install --force` to reset Git configuration.

stderr:

11:53:30 deploy-local failed: <FailedCommand> Command 'git lfs install' failed with exit code 2;
stdout:
warning: error running /usr/lib/git-core/git 'config' '--includes' '--global' '--replace-all' 'filter.lfs.clean' 'git-lfs clean -- %f': 'fatal: $HOME not set' 'exit status 128'
Run `git lfs install --force` to reset Git configuration.

stderr:
 (scap version: 4.241.0) (duration: 00m 00s)

Error: /Stage[main]/Gerrit/Scap::Target[gerrit/gerrit]/Package[gerrit/gerrit]/ensure: change from 'absent' to 'present' failed: Execution of '/usr/bin/scap deploy-local --repo gerrit/gerrit -D log_json:False' returned 1: 11:53:30 Fetch from: http://deploy2002.codfw.wmnet/gerrit/gerrit/.git
11:53:30 Unhandled error:
Traceback (most recent call last):
  File "/var/lib/scap/scap/lib/python3.11/site-packages/scap/cli.py", line 827, in run
    exit_status = app.main(app.extra_arguments)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/scap/scap/lib/python3.11/site-packages/scap/deploy.py", line 174, in main
    getattr(self, stage)()
  File "/var/lib/scap/scap/lib/python3.11/site-packages/scap/deploy.py", line 352, in fetch
    git.lfs_install()
  File "/var/lib/scap/scap/lib/python3.11/site-packages/scap/git.py", line 180, in lfs_install
    gitcmd("lfs", *lfsargs)
  File "/var/lib/scap/scap/lib/python3.11/site-packages/scap/runcmd.py", line 88, in gitcmd
    return _runcmd(["git", subcommand] + list(args), **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/scap/scap/lib/python3.11/site-packages/scap/runcmd.py", line 75, in _runcmd
    raise FailedCommand(argv, p.returncode, stdout, stderr)
scap.runcmd.FailedCommand: Command 'git lfs install' failed with exit code 2;
stdout:
warning: error running /usr/lib/git-core/git 'config' '--includes' '--global' '--replace-all' 'filter.lfs.clean' 'git-lfs clean -- %f': 'fatal: $HOME not set' 'exit status 128'
Run `git lfs install --force` to reset Git configuration.

stderr:

11:53:30 deploy-local failed: <FailedCommand> Command 'git lfs install' failed with exit code 2;
stdout:
warning: error running /usr/lib/git-core/git 'config' '--includes' '--global' '--replace-all' 'filter.lfs.clean' 'git-lfs clean -- %f': 'fatal: $HOME not set' 'exit status 128'
Run `git lfs install --force` to reset Git configuration.

stderr:
 (scap version: 4.241.0) (duration: 00m 00s)

Which I guess triggers

Notice: /Stage[main]/Profile::Java/Java::Cacert[wmf:puppetca.pem]/Exec[java__cacert_wmf:puppetca.pem]/returns: keytool error: java.io.FileNotFoundException: /usr/share/ca-certificates/wikimedia/Puppet5_Internal_CA.crt (No such file or directory)
Error: '/usr/bin/keytool -import -trustcacerts -noprompt -cacerts     -file /usr/share/ca-certificates/wikimedia/Puppet5_Internal_CA.crt -storepass changeit -alias wmf:puppetca.pem
' returned 1 instead of one of [0]
Error: /Stage[main]/Profile::Java/Java::Cacert[wmf:puppetca.pem]/Exec[java__cacert_wmf:puppetca.pem]/returns: change from 'notrun' to ['0'] failed: '/usr/bin/keytool -import -trustcacerts -noprompt -cacerts     -file /usr/share/ca-certificates/wikimedia/Puppet5_Internal_CA.crt -storepass changeit -alias wmf:puppetca.pem
' returned 1 instead of one of [0] (corrective)

and the rest of puppet execution is Skipping because of failed dependencies

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
java: update puppet certificateoperations/puppetproduction+1 -1
scap3 install provider: Set env vars for deploy_user when running scapoperations/puppetproduction+23 -11
Customize query in gerrit

Related Objects
Search...

Event Timeline

ABran-WMF created this task.Feb 18 2026, 11:58 AM
ABran-WMF triaged this task as High priority.

setting high priority to mirror the parent task

ABran-WMF updated the task description. (Show Details)Feb 18 2026, 11:59 AM
ABran-WMF added a subtask: T417777: SystemdUnitFailed - apache2.service on gerrit1003:9100.Feb 18 2026, 1:25 PM
ABran-WMF closed subtask T417777: SystemdUnitFailed - apache2.service on gerrit1003:9100 as Resolved.
ABran-WMF mentioned this in T417777: SystemdUnitFailed - apache2.service on gerrit1003:9100.
ABran-WMF moved this task from Incoming to Awaiting Input on the collaboration-services board.Feb 18 2026, 1:27 PM
hashar edited projects, added Scap; removed Gerrit.Feb 18 2026, 5:17 PM
warning: error running /usr/lib/git-core/git 'config' '--includes' '--global' '--replace-all' 'filter.lfs.clean' 'git-lfs clean -- %f': 'fatal: $HOME not set' 'exit status 128'

Because a global git config is stored under the invoking user home directory:

git-config(1)
`--global`

For writing options: write to global ~/.gitconfig file rather than the repository .git/config, write to $XDG_CONFIG_HOME/git/config file if this file exists and the ~/.gitconfig file doesn’t.

Which sounds to me the scap deploy local script does not pass the invoking user environment variable (os.environ) or at least HOME/PATH/USER. Or it might be an issue with sudo not passing them if sudo is involved.

That sounds like a scap issue and others in my group are more qualified to investigate it. I am ping the team on Slack #developer-experience.

Dzahn added a comment.EditedFeb 18 2026, 5:45 PM

also see T257317#9851645 T363196#9851637

thcipriani subscribed.Feb 18 2026, 6:57 PM

Do you have the full log of the puppet run to see more context?

gerritbot added a comment.Feb 18 2026, 8:21 PM

Change #1240372 had a related patch set uploaded (by Ahmon Dancy; author: Ahmon Dancy):

[operations/puppet@production] scap3 install provider: Set HOME for deploy_user when running scap

https://gerrit.wikimedia.org/r/1240372

gerritbot added a project: Patch-For-Review.Feb 18 2026, 8:21 PM
thcipriani added a subscriber: dancy.EditedFeb 18 2026, 9:14 PM

@dancy found a reason $HOME would not be set in the initial puppet run: the Puppet Provider's execution method deletes some env vars

gerritbot added a comment.Feb 19 2026, 12:39 AM

Change #1240372 merged by Dzahn:

[operations/puppet@production] scap3 install provider: Set env vars for deploy_user when running scap

https://gerrit.wikimedia.org/r/1240372

Maintenance_bot removed a project: Patch-For-Review.Feb 19 2026, 1:31 AM
ABran-WMF added a comment.Feb 19 2026, 6:49 AM

thanks @thcipriani @hashar @Dzahn @dancy for the fix!

I still have some issues after the Nth puppet run, I've logged all puppet-agent-run output in files in /home/arnaudb/:

arnaudb@gerrit1003:~ $ ls -l *.log
-rw-r--r-- 1 arnaudb wikidev 94744 Feb 19 06:36 puppet_run_1.log
-rw-r--r-- 1 arnaudb wikidev 36646 Feb 19 06:39 puppet_run_2.log
-rw-r--r-- 1 arnaudb wikidev 31609 Feb 19 06:41 puppet_run_3.log
-rw-r--r-- 1 arnaudb wikidev 31609 Feb 19 06:42 puppet_run_4.log

the last blocker seems to come from the puppet5 CA:

Notice: /Stage[main]/Profile::Java/Java::Cacert[wmf:puppetca.pem]/Exec[java__cacert_wmf:puppetca.pem]/returns: keytool error: java.io.FileNotFoundException: /usr/share/ca-certificates/wikimedia/Puppet5_Internal_CA.crt (No such file 
or directory)                                                                                                                                                                                                                           
Error: '/usr/bin/keytool -import -trustcacerts -noprompt -cacerts     -file /usr/share/ca-certificates/wikimedia/Puppet5_Internal_CA.crt -storepass changeit -alias wmf:puppetca.pem                                                    
' returned 1 instead of one of [0]                                                                                                                                                                                                      
Error: /Stage[main]/Profile::Java/Java::Cacert[wmf:puppetca.pem]/Exec[java__cacert_wmf:puppetca.pem]/returns: change from 'notrun' to ['0'] failed: '/usr/bin/keytool -import -trustcacerts -noprompt -cacerts     -file /usr/share/ca-c
ertificates/wikimedia/Puppet5_Internal_CA.crt -storepass changeit -alias wmf:puppetca.pem                                                                                                                                               
' returned 1 instead of one of [0] (corrective)                                                                                                                                                                                         
Notice: /Stage[main]/Profile::Gerrit/Motd::Script[replica warning]/File[/etc/update-motd.d/01-replica-warning]: Dependency Exec[java__cacert_wmf:puppetca.pem] has failures: true

it comes back consistently after the 2nd puppet-agent run:

arnaudb@gerrit1003:~ $ diff puppet_run_4.log puppet_run_3.log
184c184
< Notice: Applied catalog in 24.23 seconds
---
> Notice: Applied catalog in 24.47 seconds
gerritbot added a comment.Feb 19 2026, 7:08 AM

Change #1240477 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/puppet@production] java: update puppet certificate

https://gerrit.wikimedia.org/r/1240477

gerritbot added a project: Patch-For-Review.Feb 19 2026, 7:08 AM
ABran-WMF closed this task as Resolved.Feb 19 2026, 9:28 AM
ABran-WMF assigned this task to dancy.

closing the task as the initial issue has been fixed

gerritbot added a comment.Feb 19 2026, 1:01 PM

Change #1240477 merged by Arnaudb:

[operations/puppet@production] java: update puppet certificate

https://gerrit.wikimedia.org/r/1240477

Maintenance_bot removed a project: Patch-For-Review.Feb 19 2026, 1:31 PM
Dzahn added a comment.Feb 19 2026, 3:13 PM

the last blocker seems to come from the puppet5 CA

This sounds related to the recent work by infra foundations to remove puppet5 / puppetmasters.

ABran-WMF added subscribers: elukey, MoritzMuehlenhoff.Feb 20 2026, 7:01 AM
In T417767#11632455, @Dzahn wrote:

the last blocker seems to come from the puppet5 CA

This sounds related to the recent work by infra foundations to remove puppet5 / puppetmasters.

yes indeed! we swapped the certificates based on @MoritzMuehlenhoff and @elukey recommendations

Dzahn awarded a token.Feb 20 2026, 2:53 PM
thcipriani added a project: Essential-Work.Feb 21 2026, 12:00 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits