Puppet runs are failing on deployment-jobrunner01 because the git clone of /srv/deployment/jobrunner/jobrunner is failing with the error message "atal: repository 'http://deployment-tin.eqiad.wmflabs/jobrunner/jobrunner/.git/' not found"
On deployment-jobrunner01.deployment-prep.eqiad.wmflabs:
# git -C /srv/deployment/jobrunner/jobrunner ls-remote fatal: repository 'http://deployment-tin.eqiad.wmflabs/jobrunner/jobrunner/.git/' not found
I would blame Trebuchet . @thcipriani would you mind having a look ? That is a follow up to the bastion -> tin switch.
With puppet agent -tv --debug:
Debug: Exec[ensure_php_hhvm](provider=posix): Executing check '/usr/local/sbin/grain-ensure contains php hhvm' Debug: Executing '/usr/local/sbin/grain-ensure contains php hhvm' Debug: Exec[ensure_labsproject_deployment-prep](provider=posix): Executing check '/usr/local/sbin/grain-ensure contains labsproject deployment-prep' Debug: Executing '/usr/local/sbin/grain-ensure contains labsproject deployment-prep' Debug: Executing '/sbin/status salt-minion' Debug: Exec[ensure_trebuchet_master_deployment-tin.deployment-prep.eqiad.wmflabs](provider=posix): Executing check '/usr/local/sbin/grain-ensure contains trebuchet_master deployment-tin.deployment-prep.eqiad.wmflabs' Debug: Executing '/usr/local/sbin/grain-ensure contains trebuchet_master deployment-tin.deployment-prep.eqiad.wmflabs' Debug: Prefetching trebuchet resources for package Debug: Executing '/usr/bin/salt-call --log-level=quiet --out=json --local service.status salt-minion' Debug: Executing '/usr/bin/salt-call --log-level=quiet --out=json --local grains.get deployment_target' Debug: Executing '/usr/bin/git --git-dir /srv/deployment/jobrunner/jobrunner/.git rev-parse HEAD' Debug: Executing '/usr/bin/salt-call --log-level=quiet --out=json --local grains.get trebuchet_master' Error: Could not get latest version: 404 Not Found Error: /Stage[main]/Mediawiki::Jobrunner/Package[jobrunner]/ensure: change from 9e42f5b0ae056533b8ca27a67ed98b9219c67dbc to latest failed: Could not get latest version: 404 Not Found Notice: /Stage[main]/Mediawiki::Jobrunner/Service[jobrunner]: Dependency Package[jobrunner] has failures: true Warning: /Stage[main]/Mediawiki::Jobrunner/Service[jobrunner]: Skipping because of failed dependencies
Output of salt commands:
# /usr/bin/salt-call --log-level=quiet --out=json --local service.status salt-minion
{
"local": true
}# /usr/bin/salt-call --log-level=quiet --out=json --local grains.get deployment_target
{
"local": [
"scap/scap",
"jobrunner/jobrunner"
]
}# /usr/bin/git --git-dir /srv/deployment/jobrunner/jobrunner/.git rev-parse HEAD 9e42f5b0ae056533b8ca27a67ed98b9219c67dbc
# /usr/bin/salt-call --log-level=quiet --out=json --local grains.get trebuchet_master
{
"local": "deployment-tin.deployment-prep.eqiad.wmflabs"
}No clue about the "Could not get latest version: 404 Not Found"
I did a git deploy from tin:
$ git deploy start Deployment started. $ git deploy sync Repo: jobrunner/jobrunner Tag: jobrunner/jobrunner-sync-20160225-214251 2/2 minions completed fetch Continue? ([d]etailed/[C]oncise report,[y]es,[n]o,[r]etry): y Repo: jobrunner/jobrunner Tag: jobrunner/jobrunner-sync-20160225-214251 2/2 minions completed checkout Continue? ([d]etailed/[C]oncise report,[y]es,[n]o,[r]etry): d Repo: jobrunner/jobrunner Tag: jobrunner/jobrunner-sync-20160225-214251 2/2 minions completed checkout Details: Continue? ([d]etailed/[C]oncise report,[y]es,[n]o,[r]etry): y Deployment finished.
I ran puppet twice on deployment-jobrunner01 and it seems happy now. Not sure about the status of jobrunner service though.
Mentioned in SAL [2016-02-25T21:49:14Z] <hashar> beta did a git-deploy of jobrunner/jobrunner hoping to fix puppet run on deployment-jobrunner01 and apparently it did! T126846
puppet run. Not sure how the git clone issue got fixed, maybe Tyler helped on that front somehow when he fixed up a few puppet issues we had after the switch from bastion to tin.
The scap deploy has fixed the remaining issues I have found. Puppet is all happy.
Thank you @bd808 for the report