Scap not working in Beta
Closed, ResolvedPublic

Description

Running Scap on deployment-tin in /srv/deployment/mathoid/deploy yields:

$ scap deploy
12:26:32 Started deploy [mathoid/deploy@c9957ce] (beta)
12:26:32 Deploying Rev: HEAD = c9957ce9ddd889f66eabf41ac19b57d9b63298b4
12:26:32 Finished deploy [mathoid/deploy@c9957ce] (beta) (duration: 00m 00s)
12:26:32 Unhandled error:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/scap/cli.py", line 330, in run
    exit_status = app.main(app.extra_arguments)
  File "/usr/lib/python2.7/dist-packages/scap/deploy.py", line 705, in main
    git.tag_repo(self.deploy_info, location=self.context.root)
  File "/usr/lib/python2.7/dist-packages/scap/git.py", line 454, in tag_repo
    git('submodule', 'foreach', "git %s" % submodule_cmd)
  File "/usr/lib/python2.7/dist-packages/scap/sh.py", line 1428, in __call__
    return RunningCommand(cmd, call_args, stdin, stdout, stderr)
  File "/usr/lib/python2.7/dist-packages/scap/sh.py", line 775, in __init__
    self.wait()
  File "/usr/lib/python2.7/dist-packages/scap/sh.py", line 793, in wait
    self.handle_command_exit_code(exit_code)
  File "/usr/lib/python2.7/dist-packages/scap/sh.py", line 816, in handle_command_exit_code
    raise exc
ErrorReturnCode_1: 

  RAN: /usr/bin/git submodule foreach git tag -fa -m user mobrovac -m timestamp 2018-01-04T12:26:32.188215 -- scap/sync/2018-01-04/0003 c9957ce9ddd889f66eabf41ac19b57d9b63298b4

  STDOUT:
Entering 'src'


  STDERR:
fatal: too many params
Stopping at 'src'; script returned non-zero status.

12:26:32 deploy failed: <ErrorReturnCode_1> 

  RAN: /usr/bin/git submodule foreach git tag -fa -m user mobrovac -m timestamp 2018-01-04T12:26:32.188215 -- scap/sync/2018-01-04/0003 c9957ce9ddd889f66eabf41ac19b57d9b63298b4

  STDOUT:
Entering 'src'


  STDERR:
fatal: too many params
Stopping at 'src'; script returned non-zero status.

The same problem is present in other deployment repos on deployment-tin.

mobrovac created this task.Jan 4 2018, 12:29 PM
mobrovac triaged this task as High priority.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 4 2018, 12:29 PM

Thnx for the fix. Please make it a thing to test Scap in Beta when a new version lands there.

Perhaps we should just skip the auto-upgrade and install the testing packages in an alternate location ... but that would likely lead to less testing, not more.. hmm

Perhaps we should just skip the auto-upgrade and install the testing packages in an alternate location ... but that would likely lead to less testing, not more.. hmm

Auto-upgrade is a useful feature, imho, provided the normal deployment process is ensured not to break after a change lands.

mobrovac reopened this task as Open.Jan 5 2018, 6:53 PM

This is still an issue even though the params are now quoted:

18:50:25 Started deploy [mathoid/deploy@c9957ce] (beta)
18:50:25 Deploying Rev: HEAD = c9957ce9ddd889f66eabf41ac19b57d9b63298b4
18:50:25 Prepare config deploy
18:50:25 Config deploy file: /srv/deployment/mathoid/deploy/scap/config-files.yaml
18:50:25 Update DEPLOY_HEAD
18:50:25 Creating /srv/deployment/mathoid/deploy/.git/DEPLOY_HEAD
18:50:25 Finished deploy [mathoid/deploy@c9957ce] (beta) (duration: 00m 00s)
18:50:25 Unhandled error:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/scap/cli.py", line 330, in run
    exit_status = app.main(app.extra_arguments)
  File "/usr/lib/python2.7/dist-packages/scap/deploy.py", line 705, in main
    git.tag_repo(self.deploy_info, location=self.context.root)
  File "/usr/lib/python2.7/dist-packages/scap/git.py", line 455, in tag_repo
    git('submodule', 'foreach', "git {0}".format(" ".join(args)))
  File "/usr/lib/python2.7/dist-packages/scap/sh.py", line 1428, in __call__
    return RunningCommand(cmd, call_args, stdin, stdout, stderr)
  File "/usr/lib/python2.7/dist-packages/scap/sh.py", line 775, in __init__
    self.wait()
  File "/usr/lib/python2.7/dist-packages/scap/sh.py", line 793, in wait
    self.handle_command_exit_code(exit_code)
  File "/usr/lib/python2.7/dist-packages/scap/sh.py", line 816, in handle_command_exit_code
    raise exc
ErrorReturnCode_1: 

  RAN: /usr/bin/git submodule foreach git tag -fa -m 'user mobrovac' -m 'timestamp 2018-01-05T18:50:25.084892' -- scap/sync/2018-01-05/0001 c9957ce9ddd889f66eabf41ac19b57d9b63298b4

  STDOUT:
Entering 'src'


  STDERR:
fatal: bad object type.
Stopping at 'src'; script returned non-zero status.

18:50:25 deploy failed: <ErrorReturnCode_1> 

  RAN: /usr/bin/git submodule foreach git tag -fa -m 'user mobrovac' -m 'timestamp 2018-01-05T18:50:25.084892' -- scap/sync/2018-01-05/0001 c9957ce9ddd889f66eabf41ac19b57d9b63298b4

  STDOUT:
Entering 'src'


  STDERR:
fatal: bad object type.
Stopping at 'src'; script returned non-zero status.

I would really like only properly tested versions on Beta because this is getting absurd, honestly.

@mobrovac: beta is where proper testing takes place.

I'm working on reverting the problematic change. Just give me a few more minutes.

@mobrovac: beta is where proper testing takes place.

If you define testing as not working even for the basic use case, then yes, I concur. I don't think it's insane to require that the basic deployment workflow works after Scap gets updated in Beta.

Ok I've installed 3.7.5 on deployment-tin, this should resolve the issue with tagging.

well now I need to force the right scap version on all of beta hosts...I'm gonna try to figure out how to use cumin to do that.

Ok I force-downgraded scap on deployment-prep with cumin, as follows:

sudo cumin 'O{project:deployment-prep}' 'dpkg-query --status scap && DEBIAN_FRONTEND=noninteractive apt-get -q -y --force-yes -o \  DPkg::Options::="--force-confdef" -o DPkg::Options::="--force-confold" \
  install scap || true'

Hm, I'm still getting the same message when trying to deploy Mathoid from deployment-tin, and it seems Scap hasn't been downgraded there:

$ scap version
3.8.0-1~20180105205453.271

Is this problem related? I was about to go and report a separate bug, but...

Info: Applying configuration version '1515367485'
Error: Could not update: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install scap' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
The following packages will be DOWNGRADED:
  scap
0 upgraded, 0 newly installed, 1 downgraded, 0 to remove and 2 not upgraded.
Need to get 112 kB of archives.
After this operation, 1024 B disk space will be freed.
E: There are problems and -y was used without --force-yes
Error: /Stage[main]/Scap/Package[scap]/ensure: change from 3.8.0-1~20180105205453.271 to 3.7.4-3+0~20180106122359.272~1.gbp3819c6 failed: Could not update: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install scap' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
The following packages will be DOWNGRADED:
  scap
0 upgraded, 0 newly installed, 1 downgraded, 0 to remove and 2 not upgraded.
Need to get 112 kB of archives.
After this operation, 1024 B disk space will be freed.
E: There are problems and -y was used without --force-yes
Arlolra added a subscriber: Arlolra.Jan 8 2018, 5:54 PM

@mobrovac: your problem should also be fixed, as of the aforementioned build. Going forward, we will be making some changes to the scap CI process to hopefully address this conflict of interest between testing scap and testing everything else on beta which depends on a working scap environment for deployments.

Heh, as far as I can see, the build completed, but I'm still stuck with the same problem and the same Scap version on deployment-tin as before. I tried forcing a Puppet run, but no luck:

Error: Could not update: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install scap' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
The following packages will be DOWNGRADED:
  scap
0 upgraded, 0 newly installed, 1 downgraded, 0 to remove and 0 not upgraded.
Need to get 112 kB of archives.
After this operation, 1024 B disk space will be freed.
E: There are problems and -y was used without --force-yes
Error: /Stage[main]/Scap/Package[scap]/ensure: change from 3.8.0-1~20180105205453.271 to 3.7.4-3+0~20180106122359.272~1.gbp3819c6 failed: Could not update: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install scap' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
The following packages will be DOWNGRADED:
  scap
0 upgraded, 0 newly installed, 1 downgraded, 0 to remove and 0 not upgraded.
Need to get 112 kB of archives.
After this operation, 1024 B disk space will be freed.
E: There are problems and -y was used without --force-yes

dependent builds are stuck... I'm working on it

greg added a subscriber: greg.Jan 8 2018, 6:38 PM
18:32:32       +thcipriani | !log doing https://www.mediawiki.org/wiki/Continuous_integration/Architecture/Troubleshooting#Jenkins_executioner_lock to fix
                           | deployment-tin executioner lock stalling postmerge.

@Krenair: that should be fixed as soon as jenkins is finished building https://integration.wikimedia.org/ci/job/phabricator-jessie-commits/896/

It looks like this has started and finished but puppet still shows a package downgrade error

That should be resolved now. Package upgraded properly on the nodes I've tested.

twentyafterfour@deployment-tin:/srv/deployment/mathoid/deploy$ scap deploy
20:28:58 Started deploy [mathoid/deploy@c9957ce] (beta)
20:28:58 Deploying Rev: HEAD = c9957ce9ddd889f66eabf41ac19b57d9b63298b4
20:28:58 Started deploy [mathoid/deploy@c9957ce] (beta): (no justification provided)
20:28:58 
== DEFAULT ==
:* deployment-mathoid.deployment-prep.eqiad.wmflabs
mathoid/deploy: fetch stage(s): 100% (ok: 1; fail: 0; left: 0)                  
mathoid/deploy: config_deploy stage(s): 100% (ok: 1; fail: 0; left: 0)          
mathoid/deploy: promote and restart_service stage(s): 100% (ok: 1; fail: 0; left: 0)
20:29:30 
== DEFAULT ==
:* deployment-mathoid.deployment-prep.eqiad.wmflabs
mathoid/deploy: finalize stage(s): 100% (ok: 1; fail: 0; left: 0)               
20:29:33 Finished deploy [mathoid/deploy@c9957ce] (beta): (no justification provided) (duration: 00m 35s)
20:29:33 Finished deploy [mathoid/deploy@c9957ce] (beta) (duration: 00m 35s)
mmodell closed this task as Resolved.Jan 8 2018, 8:30 PM