Page MenuHomePhabricator
Create Task
Maniphest T184176

Scap not working in Beta
Closed, ResolvedPublic
Actions

Description

Running Scap on deployment-tin in /srv/deployment/mathoid/deploy yields:

$ scap deploy
12:26:32 Started deploy [mathoid/deploy@c9957ce] (beta)
12:26:32 Deploying Rev: HEAD = c9957ce9ddd889f66eabf41ac19b57d9b63298b4
12:26:32 Finished deploy [mathoid/deploy@c9957ce] (beta) (duration: 00m 00s)
12:26:32 Unhandled error:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/scap/cli.py", line 330, in run
    exit_status = app.main(app.extra_arguments)
  File "/usr/lib/python2.7/dist-packages/scap/deploy.py", line 705, in main
    git.tag_repo(self.deploy_info, location=self.context.root)
  File "/usr/lib/python2.7/dist-packages/scap/git.py", line 454, in tag_repo
    git('submodule', 'foreach', "git %s" % submodule_cmd)
  File "/usr/lib/python2.7/dist-packages/scap/sh.py", line 1428, in __call__
    return RunningCommand(cmd, call_args, stdin, stdout, stderr)
  File "/usr/lib/python2.7/dist-packages/scap/sh.py", line 775, in __init__
    self.wait()
  File "/usr/lib/python2.7/dist-packages/scap/sh.py", line 793, in wait
    self.handle_command_exit_code(exit_code)
  File "/usr/lib/python2.7/dist-packages/scap/sh.py", line 816, in handle_command_exit_code
    raise exc
ErrorReturnCode_1: 

  RAN: /usr/bin/git submodule foreach git tag -fa -m user mobrovac -m timestamp 2018-01-04T12:26:32.188215 -- scap/sync/2018-01-04/0003 c9957ce9ddd889f66eabf41ac19b57d9b63298b4

  STDOUT:
Entering 'src'


  STDERR:
fatal: too many params
Stopping at 'src'; script returned non-zero status.

12:26:32 deploy failed: <ErrorReturnCode_1> 

  RAN: /usr/bin/git submodule foreach git tag -fa -m user mobrovac -m timestamp 2018-01-04T12:26:32.188215 -- scap/sync/2018-01-04/0003 c9957ce9ddd889f66eabf41ac19b57d9b63298b4

  STDOUT:
Entering 'src'


  STDERR:
fatal: too many params
Stopping at 'src'; script returned non-zero status.

The same problem is present in other deployment repos on deployment-tin.

Revisions and Commits

rMSCA Scap
Restricted Differential RevisionrMSCA0552a1bad902 Quote the -m arguments because they contain a space.

Related Objects

Event Timeline

• mobrovac created this task.Jan 4 2018, 12:29 PM
• mobrovac triaged this task as High priority.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 4 2018, 12:29 PM
ArielGlenn subscribed.Jan 4 2018, 1:26 PM
thcipriani added a comment.Jan 4 2018, 4:29 PM

This one seems related to https://github.com/wikimedia/scap/commit/5acb672bea5beecf677e791eb0ca332793412201 and T182865 @mmodell do you have some time to check this out?

• mmodell claimed this task.Jan 4 2018, 4:30 PM

@thcipriani: I'm on it

Restricted Application added a project: Release-Engineering-Team (Kanban). · View Herald TranscriptJan 4 2018, 4:30 PM
• mmodell added a revision: Restricted Differential Revision.Jan 4 2018, 5:04 PM
• mmodell closed this task as Resolved by committing rMSCA0552a1bad902: Quote the -m arguments because they contain a space..Jan 4 2018, 5:44 PM
• mmodell added a commit: rMSCA0552a1bad902: Quote the -m arguments because they contain a space..
• mobrovac added a comment.Jan 4 2018, 5:47 PM

Thnx for the fix. Please make it a thing to test Scap in Beta when a new version lands there.

• mmodell added a comment.Jan 4 2018, 5:49 PM

Perhaps we should just skip the auto-upgrade and install the testing packages in an alternate location ... but that would likely lead to less testing, not more.. hmm

• mobrovac added a comment.Jan 4 2018, 5:52 PM
In T184176#3875873, @mmodell wrote:

Perhaps we should just skip the auto-upgrade and install the testing packages in an alternate location ... but that would likely lead to less testing, not more.. hmm

Auto-upgrade is a useful feature, imho, provided the normal deployment process is ensured not to break after a change lands.

• mobrovac reopened this task as Open.Jan 5 2018, 6:53 PM

This is still an issue even though the params are now quoted:

18:50:25 Started deploy [mathoid/deploy@c9957ce] (beta)
18:50:25 Deploying Rev: HEAD = c9957ce9ddd889f66eabf41ac19b57d9b63298b4
18:50:25 Prepare config deploy
18:50:25 Config deploy file: /srv/deployment/mathoid/deploy/scap/config-files.yaml
18:50:25 Update DEPLOY_HEAD
18:50:25 Creating /srv/deployment/mathoid/deploy/.git/DEPLOY_HEAD
18:50:25 Finished deploy [mathoid/deploy@c9957ce] (beta) (duration: 00m 00s)
18:50:25 Unhandled error:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/scap/cli.py", line 330, in run
    exit_status = app.main(app.extra_arguments)
  File "/usr/lib/python2.7/dist-packages/scap/deploy.py", line 705, in main
    git.tag_repo(self.deploy_info, location=self.context.root)
  File "/usr/lib/python2.7/dist-packages/scap/git.py", line 455, in tag_repo
    git('submodule', 'foreach', "git {0}".format(" ".join(args)))
  File "/usr/lib/python2.7/dist-packages/scap/sh.py", line 1428, in __call__
    return RunningCommand(cmd, call_args, stdin, stdout, stderr)
  File "/usr/lib/python2.7/dist-packages/scap/sh.py", line 775, in __init__
    self.wait()
  File "/usr/lib/python2.7/dist-packages/scap/sh.py", line 793, in wait
    self.handle_command_exit_code(exit_code)
  File "/usr/lib/python2.7/dist-packages/scap/sh.py", line 816, in handle_command_exit_code
    raise exc
ErrorReturnCode_1: 

  RAN: /usr/bin/git submodule foreach git tag -fa -m 'user mobrovac' -m 'timestamp 2018-01-05T18:50:25.084892' -- scap/sync/2018-01-05/0001 c9957ce9ddd889f66eabf41ac19b57d9b63298b4

  STDOUT:
Entering 'src'


  STDERR:
fatal: bad object type.
Stopping at 'src'; script returned non-zero status.

18:50:25 deploy failed: <ErrorReturnCode_1> 

  RAN: /usr/bin/git submodule foreach git tag -fa -m 'user mobrovac' -m 'timestamp 2018-01-05T18:50:25.084892' -- scap/sync/2018-01-05/0001 c9957ce9ddd889f66eabf41ac19b57d9b63298b4

  STDOUT:
Entering 'src'


  STDERR:
fatal: bad object type.
Stopping at 'src'; script returned non-zero status.

I would really like only properly tested versions on Beta because this is getting absurd, honestly.

• mmodell added a comment.Jan 5 2018, 6:55 PM

@mobrovac: beta is where proper testing takes place.

• mmodell added a comment.Jan 5 2018, 6:56 PM

I'm working on reverting the problematic change. Just give me a few more minutes.

• mobrovac added a comment.Jan 5 2018, 6:58 PM
In T184176#3878848, @mmodell wrote:

@mobrovac: beta is where proper testing takes place.

If you define testing as not working even for the basic use case, then yes, I concur. I don't think it's insane to require that the basic deployment workflow works after Scap gets updated in Beta.

• mmodell added a comment.Jan 5 2018, 7:31 PM

Ok I've installed 3.7.5 on deployment-tin, this should resolve the issue with tagging.

• mmodell added a comment.Jan 5 2018, 7:38 PM

well now I need to force the right scap version on all of beta hosts...I'm gonna try to figure out how to use cumin to do that.

• mmodell added a comment.Jan 5 2018, 8:09 PM

Ok I force-downgraded scap on deployment-prep with cumin, as follows:

sudo cumin 'O{project:deployment-prep}' 'dpkg-query --status scap && DEBIAN_FRONTEND=noninteractive apt-get -q -y --force-yes -o \  DPkg::Options::="--force-confdef" -o DPkg::Options::="--force-confold" \
  install scap || true'
• mobrovac added a comment.Jan 6 2018, 11:42 AM

Hm, I'm still getting the same message when trying to deploy Mathoid from deployment-tin, and it seems Scap hasn't been downgraded there:

$ scap version
3.8.0-1~20180105205453.271
Krenair subscribed.Jan 7 2018, 11:27 PM

Is this problem related? I was about to go and report a separate bug, but...

Info: Applying configuration version '1515367485'
Error: Could not update: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install scap' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
The following packages will be DOWNGRADED:
  scap
0 upgraded, 0 newly installed, 1 downgraded, 0 to remove and 2 not upgraded.
Need to get 112 kB of archives.
After this operation, 1024 B disk space will be freed.
E: There are problems and -y was used without --force-yes
Error: /Stage[main]/Scap/Package[scap]/ensure: change from 3.8.0-1~20180105205453.271 to 3.7.4-3+0~20180106122359.272~1.gbp3819c6 failed: Could not update: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install scap' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
The following packages will be DOWNGRADED:
  scap
0 upgraded, 0 newly installed, 1 downgraded, 0 to remove and 2 not upgraded.
Need to get 112 kB of archives.
After this operation, 1024 B disk space will be freed.
E: There are problems and -y was used without --force-yes
Krenair mentioned this in T184238: Puppet broken on deployment-eventlogging04 due to missing directory '/var/lib/superset'?.Jan 7 2018, 11:31 PM
Arlolra subscribed.Jan 8 2018, 5:54 PM
• mmodell added a comment.Jan 8 2018, 6:09 PM

@Krenair: that should be fixed as soon as jenkins is finished building https://integration.wikimedia.org/ci/job/phabricator-jessie-commits/896/

• mmodell added a comment.Jan 8 2018, 6:11 PM

@mobrovac: your problem should also be fixed, as of the aforementioned build. Going forward, we will be making some changes to the scap CI process to hopefully address this conflict of interest between testing scap and testing everything else on beta which depends on a working scap environment for deployments.

• mobrovac added a comment.Jan 8 2018, 6:29 PM

Heh, as far as I can see, the build completed, but I'm still stuck with the same problem and the same Scap version on deployment-tin as before. I tried forcing a Puppet run, but no luck:

Error: Could not update: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install scap' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
The following packages will be DOWNGRADED:
  scap
0 upgraded, 0 newly installed, 1 downgraded, 0 to remove and 0 not upgraded.
Need to get 112 kB of archives.
After this operation, 1024 B disk space will be freed.
E: There are problems and -y was used without --force-yes
Error: /Stage[main]/Scap/Package[scap]/ensure: change from 3.8.0-1~20180105205453.271 to 3.7.4-3+0~20180106122359.272~1.gbp3819c6 failed: Could not update: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install scap' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
The following packages will be DOWNGRADED:
  scap
0 upgraded, 0 newly installed, 1 downgraded, 0 to remove and 0 not upgraded.
Need to get 112 kB of archives.
After this operation, 1024 B disk space will be freed.
E: There are problems and -y was used without --force-yes
• mmodell added a comment.Jan 8 2018, 6:29 PM

dependent builds are stuck... I'm working on it

greg subscribed.Jan 8 2018, 6:38 PM
18:32:32       +thcipriani | !log doing https://www.mediawiki.org/wiki/Continuous_integration/Architecture/Troubleshooting#Jenkins_executioner_lock to fix
                           | deployment-tin executioner lock stalling postmerge.
Krenair added a comment.Jan 8 2018, 7:06 PM
In T184176#3883712, @mmodell wrote:

@Krenair: that should be fixed as soon as jenkins is finished building https://integration.wikimedia.org/ci/job/phabricator-jessie-commits/896/

It looks like this has started and finished but puppet still shows a package downgrade error

• mmodell added a comment.Jan 8 2018, 8:24 PM

That should be resolved now. Package upgraded properly on the nodes I've tested.

• mmodell added a comment.Jan 8 2018, 8:29 PM
twentyafterfour@deployment-tin:/srv/deployment/mathoid/deploy$ scap deploy
20:28:58 Started deploy [mathoid/deploy@c9957ce] (beta)
20:28:58 Deploying Rev: HEAD = c9957ce9ddd889f66eabf41ac19b57d9b63298b4
20:28:58 Started deploy [mathoid/deploy@c9957ce] (beta): (no justification provided)
20:28:58 
== DEFAULT ==
:* deployment-mathoid.deployment-prep.eqiad.wmflabs
mathoid/deploy: fetch stage(s): 100% (ok: 1; fail: 0; left: 0)                  
mathoid/deploy: config_deploy stage(s): 100% (ok: 1; fail: 0; left: 0)          
mathoid/deploy: promote and restart_service stage(s): 100% (ok: 1; fail: 0; left: 0)
20:29:30 
== DEFAULT ==
:* deployment-mathoid.deployment-prep.eqiad.wmflabs
mathoid/deploy: finalize stage(s): 100% (ok: 1; fail: 0; left: 0)               
20:29:33 Finished deploy [mathoid/deploy@c9957ce] (beta): (no justification provided) (duration: 00m 35s)
20:29:33 Finished deploy [mathoid/deploy@c9957ce] (beta) (duration: 00m 35s)
• mmodell closed this task as Resolved.Jan 8 2018, 8:30 PM
thcipriani mentioned this in T184628: Rethink beta scap deployment.Jan 10 2018, 5:11 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits