Page MenuHomePhabricator
Differential D1114

Check for fatals in AbstractSync by calling `mwscript eval.php`
ClosedPublic
Actions

Authored by mmodell on Sep 20 2018, 11:43 PM.
Tags
Referenced Files
Unknown Object (File)
Aug 27 2023, 3:26 PM
Unknown Object (File)
Aug 24 2023, 11:12 PM
Unknown Object (File)
Aug 18 2023, 8:18 PM
Unknown Object (File)
Aug 18 2023, 12:33 PM
Unknown Object (File)
Aug 13 2023, 12:55 PM
Unknown Object (File)
Aug 13 2023, 9:43 AM
Unknown Object (File)
Aug 5 2023, 7:04 PM
Unknown Object (File)
Aug 5 2023, 2:32 PM
View All 59 Files
Subscribers
• jenkins

Details

Maniphest Tasks
T121597: Implement MediaWiki pre-promote checks
Reviewers
Krinkle
thcipriani
Group Reviewers
Release-Engineering-Team
Commits
rMWc9b4abef2e6b: Update patch set 1
rMSCAd546dcfc710f: Check for fatals in AbstractSync by calling `mwscript eval.php`
Patch without arc
git checkout -b D1114 && curl -L https://phabricator.wikimedia.org/D1114?download=true | git apply
Summary

This calls mwscript eval.php with "1" on stdin and then checks
for a successful return code. If mwscript exits with an error then
abort the sync and log the command's stderr output.

Test Plan

untested

Diff Detail

Repository
rMSCA Scap
Branch
master
Lint
Lint Passed
Unit
Tests Passed

Event Timeline

mmodell created this revision.Sep 20 2018, 11:43 PM
Restricted Application added a reviewer: Release-Engineering-Team. · View Herald TranscriptSep 20 2018, 11:43 PM
Restricted Application added a project: Release-Engineering-Team. · View Herald Transcript
• jenkins added a subscriber: • jenkins.Sep 20 2018, 11:43 PM

Build has FAILED

Test Name
tox -vBuild Details

tox -v log

Got exit code 1 from PY_COLORS=1 tox -v

doc develop-inst: /srv/jenkins-workspace/workspace/phabricator-jessie-diffs
  /srv/jenkins-workspace/workspace/phabricator-jessie-diffs$ /srv/jenkins-workspace/workspace/phabricator-jessie-diffs/.tox/doc/bin/pip install -e /srv/jenkins-workspace/workspace/phabricator-jessie-diffs >/srv/jenkins-workspace/workspace/phabricator-jessie-diffs/.tox/doc/log/doc-2.log
  /srv/jenkins-workspace/workspace/phabricator-jessie-diffs$ /srv/jenkins-workspace/workspace/phabricator-jessie-diffs/.tox/doc/bin/pip freeze >/srv/jenkins-workspace/workspace/phabricator-jessie-diffs/.tox/doc/log/doc-3.log
doc installed: actdiag==0.5.4,alabaster==0.7.11,argh==0.26.2,Babel==2.6.0,backports-abc==0.5,blockdiag==1.5.4,certifi==2018.8.24,chardet==3.0.4,configparser==3.5.0,conftool==1.0.2,dnspython==1.15.0,docutils==0.14,funcparserlib==0.3.6,functools32==3.2.3.post2,futures==3.2.0,idna==2.7,imagesize==1.1.0,Jinja2==2.10,jsonschema==2.6.0,livereload==2.5.2,MarkupSafe==1.0,packaging==17.1,pathtools==0.1.2,Pillow==5.2.0,port-for==0.3.1,psutil==5.4.7,pyaml==17.12.1,Pygments==2.2.0,pyparsing==2.2.1,python-etcd==0.4.5,pytz==2018.5,PyYAML==3.13,requests==2.19.1,-e git+https://phabricator.wikimedia.org/source/scap.git@3641743790119b0efee060c0d1e2f2fdcc36e097#egg=Scap,singledispatch==3.4.0.3,six==1.11.0,snowballstemmer==1.2.1,Sphinx==1.8.0,sphinx-autobuild==0.7.1,sphinxcontrib-actdiag==0.8.5,sphinxcontrib-blockdiag==1.5.5,sphinxcontrib-programoutput==0.11,sphinxcontrib-websupport==1.1.0,tornado==5.1.1,typing==3.6.6,urllib3==1.23,watchdog==0.9.0,webcolors==1.8.1
doc runtests: PYTHONHASHSEED='1343528468'
doc runtests: commands[0] | sphinx-build -W -b html docs/ docs/_build/html
  /srv/jenkins-workspace/workspace/phabricator-jessie-diffs$ /srv/jenkins-workspace/workspace


Link to build: https://integration.wikimedia.org/ci/job/phabricator-jessie-diffs/1159/
See console output for more information: https://integration.wikimedia.org/ci/job/phabricator-jessie-diffs/1159/console
Harbormaster returned this revision to the author for changes because remote builds failed.Sep 20 2018, 11:43 PM
Harbormaster failed remote builds in Restricted Buildable!
• jenkins added a comment.Sep 20 2018, 11:45 PM

Build has FAILED

Link to build: https://integration.wikimedia.org/ci/job/differential-docker-test/518/
See console output for more information: https://integration.wikimedia.org/ci/job/differential-docker-test/518/console

mmodell updated this revision to Diff 2907.Sep 20 2018, 11:51 PM

fix py3 error

Krinkle added inline comments.Sep 20 2018, 11:52 PM
scap/main.py
204

PHP unfortunately doesn't set the exit code unless the error was fatal. This may need to assert non-empty stderr unconditionally.

Below is an example of that accesses an undefined variable:

$ echo 1 | mwscript eval.php --wiki test2wiki >/dev/null
stderr> [Thu Sep 20 20:05:03 2018] Notice: Undefined variable: krinkle in /srv/mediawiki/php-1.32.0-wmf.22/includes/Setup.php on line 38

$ echo $?
exitcode> 0
• jenkins added a comment.Sep 20 2018, 11:52 PM

Build has FAILED

Link to build: https://integration.wikimedia.org/ci/job/differential-docker-test/519/
See console output for more information: https://integration.wikimedia.org/ci/job/differential-docker-test/519/console

Harbormaster returned this revision to the author for changes because remote builds failed.Sep 20 2018, 11:52 PM
Harbormaster failed remote builds in Restricted Buildable!
• jenkins added a comment.Sep 20 2018, 11:52 PM

Build has FAILED

Test Name
tox -vBuild Details

tox -v log

Got exit code 1 from PY_COLORS=1 tox -v

doc develop-inst: /srv/jenkins-workspace/workspace/phabricator-jessie-diffs
  /srv/jenkins-workspace/workspace/phabricator-jessie-diffs$ /srv/jenkins-workspace/workspace/phabricator-jessie-diffs/.tox/doc/bin/pip install -e /srv/jenkins-workspace/workspace/phabricator-jessie-diffs >/srv/jenkins-workspace/workspace/phabricator-jessie-diffs/.tox/doc/log/doc-2.log
  /srv/jenkins-workspace/workspace/phabricator-jessie-diffs$ /srv/jenkins-workspace/workspace/phabricator-jessie-diffs/.tox/doc/bin/pip freeze >/srv/jenkins-workspace/workspace/phabricator-jessie-diffs/.tox/doc/log/doc-3.log
doc installed: actdiag==0.5.4,alabaster==0.7.11,argh==0.26.2,Babel==2.6.0,backports-abc==0.5,blockdiag==1.5.4,certifi==2018.8.24,chardet==3.0.4,configparser==3.5.0,conftool==1.0.2,dnspython==1.15.0,docutils==0.14,funcparserlib==0.3.6,functools32==3.2.3.post2,futures==3.2.0,idna==2.7,imagesize==1.1.0,Jinja2==2.10,jsonschema==2.6.0,livereload==2.5.2,MarkupSafe==1.0,packaging==17.1,pathtools==0.1.2,Pillow==5.2.0,port-for==0.3.1,psutil==5.4.7,pyaml==17.12.1,Pygments==2.2.0,pyparsing==2.2.1,python-etcd==0.4.5,pytz==2018.5,PyYAML==3.13,requests==2.19.1,-e git+https://phabricator.wikimedia.org/source/scap.git@c12f217f0507b8ecd9dc1c7e409af12ea34f77c0#egg=Scap,singledispatch==3.4.0.3,six==1.11.0,snowballstemmer==1.2.1,Sphinx==1.8.0,sphinx-autobuild==0.7.1,sphinxcontrib-actdiag==0.8.5,sphinxcontrib-blockdiag==1.5.5,sphinxcontrib-programoutput==0.11,sphinxcontrib-websupport==1.1.0,tornado==5.1.1,typing==3.6.6,urllib3==1.23,watchdog==0.9.0,webcolors==1.8.1
doc runtests: PYTHONHASHSEED='4220580003'
doc runtests: commands[0] | sphinx-build -W -b html docs/ docs/_build/html
  /srv/jenkins-workspace/workspace/phabricator-jessie-diffs$ /srv/jenkins-workspace/workspace


Link to build: https://integration.wikimedia.org/ci/job/phabricator-jessie-diffs/1160/
See console output for more information: https://integration.wikimedia.org/ci/job/phabricator-jessie-diffs/1160/console
mmodell updated this revision to Diff 2908.Sep 20 2018, 11:54 PM

Fix sphinx doc error

• jenkins added a comment.Sep 20 2018, 11:55 PM

Build has FAILED

Link to build: https://integration.wikimedia.org/ci/job/differential-docker-test/520/
See console output for more information: https://integration.wikimedia.org/ci/job/differential-docker-test/520/console

Harbormaster returned this revision to the author for changes because remote builds failed.Sep 20 2018, 11:55 PM
Harbormaster failed remote builds in Restricted Buildable!
• jenkins added a comment.Sep 20 2018, 11:56 PM

Build has FAILED

Test Name
tox -vBuild Details

tox -v log

Got exit code 1 from PY_COLORS=1 tox -v

doc develop-inst: /srv/jenkins-workspace/workspace/phabricator-jessie-diffs
  /srv/jenkins-workspace/workspace/phabricator-jessie-diffs$ /srv/jenkins-workspace/workspace/phabricator-jessie-diffs/.tox/doc/bin/pip install -e /srv/jenkins-workspace/workspace/phabricator-jessie-diffs >/srv/jenkins-workspace/workspace/phabricator-jessie-diffs/.tox/doc/log/doc-2.log
  /srv/jenkins-workspace/workspace/phabricator-jessie-diffs$ /srv/jenkins-workspace/workspace/phabricator-jessie-diffs/.tox/doc/bin/pip freeze >/srv/jenkins-workspace/workspace/phabricator-jessie-diffs/.tox/doc/log/doc-3.log
doc installed: actdiag==0.5.4,alabaster==0.7.11,argh==0.26.2,Babel==2.6.0,backports-abc==0.5,blockdiag==1.5.4,certifi==2018.8.24,chardet==3.0.4,configparser==3.5.0,conftool==1.0.2,dnspython==1.15.0,docutils==0.14,funcparserlib==0.3.6,functools32==3.2.3.post2,futures==3.2.0,idna==2.7,imagesize==1.1.0,Jinja2==2.10,jsonschema==2.6.0,livereload==2.5.2,MarkupSafe==1.0,packaging==17.1,pathtools==0.1.2,Pillow==5.2.0,port-for==0.3.1,psutil==5.4.7,pyaml==17.12.1,Pygments==2.2.0,pyparsing==2.2.1,python-etcd==0.4.5,pytz==2018.5,PyYAML==3.13,requests==2.19.1,-e git+https://phabricator.wikimedia.org/source/scap.git@0a2da9b48ba0aad7e68621d1629235caef4dc6d6#egg=Scap,singledispatch==3.4.0.3,six==1.11.0,snowballstemmer==1.2.1,Sphinx==1.8.0,sphinx-autobuild==0.7.1,sphinxcontrib-actdiag==0.8.5,sphinxcontrib-blockdiag==1.5.5,sphinxcontrib-programoutput==0.11,sphinxcontrib-websupport==1.1.0,tornado==5.1.1,typing==3.6.6,urllib3==1.23,watchdog==0.9.0,webcolors==1.8.1
doc runtests: PYTHONHASHSEED='395066301'
doc runtests: commands[0] | sphinx-build -W -b html docs/ docs/_build/html
  /srv/jenkins-workspace/workspace/phabricator-jessie-diffs$ /srv/jenkins-workspace/workspace/


Link to build: https://integration.wikimedia.org/ci/job/phabricator-jessie-diffs/1161/
See console output for more information: https://integration.wikimedia.org/ci/job/phabricator-jessie-diffs/1161/console
mmodell updated this revision to Diff 2909.Sep 21 2018, 12:01 AM

Require nonempty stderr, raise an exception even when exit code is zero.

mmodell marked an inline comment as done.Sep 21 2018, 12:02 AM
mmodell added inline comments.
scap/main.py
204

Ok I've added a check for nonempty stderr output.

Harbormaster failed remote builds in Restricted Buildable!Sep 21 2018, 12:02 AM
• jenkins added a comment.Sep 21 2018, 12:02 AM

Build has FAILED

Link to build: https://integration.wikimedia.org/ci/job/differential-docker-test/521/
See console output for more information: https://integration.wikimedia.org/ci/job/differential-docker-test/521/console

Harbormaster returned this revision to the author for changes because remote builds failed.Sep 21 2018, 12:02 AM
• jenkins added a comment.Sep 21 2018, 12:02 AM

Build has FAILED

Test Name
tox -vBuild Details

tox -v log

Got exit code 1 from PY_COLORS=1 tox -v

doc develop-inst: /srv/jenkins-workspace/workspace/phabricator-jessie-diffs
  /srv/jenkins-workspace/workspace/phabricator-jessie-diffs$ /srv/jenkins-workspace/workspace/phabricator-jessie-diffs/.tox/doc/bin/pip install -e /srv/jenkins-workspace/workspace/phabricator-jessie-diffs >/srv/jenkins-workspace/workspace/phabricator-jessie-diffs/.tox/doc/log/doc-2.log
  /srv/jenkins-workspace/workspace/phabricator-jessie-diffs$ /srv/jenkins-workspace/workspace/phabricator-jessie-diffs/.tox/doc/bin/pip freeze >/srv/jenkins-workspace/workspace/phabricator-jessie-diffs/.tox/doc/log/doc-3.log
doc installed: actdiag==0.5.4,alabaster==0.7.11,argh==0.26.2,Babel==2.6.0,backports-abc==0.5,blockdiag==1.5.4,certifi==2018.8.24,chardet==3.0.4,configparser==3.5.0,conftool==1.0.2,dnspython==1.15.0,docutils==0.14,funcparserlib==0.3.6,functools32==3.2.3.post2,futures==3.2.0,idna==2.7,imagesize==1.1.0,Jinja2==2.10,jsonschema==2.6.0,livereload==2.5.2,MarkupSafe==1.0,packaging==17.1,pathtools==0.1.2,Pillow==5.2.0,port-for==0.3.1,psutil==5.4.7,pyaml==17.12.1,Pygments==2.2.0,pyparsing==2.2.1,python-etcd==0.4.5,pytz==2018.5,PyYAML==3.13,requests==2.19.1,-e git+https://phabricator.wikimedia.org/source/scap.git@8a229b224d9eb9d0fc5984dde75919e2c4e84d05#egg=Scap,singledispatch==3.4.0.3,six==1.11.0,snowballstemmer==1.2.1,Sphinx==1.8.0,sphinx-autobuild==0.7.1,sphinxcontrib-actdiag==0.8.5,sphinxcontrib-blockdiag==1.5.5,sphinxcontrib-programoutput==0.11,sphinxcontrib-websupport==1.1.0,tornado==5.1.1,typing==3.6.6,urllib3==1.23,watchdog==0.9.0,webcolors==1.8.1
doc runtests: PYTHONHASHSEED='1213035992'
doc runtests: commands[0] | sphinx-build -W -b html docs/ docs/_build/html
  /srv/jenkins-workspace/workspace/phabricator-jessie-diffs$ /srv/jenkins-workspace/workspace


Link to build: https://integration.wikimedia.org/ci/job/phabricator-jessie-diffs/1162/
See console output for more information: https://integration.wikimedia.org/ci/job/phabricator-jessie-diffs/1162/console
mmodell updated this revision to Diff 2910.Sep 21 2018, 12:12 AM
mmodell marked an inline comment as done.

fix flake8 and sphinx doc build errors - tests should all pass now.

Harbormaster completed remote builds in Restricted Buildable.Sep 21 2018, 12:13 AM
mmodell requested review of this revision.Sep 21 2018, 12:13 AM
mmodell added a comment.Sep 21 2018, 12:15 AM

Ok I still haven't tested this in an environment similar to prod. I'll try installing and testing this patch in beta next.

thcipriani added a comment.Sep 21 2018, 10:07 PM

Looks good! I did some basic testing in beta and it seems like it'll work. Couple of nitpicks inline. Nice quick work on this :)

scap/main.py
203

Probably could drop len here.

208

Hrm, not sure we should announce the output since it could be incredibly long or dump private info. Might be good for the RuntimeError message, though.

mmodell planned changes to this revision.Sep 21 2018, 10:09 PM
mmodell added inline comments.
scap/main.py
203

true

208

hmm good point.

mmodell updated this revision to Diff 2911.Sep 21 2018, 10:28 PM

Don't announce the stderr output

Harbormaster completed remote builds in Restricted Buildable.Sep 21 2018, 10:29 PM
thcipriani accepted this revision.Sep 21 2018, 10:59 PM

lgtm! Let's land try this in beta!

One nitpick inline.

scap/main.py
209

Using raise(RuntimeError()) vs raise RuntimeError(...) elsewhere in the file/patch.

This revision is now accepted and ready to land.Sep 21 2018, 10:59 PM
mmodell updated this revision to Diff 2912.Sep 24 2018, 4:19 PM

fix nit

Harbormaster completed remote builds in Restricted Buildable.Sep 24 2018, 4:24 PM
Closed by commit rMSCAd546dcfc710f: Check for fatals in AbstractSync by calling `mwscript eval.php` (authored by mmodell). · Explain WhySep 24 2018, 6:20 PM
This revision was automatically updated to reflect the committed changes.
mmodell mentioned this in T204383: Update Debian Package for Scap to 3.8.7-1.Oct 3 2018, 1:53 PM
Stashbot mentioned this in T121597: Implement MediaWiki pre-promote checks.Oct 9 2018, 6:19 PM
• satoshinakamoto added a commit: rMWc9b4abef2e6b: Update patch set 1.Dec 20 2021, 5:33 PM

Revision Contents
Changeset List

PathSize
scap/
26 lines
tests/
scap/
6 lines

Diff 2907

View Options

scap/main.py

Loading...
View Options

tests/scap/test_checks.py

Loading...
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL