We should be able to use Quibble to clone the repositories / vendor.git and run composer:
- Queries
- All Stories
- Search
- Advanced Search
- Transactions
- Transaction Logs
Feed Advanced Search
Advanced Search
Advanced Search
Apr 12 2019
Apr 12 2019
hashar added a comment to T189567: Phaseout CI mediawiki config / extensions_load.txt to load extensions.
hashar added a comment to T189567: Phaseout CI mediawiki config / extensions_load.txt to load extensions.
I am back on this and the usage has been extended to the phan jobs unfortunately.
hashar closed T220388: Usage of composer validate on mediawiki/core and Wikibase* extensions as Resolved.
Apr 11 2019
Apr 11 2019
Context manager to time stuff
Emit a warning for EXT_DEPENDENCIES and SKIN_DEPENDENCIES
hashar closed T169008: Figure out the package conflict between libtidy-dev from sury and hhvm-tidy as Declined.
I think this one is too old, I don't remember having that issue with the Docker packages. Moreover we have dropped tidy in favor of remexhtml so there is not much point in keeping a bug for libtidy.
Default to use 4 git workers
I think that is good enough for now. We can add more --git-parallel later on.
REL1_27 and the outdated composer-merge-plugin are gone.
hashar closed T203084: Migrate DonationInterface CI job to Docker, a subtask of T183512: [EPIC] Migrate Mediawiki jobs from Nodepool to Docker, as Resolved.
hashar added a comment to T40863: Put the shorturl in a better place: current design is extremely ugly and confusing.
In T40863#5104161, @Ladsgroup wrote:I guess we can just undeploy this extension in favor of Url Shortener
Add tox setup to run flake8
Default to use 4 git workers
Seems it is working fine, will make it the default in a future Quibble version.
My code logic was wrong. The intent is to ignore error when trying to submit a patch that is already merged by returning early. But my patch had 'submit' not in action which does the opposite.
The patch has some issues though:
2019-04-11 00:55:50,226 DEBUG zuul.reporter.gerrit.Reporter: Report change <Change 0x7f2e586e6390 502894,5>, params {'verified': -1}, message: Gate pipeline build failed.Apr 10 2019
Apr 10 2019
A fast way to delete all Zuul references:
So I think we can just prune all references from time to time:
rm .git/packed_refs find .git/refs/ -mindepth 1 -delete git fetch git pack-refs
I could not reproduce on my local machine given a basic test script which creates several heads:
import git import logging
Down to just 10 seconds after running git remote prune origin.
2019-04-10 14:12:33,095 DEBUG zuul.Repo: Resetting repository /srv/zuul/git/mediawiki/core 2019-04-10 14:12:33,096 DEBUG zuul.Repo: Updating repository /srv/zuul/git/mediawiki/core 2019-04-10 14:12:43,281 DEBUG zuul.Repo: Checking out 37e13f0c6eb3f8bccc49a646de477c2c7bb0d0c7
Seems like zuul-merger does not prune stall references:
zuul@contint2001:/srv/zuul/git/mediawiki/core$ git remote prune -n origin Pruning origin URL: ssh://jenkins-bot@gerrit.wikimedia.org:29418/mediawiki/core * [would prune] origin/REL1_1 * [would prune] origin/REL1_10 * [would prune] origin/REL1_11 * [would prune] origin/REL1_12 * [would prune] origin/REL1_13 * [would prune] origin/REL1_14 * [would prune] origin/REL1_15 * [would prune] origin/REL1_16 * [would prune] origin/REL1_17 * [would prune] origin/REL1_18 * [would prune] origin/REL1_19 * [would prune] origin/REL1_2 * [would prune] origin/REL1_20 * [would prune] origin/REL1_21 * [would prune] origin/REL1_22 * [would prune] origin/REL1_24 * [would prune] origin/REL1_3 * [would prune] origin/REL1_4 * [would prune] origin/REL1_5 * [would prune] origin/REL1_6 * [would prune] origin/REL1_7 * [would prune] origin/REL1_8 * [would prune] origin/REL1_9 * [would prune] origin/fundraising/REL1_22 * [would prune] origin/fundraising/REL1_23 * [would prune] origin/fundraising/REL1_25 * [would prune] origin/fundraising/REL1_26 * [would prune] origin/sandbox/jdlrobson/design * [would prune] origin/sandbox/twentyafterfour/group0 * [would prune] origin/wmf/1.29.0-wmf.1 * [would prune] origin/wmf/1.29.0-wmf.10 * [would prune] origin/wmf/1.29.0-wmf.11 * [would prune] origin/wmf/1.29.0-wmf.12 * [would prune] origin/wmf/1.29.0-wmf.13 * [would prune] origin/wmf/1.29.0-wmf.14 * [would prune] origin/wmf/1.29.0-wmf.15 * [would prune] origin/wmf/1.29.0-wmf.16 * [would prune] origin/wmf/1.29.0-wmf.17 * [would prune] origin/wmf/1.29.0-wmf.18 * [would prune] origin/wmf/1.29.0-wmf.19 * [would prune] origin/wmf/1.29.0-wmf.2 * [would prune] origin/wmf/1.29.0-wmf.20 * [would prune] origin/wmf/1.29.0-wmf.21 * [would prune] origin/wmf/1.29.0-wmf.3 * [would prune] origin/wmf/1.29.0-wmf.4 * [would prune] origin/wmf/1.29.0-wmf.5 * [would prune] origin/wmf/1.29.0-wmf.6 * [would prune] origin/wmf/1.29.0-wmf.7 * [would prune] origin/wmf/1.29.0-wmf.8 * [would prune] origin/wmf/1.29.0-wmf.9 * [would prune] origin/wmf/1.30.0-wmf.1 * [would prune] origin/wmf/1.30.0-wmf.10 * [would prune] origin/wmf/1.30.0-wmf.11 * [would prune] origin/wmf/1.30.0-wmf.12 * [would prune] origin/wmf/1.30.0-wmf.13 * [would prune] origin/wmf/1.30.0-wmf.14 * [would prune] origin/wmf/1.30.0-wmf.15 * [would prune] origin/wmf/1.30.0-wmf.16 * [would prune] origin/wmf/1.30.0-wmf.17 * [would prune] origin/wmf/1.30.0-wmf.18 * [would prune] origin/wmf/1.30.0-wmf.19 * [would prune] origin/wmf/1.30.0-wmf.2 * [would prune] origin/wmf/1.30.0-wmf.3 * [would prune] origin/wmf/1.30.0-wmf.4 * [would prune] origin/wmf/1.30.0-wmf.5 * [would prune] origin/wmf/1.30.0-wmf.6 * [would prune] origin/wmf/1.30.0-wmf.7 * [would prune] origin/wmf/1.30.0-wmf.9 * [would prune] origin/wmf/1.31.0-wmf.1 * [would prune] origin/wmf/1.31.0-wmf.10 * [would prune] origin/wmf/1.31.0-wmf.11 * [would prune] origin/wmf/1.31.0-wmf.12 * [would prune] origin/wmf/1.31.0-wmf.15 * [would prune] origin/wmf/1.31.0-wmf.16 * [would prune] origin/wmf/1.31.0-wmf.17 * [would prune] origin/wmf/1.31.0-wmf.2 * [would prune] origin/wmf/1.31.0-wmf.20 * [would prune] origin/wmf/1.31.0-wmf.21 * [would prune] origin/wmf/1.31.0-wmf.22 * [would prune] origin/wmf/1.31.0-wmf.23 * [would prune] origin/wmf/1.31.0-wmf.24 * [would prune] origin/wmf/1.31.0-wmf.25 * [would prune] origin/wmf/1.31.0-wmf.26 * [would prune] origin/wmf/1.31.0-wmf.27 * [would prune] origin/wmf/1.31.0-wmf.28 * [would prune] origin/wmf/1.31.0-wmf.29 * [would prune] origin/wmf/1.31.0-wmf.3 * [would prune] origin/wmf/1.31.0-wmf.30 * [would prune] origin/wmf/1.31.0-wmf.4 * [would prune] origin/wmf/1.31.0-wmf.5 * [would prune] origin/wmf/1.31.0-wmf.6 * [would prune] origin/wmf/1.31.0-wmf.7 * [would prune] origin/wmf/1.31.0-wmf.8 * [would prune] origin/wmf/1.32.0-wmf.1 * [would prune] origin/wmf/1.32.0-wmf.10 * [would prune] origin/wmf/1.32.0-wmf.12 * [would prune] origin/wmf/1.32.0-wmf.13 * [would prune] origin/wmf/1.32.0-wmf.14 * [would prune] origin/wmf/1.32.0-wmf.15 * [would prune] origin/wmf/1.32.0-wmf.16 * [would prune] origin/wmf/1.32.0-wmf.18 * [would prune] origin/wmf/1.32.0-wmf.19 * [would prune] origin/wmf/1.32.0-wmf.2 * [would prune] origin/wmf/1.32.0-wmf.20 * [would prune] origin/wmf/1.32.0-wmf.22 * [would prune] origin/wmf/1.32.0-wmf.23 * [would prune] origin/wmf/1.32.0-wmf.24 * [would prune] origin/wmf/1.32.0-wmf.26 * [would prune] origin/wmf/1.32.0-wmf.3 * [would prune] origin/wmf/1.32.0-wmf.4 * [would prune] origin/wmf/1.32.0-wmf.5 * [would prune] origin/wmf/1.32.0-wmf.6 * [would prune] origin/wmf/1.32.0-wmf.7 * [would prune] origin/wmf/1.32.0-wmf.8 * [would prune] origin/wmf/1.32.0-wmf.999 * [would prune] origin/wmf/1.33-wmf.8 * [would prune] origin/wmf/1.33.0-wmf.1 * [would prune] origin/wmf/1.33.0-wmf.12 * [would prune] origin/wmf/1.33.0-wmf.13 * [would prune] origin/wmf/1.33.0-wmf.14 * [would prune] origin/wmf/1.33.0-wmf.16 * [would prune] origin/wmf/1.33.0-wmf.2 * [would prune] origin/wmf/1.33.0-wmf.3 * [would prune] origin/wmf/1.33.0-wmf.4 * [would prune] origin/wmf/1.33.0-wmf.6 * [would prune] origin/wmf/1.33.0-wmf.8 * [would prune] origin/wmf/1.33.0-wmf.9 * [would prune] origin/wmf/php-1.32.0-wmf.15 * [would prune] origin/wmf/testing
hashar updated the task description for T220606: zuul-merger takes a while to recreate repository branches.
Another slowness in zuul-merger is T220606: zuul-merger takes a while to recreate repository branches
Something somehow got enhanced and it is way faster nowadays. Either due to gallium disk that was slow, network, better Gerrit, optimizations of git or whatever.
$ git ls-remote .|grep -c refs/zuul 95826
hashar closed T70481: Zuul repositories have too many refs causing slow updates, a subtask of T70480: Scale merge operations (tracking), as Resolved.
Travis does that using folding markers. Eg:
travis_fold:start:Some step // some stuff there travis_fold:end:Some step
hashar added a comment to T220561: The Fresnel job output (linked in Gerrit) should not start with unrelated setup steps.
I am also wondering whether we should have Quibble to output predetermined markers which would make the plugin config easier/simpler for a lot of use cases. I have filled T220586 to investigate :]
Apr 9 2019
Apr 9 2019
hashar added a comment to T220274: sql files of extension CreditsSource failing on unit tests due to default on text column.
Note that CreditsSource is used on Wikivoyage wikis:
wmf-config/InitialiseSettings.php
'wmgUseCreditsSource' => [ 'default' => false, 'wikivoyage' => true, ],
hashar added a project to T220514: MediaWiki web installer does not show extension when their dependency is missing: MW-1.32-release.
The extensions being hidden/skipped is due to https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/461280/ install.php: Allow extensions and skins to be specified which is in MW-1.32-release.
hashar updated the task description for T220514: MediaWiki web installer does not show extension when their dependency is missing.
hashar renamed T220514: MediaWiki web installer does not show extension when their dependency is missing from MediaWiki installer ignore extension when a requires is missing instead of failling to MediaWiki web installer do not show extension when their dependency is missing.
hashar added a comment to T219694: Enable compression for MW web responses in Jenkins jobs (e.g. Quibble, Fresnel).
The php package should be https://packages.debian.org/source/stretch/php7.0 , with the git repository supposedly on gitlab:
hashar added a comment to T219694: Enable compression for MW web responses in Jenkins jobs (e.g. Quibble, Fresnel).
Quibble indeed uses php -S which saved me to have to deal with a frontend web server and have to configure it. Anyway, since the webserver is the PHP CLI it eventually loads /etc/php/7.0/cli/php.ini
hashar added a comment to T212267: MW Database tests should cover when a database name contains dashes.
Indeed, sounds better to have some test written for DatabaseDomain :)
(do not submit) Debugging installer!
hashar added a comment to T191182: Migrate active repositories in Phabricator Differential to GitLab.
We could do something similar with Gerrit. Roughly:
@EddieGP I wanted to remove the check pipeline, notably because most of those jobs were legacy anyway and have since been replaced by better alternative. An example is untrusted developers being able to run the linters/style checker via composer test and/or npm test.
In T208988#5034498, @Tgr wrote:Same issue in https://integration.wikimedia.org/ci/job/wmf-quibble-core-vendor-mysql-hhvm-docker/10779/console (for 497357):
16:24:01 ResourceLoaderFileModule::readStyleFile: style file not found: "/workspace/src/extensions/VisualEditor/lib/ve/lib/color-picker/color-picker.css" ... 16:24:01 Message 'visualeditor-diff-no-changes' required by 'ext.visualEditor.mwsave' must existSeems like either a bug in quibble's submodule initialization logic, or git clone flakiness.
Apr 8 2019
Apr 8 2019
Clone requirements from ext dependencies
Clone requirements from ext dependencies
Basic reader for MediaWiki registration system
hashar triaged T212267: MW Database tests should cover when a database name contains dashes as Low priority.
hashar removed a project from T220388: Usage of composer validate on mediawiki/core and Wikibase* extensions: Patch-For-Review.
The job no more exists since roughly Feb 19th 2019.
hashar closed T218559: puppet broken on integration WMCS instances due to openstack Debian packages as Resolved.
Seems good now.
hashar closed T208426: Deploy Zuul 2.5.1-wmf6, a subtask of T162787: Upgrade pbr for zuul, as Resolved.
I have upgraded Zuul to 2.5.1-wmf.6 (T208426). This is now pending verification which I guess can be done by looking at the Zuul error.log.
I have upgraded Zuul to 2.5.1-wmf6 (T208426).
hashar closed T214807: Patch zuul to remove /p/ from /info/refs call, a subtask of T200739: Upgrade to Gerrit 2.16.13, as Resolved.
I have upgraded Zuul to 2.5.1-wmf6 (T208426):
contint1001:~$ zuul --version Zuul version: 2.5.1-wmf6
hashar closed T162787: Upgrade pbr for zuul, a subtask of T218559: puppet broken on integration WMCS instances due to openstack Debian packages, as Resolved.
hashar closed T198968: Zuul get confused when querying a change that yields two entries as Resolved.
I have upgraded Zuul to 2.5.1-wmf6 (T208426). It now queries changes with change:%s instead of just %s and thus we would no more have a collision of the request change number with some random patchset from a different repo/change.
I have upgraded Zuul to 2.5.1-wmf6 (T208426):
contint1001:~$ dpkg -L zuul|grep GitPython /usr/share/python/zuul/lib/python2.7/site-packages/GitPython-2.1.11.dist-info
hashar closed T214739: Update zuul's GitPython dep, a subtask of T200739: Upgrade to Gerrit 2.16.13, as Resolved.
hashar closed T220237: quibble-vendor-mysql-php73-docker fails on experimental run with filesystem error as Resolved.
I have just rebuild the image. Should be good now :]
hashar added a comment to T220237: quibble-vendor-mysql-php73-docker fails on experimental run with filesystem error.
There is no such digest on the build host:
contint1001:~$ sudo docker image ls --all --no-trunc|grep 769123a16e79cf3be7d995570273a3765a87370b50fe3ff9440e4a9f3e4fcbb9 contint1001:~$
hashar added a comment to T220237: quibble-vendor-mysql-php73-docker fails on experimental run with filesystem error.
Indeed the image docker-registry.wikimedia.org/releng/quibble-stretch-php73:0.0.31-1 has not been downloaded on any of the CI slaves and I have the same issue when pulling it locally. Its latest changelog is:
debian/changelog
quibble-stretch-php73 (0.0.31-1) wikimedia; urgency=medium
hashar added a comment to T188488: Castor rsync causes: rsync: failed to set times on "/cache/.": Operation not permitted (1) .
From another comment I made:
Emit a warning for EXT_DEPENDENCIES and SKIN_DEPENDENCIES
And https://gerrit.wikimedia.org/r/501794 changes phan invocation to no more enforce the progress bar.
We looked at vipscaler a few years ago but we are no more interested in it as far as I can tell.
hashar closed T53449: Improve JPG sharpening algorithm, a subtask of T53451: Use VipsScaler to scale JPGs on WMF wikis, as Declined.
We looked at vipscaler a few years ago but we are no more interested in it as far as I can tell.
hashar closed T53275: Vips scalar hits memory limit when scaling large (100MB) progressive jpeg file as Declined.
We looked at vipscaler a few years ago but we are no more interested in it as far as I can tell.
Apr 5 2019
Apr 5 2019
6kaaaaaaaaaaaa
%26tbaaaaaaaaaaa
etbaaaaaaaaaaa
bubaaaaaaaaaaa
Or it is because the cache were being flushed and its a redhearing
And for the record Gerrit had a thread for Zuul doing:
Solved by restarting Gerrit due to an issue which I think is already tracked somewhere else.
Then looking in the Zuul error.log:
2019-04-05 21:38:34,742 ERROR zuul.GerritEventConnector: Exception moving Gerrit event:
Traceback (most recent call last):
File "/usr/share/python/zuul/local/lib/python2.7/site-packages/zuul/connection/gerrit.py", line 128, in run
self._handleEvent()
File "/usr/share/python/zuul/local/lib/python2.7/site-packages/zuul/connection/gerrit.py", line 116, in _handleEvent
event.change_number, event.patch_number, refresh=True)
File "/usr/share/python/zuul/local/lib/python2.7/site-packages/zuul/source/gerrit.py", line 173, in _getChange
self._updateChange(change, history)
File "/usr/share/python/zuul/local/lib/python2.7/site-packages/zuul/source/gerrit.py", line 238, in _updateChange
data = self.connection.query(change.number)
File "/usr/share/python/zuul/local/lib/python2.7/site-packages/zuul/connection/gerrit.py", line 291, in query
out, err = self._ssh(cmd)
File "/usr/share/python/zuul/local/lib/python2.7/site-packages/zuul/connection/gerrit.py", line 387, in _ssh
raise Exception("Gerrit error executing %s" % command)
Exception: Gerrit error executing gerrit query --format json --all-approvals --comments --commit-message --current-patch-set --dependencies --files --patch-sets --submit-records 501765[2019-04-05 17:28:48,600 +0000] 2765248a jenkins-bot a/75 LOGIN FROM 2620:0:861:1:208:80:154:17
So Zuul has two connection established:
contint1001$ sudo netstat -tp|grep gerrit tcp6 0 0 contint1001.wikim:53720 gerrit.wikimedia.:29418 ESTABLISHED 24045/python tcp6 0 0 contint1001.wikim:53670 gerrit.wikimedia.:29418 ESTABLISHED 24045/python
'gerrit show-connections' Session User Remote Host -------------------------------------------------------------- 2765248a jenkins-bot contint1001.wikimedi a746f4a4 jenkins-bot contint1001.wikimedi
hashar added a comment to T212180: Clone gerrit repo mediawiki/extensions/JADE to mediawiki/extensions/Jade.
Yes it should be archived (and not deleted since we would lost changes and reviews made to it).
hashar added a comment to T189560: mediawiki/vendor REL1_* no longer ship dependencies for wmf extensions that are not in the mediawiki tarball.
Sorry I have a long backlog. Yes the wmf-quibble jobs are really just meant for the wmf branches.
Emit a warning for EXT_DEPENDENCIES and SKIN_DEPENDENCIES
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL