http://en.wikipedia.beta.wmflabs.org/wiki/Special:Version shows that the version of VisualEditor on beta labs is from May 28. The version of Parsoid on beta labs shows no date.
It would be convenient to have VE/Parsoid available on beta labs as well as on test2wiki and mediawiki.org
Version: unspecified
Severity: critical
Whiteboard: rmqa-2013
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | None | T51846 mediawiki/extensions.git does not update some extensions | |||
| Declined | None | T51906 monitor mediawiki extensions superproject misconfiguration |
The mediawiki/extensions.git is not updating VisualEditor extension:
$ git submodule update --init VisualEditor
remote: Counting objects: 3887, done
remote: Finding sources: 100% (5930/5930)
remote: Getting sizes: 100% (1370/1370)
remote: Compressing objects: 21% (288/1370)
remote: Total 5930 (delta 3846), reused 5495 (delta 3829)
Receiving objects: 100% (5930/5930), 2.95 MiB | 622 KiB/s, done.
Resolving deltas: 100% (4091/4091), completed with 275 local objects.
From https://gerrit.wikimedia.org/r/p/mediawiki/extensions/VisualEditor
65602e1..ed1c06e master -> origin/master
Submodule path 'VisualEditor': checked out '5add8cc4c0ea5b305525c30d8af5261406e5d355'
$ cd VisualEditor
$ git rev-parse HEAD origin/master
5add8cc4c0ea5b305525c30d8af5261406e5d355
ed1c06ee6b36851ba1f6e3a68d0584da4c20be46
HEAD should points to the same as origin/master.
Parsoid is updated though:
$ git rev-parse HEAD origin/master
bf8d3dff339e5b3e10f0667850d0114f49db131c
bf8d3dff339e5b3e10f0667850d0114f49db131c
Moving the bug under Wikimedia > Git/Gerrit . Will poke Chad / Sam about it.
On beta, origin/master points to 5add8cc4c0ea5b305525c30d8af5261406e5d355 which mean VisualEditor remote is not being fetched when running 'git pull && git submodule update --init'.
Fixed.
gerrit> select * from submodule_subscriptions where submodule_project_name like '%VisualEditor%';
submodule_project_name | submodule_branch_name | super_project_project_name | super_project_branch_name | submodule_path
-----------------------+-----------------------+----------------------------+---------------------------+---------------
VisualEditor | refs/heads/master | mediawiki/extensions | refs/heads/master | VisualEditor
(1 row; 2 ms)
gerrit> update submodule_subscriptions set submodule_project_name = 'mediawiki/extensions/VisualEditor' where submodule_path = 'VisualEditor';
UPDATE 1; 2 ms
gerrit> select * from submodule_subscriptions where submodule_project_name like '%VisualEditor%';
submodule_project_name | submodule_branch_name | super_project_project_name | super_project_branch_name | submodule_path
----------------------------------+-----------------------+----------------------------+---------------------------+---------------
mediawiki/extensions/VisualEditor | refs/heads/master | mediawiki/extensions | refs/heads/master | VisualEditor
(1 row; 1 ms)
Was the extension at one point pointing to the wrong repository? Will file a bug upstream, since I guess this should've updated itself when the submodule changed.
I have no idea what went wrong :/
Filled bug 49906 to monitor such issues.
Thank you Chad!
http://en.wikipedia.beta.wmflabs.org/wiki/Special:Version
VisualEditor (Version 0.1.0) (4b74101)
19:00, 20 June 2013
:-)
VisualEditor is lagged out again:
$ cd extensions
$ git remote update
$ git rebase
$ git submodule update --init VisualEditor
Submodule path 'VisualEditor': checked out '46c3d48ba7779254581bfad017c0804588a1983d'
$
Then looking at HEAD (that is the checked out version above) and origin/master:
$ git log --oneline HEAD..origin/master
c331c19 Merge "Minor performance optimization and cleanup in FocusableNode"
daa83d2 Minor performance optimization and cleanup in FocusableNode
a08da9f Make node resizing happen inside onAttributeChange
f8d7314 Merge "Simplify ve.ce.ResizableNode by removing code for 'transition' which is not used anyway"
87667bd Merge "Make toolbar look correct with non-standard browser font size settings"
b0b832a Make toolbar look correct with non-standard browser font size settings
59e7a7b Simplify ve.ce.ResizableNode by removing code for 'transition' which is not used anyway
$
$ git rev-parse origin/master
c331c1980ec37a4d6926f138fd1e81879d5db299
$ git rev-parse HEAD
46c3d48ba7779254581bfad017c0804588a1983d
$
Setting Importance etc since today is "D Day" for Visual Editor on ENWiki. Would be great to have betalabs running up to date VE on the day we roll out to a large audience :)
Worked around (decreasing priority), but keeping this open as it might bite us again (as bug 49906 is still open).
Got broken again a couple hours ago:
| \ |
| * 31104d5 - Bind listener to keyup to capture arrows & better ma... |
which shows HEAD not pointing to the same commit as origin/HEAD :(
Change 73736 had a related patch set uploaded by Hashar:
update VisualEditor to latest master
It is apparently totally broken right now. The check-sync.sh script reports a lot of extensions has not being up to date :/
ERROR! DataTypes is lagging behind.
ERROR! DataValues is lagging behind.
ERROR! DidYouKnow is lagging behind.
ERROR! Diff is lagging behind.
ERROR! DisableAccount is lagging behind.
ERROR! Disambiguator is lagging behind.
ERROR! DonationInterface is lagging behind.
ERROR! Duplicator is lagging behind.
ERROR! EImage is lagging behind.
ERROR! Echo is lagging behind.
ERROR! EducationProgram is lagging behind.
ERROR! EventLogging is lagging behind.
ERROR! ExtensionDistributor is lagging behind.
ERROR! GettingStarted is lagging behind.
ERROR! GlobalBlocking is lagging behind.
ERROR! GuidedTour is lagging behind.
ERROR! Hovergallery is lagging behind.
ERROR! InlineCategorizer is lagging behind.
ERROR! Insider is lagging behind.
ERROR! LiquidThreads is lagging behind.
ERROR! Maps is lagging behind.
ERROR! MarkAsHelpful is lagging behind.
ERROR! Math is lagging behind.
ERROR! MobileFrontend is lagging behind.
ERROR! MoodBar is lagging behind.
ERROR! Mpdf is lagging behind.
ERROR! OAuth is lagging behind.
ERROR! Parsoid is lagging behind.
ERROR! PdfExport is lagging behind.
ERROR! PdfHandler is lagging behind.
ERROR! PerPageLicense is lagging behind.
ERROR! PronunciationRecording is lagging behind.
ERROR! RelatedArticles is lagging behind.
ERROR! RevisionCommentSupplement is lagging behind.
ERROR! SecurePoll is lagging behind.
ERROR! SemanticMediaWiki is lagging behind.
ERROR! Thanks is lagging behind.
ERROR! TimedMediaHandler is lagging behind.
ERROR! Translate is lagging behind.
ERROR! TwnMainPage is lagging behind.
ERROR! UniversalLanguageSelector is lagging behind.
ERROR! UploadWizard is lagging behind.
ERROR! ValueFormatters is lagging behind.
ERROR! ValueParsers is lagging behind.
ERROR! ValueValidators is lagging behind.
ERROR! ValueView is lagging behind.
ERROR! Vector is lagging behind.
ERROR! WikiEditor is lagging behind.
ERROR! Wikibase is lagging behind.
ERROR! WikibaseDataModel is lagging behind.
ERROR! WikibaseDatabase is lagging behind.
ERROR! WikibaseQuery is lagging behind.
ERROR! WikibaseQueryEngine is lagging behind.
ERROR! WikimediaMaintenance is lagging behind.
ERROR! WikimediaMessages is lagging behind.
ERROR! ZeroRatedMobileAccess is lagging behind.
Sorry DataTypes was dirty in my local copy. That caused git to not update the remaining extensions. It chokes currently on:
Cloning into 'ValueFormatters'...
remote: Counting objects: 4, done
remote: Finding sources: 100% (4/4)
remote: Getting sizes: 100% (3/3)
remote: Total 4 (delta 0), reused 4 (delta 0)
Unpacking objects: 100% (4/4), done.
Submodule path 'ValueFormatters': checked out 'b466dde64555d82fcefcdd0b1fe838de2e3acada'
fatal: Needed a single revision
Unable to find current revision in submodule path 'ValueParsers'
Does anybody plan to investigate on this ticket?
Or is this "working for us" right now?
Wondering if this should still be open, and what's the way forward.
(In reply to comment #21)
Does anybody plan to investigate on this ticket?
Or is this "working for us" right now?Wondering if this should still be open, and what's the way forward.
The problem was that there was more than one git repos on gerrit called "VisualEditor" (namely, mediawiki/extensions/VisualEditor.git and VisualEditor.git with its subsidiaries VisualEditor/core.git, VisualEditor/plugins/<foo>.git etc.). We created the other repos to move the code out of MW and make VE properly stand-alone and shippable, but thought we have split the code, we hadn't yet moved the core of VE into the new repo).
As a quick hack, Chad deleted the extra repos, which seems to have fixed everything, which means the issue is fixed (and has a known cause), but this isn't sustainable in the longer term as we will likely want to actually do the repo split some time soon (though we could work around the restriction on repo names)…
Marking as "FIXED", but it's more like "AVOIDED AT SOME COST".
The root cause is that there are two Gerrit projects being named VisualEditor and there is a bug in Gerrit that get it confused about that and break the automatic update of mediawiki/extensions.git
To restore automatic update, we would need to register the extensions locally by iterating over gerrit ls-projects -p mediawiki/extensions/ and then use git submodule update --init as we did previously.
We need to do this as a post-merge job in Jenkins for updating the meta-repo. The Beta config should be able to remain as-is.
(In reply to comment #28)
We need to do this as a post-merge job in Jenkins for updating the meta-repo.
The Beta config should be able to remain as-is.
I am not sure we could do a post-merge job since Zuul does not support triggering a job according to a project wildcard such as mediawiki/extensions/*
I thought we could adapt the 6 minutes wmf-beta-autoupdater.py script, make it fetch the list of extensions generated at: https://gerrit.wikimedia.org/mediawiki-extensions.txt then do a git submodule add on all of them and then update them all.
(In reply to comment #29)
(In reply to comment #28)
We need to do this as a post-merge job in Jenkins for updating the meta-repo.
The Beta config should be able to remain as-is.I am not sure we could do a post-merge job since Zuul does not support
triggering a job according to a project wildcard such as
mediawiki/extensions/*
We don't need to for all repos. Just VisualEditor since it's broken. So we'd adjust the VE zuul/jenkins config to update mediawiki/extensions.git after VE merges.
I thought we could adapt the 6 minutes wmf-beta-autoupdater.py script, make
it
fetch the list of extensions generated at:
https://gerrit.wikimedia.org/mediawiki-extensions.txt then do a git
submodule
add on all of them and then update them all.
Ew. I'd rather keep everything else as it is, keep the workaround to one place (the VE zuul/jenkins config) until the upstream fix takes place. No need to change our infrastructure.
Change 107574 had a related patch set uploaded by Hashar:
unregister VisualEditor (replication broken in Gerrit)
Change 107575 had a related patch set uploaded by Hashar:
beta: pull VisualEditor individually
The patches above:
Change 107574 abandoned by Hashar:
unregister VisualEditor (replication broken in Gerrit)
Reason:
Per Chad, will use a Jenkins job to update mediawiki/extensions.git
Change 107575 abandoned by Hashar:
beta: pull VisualEditor individually
Reason:
Per Chad, will use a Jenkins job to update mediawiki/extensions.git
Change 107841 had a related patch set uploaded by Hashar:
Script to sync VisualEditor in mediawiki/extensions.git
Change 107841 merged by jenkins-bot:
Script to sync VisualEditor in mediawiki/extensions.git
Change 107853 had a related patch set uploaded by Hashar:
contint: invoke gerrit-sync-ve-push.sh as jenkins
Change 107853 merged by Andrew Bogott:
contint: invoke gerrit-sync-ve-push.sh as jenkins
Change 107862 had a related patch set uploaded by Hashar:
mwext-VisualEditor-sync-gerrit
I have added a couple Jenkins slave scripts in integration/jenkins.git
bin/gerrit-sync-ve.sh
bin/gerrit-sync-ve-push.sh
That would update the VisualEditor on a local copy of mediawiki/extensions.git
Then had to get a jenkins-bot@wikimedia.org email address registered and assigned to the jenkins-bot Gerrit user.
The job mwext-VisualEditor-sync-gerrit still needs to be triggered by Zuul on postmerge. That is an easy change though. Will do whenever I am sure the job is working properly.
Gerrit still believe that jenkins-bot user does not have any email address despite the address being shown in virt0 LDAP.
The suspect is that the LDAP replication to virt1000 is broken and that is the server Gerrit is using as a primary.
Whenever Gerrit learns about jenkins-bot user, the script I wrote should be able to push its change.
The last build I triggered ( https://integration.wikimedia.org/ci/job/mwext-VisualEditor-sync-gerrit/8/console ) yields:
7:14:47 remote: ERROR: In commit c414977bcdbcbdf6331d9fe2e627bc0768695966
17:14:47 remote: ERROR: committer email address jenkins-bot@wikimedia.org
17:14:47 remote: ERROR: does not match your user account.
17:14:47 remote: ERROR:
17:14:47 remote: ERROR: You have not registered any email addresses.
I've manually updated the database until LDAP gets back in sync. You *should* be able to push now.
When jenkins-bot push to Gerrit, Gerrit fetch the email address of the user from the LDAP directory and compare it to the Commiter field of each commit being pushed.
Gerrit points to LDAP server virt1000 which is no more replicated from virt1. Hence the jenkins-bot record known to virt1000 is still lacking the email field and thus Gerrit thinks jenkins-bot has no email.
This is thus blocked until the virt1 -> virt1000 LDAP replication is fixed.
The Gerrit replication got fixed last week. The bot was not able to connect because Gerrit cache accounts credential indefinitely (lowered to 7 days by Chad with https://gerrit.wikimedia.org/r/#/c/108715/ ).
I did a few tweaks to adjust the shell script and granted jenkins-bot the ability to CR+2 and V+2 on mediawiki/extensions.git .
The first change that self merged is https://gerrit.wikimedia.org/r/#/c/108732/
Gotta add triggers in Zuul.
Change 108810 had a related patch set uploaded by Hashar:
trigger mwext-VisualEditor-sync-gerrit on postmerge
Change 108810 merged by jenkins-bot:
trigger mwext-VisualEditor-sync-gerrit on postmerge
Change 108837 had a related patch set uploaded by Hashar:
mwext-VisualEditor-sync-gerrit on master branch only
Change 108837 merged by jenkins-bot:
mwext-VisualEditor-sync-gerrit on master branch only
After much madness, this is now fixed. I had to write a bunch of shell slave scripts to let us properly push the VE update change to mediawiki/extensions.git and self merge them.
The job is:
https://integration.wikimedia.org/ci/job/mwext-VisualEditor-sync-gerrit/
It managed to merge an update a few minutes ago:
https://gerrit.wikimedia.org/r/#/c/109113/
I guess the issue is fixed now. Sorry for the long time it took to get this fixed.
Fixed in https://gerrit-review.googlesource.com/#/c/69891/
Upgrading to gerrit 2.12 which will happen soon will fix the problem.
Mentioned in SAL [2016-09-08T10:02:05Z] <hashar> Delete mwext-VisualEditor-sync-gerrit job, already got removed by ostriches in 139d17c8f1c4bcf2bb761e13a6501e4d85684066 . The issue in Gerrit (T51846) has been fixed. Poke T86659 , one less job on slaves.
Mentioned in SAL [2016-09-08T10:03:29Z] <hashar> Delete Jenkins job https://integration.wikimedia.org/ci/job/mwext-VisualEditor-sync-gerrit/ that has been left behind. It is no more needed. T51846 T86659
Change 309272 had a related patch set uploaded (by Hashar):
Remove VE script to sync the Gerrit repo
Change 309275 had a related patch set uploaded (by Hashar):
ci::master: drop mwext-sync leftover
This is confirmed to have been fixed when we have upgraded to Gerrit 2.12.
The workaround script and Jenkins job have been removed end of July, the recent changes I have sent above are merely for clean up.
Change 311161 had a related patch set uploaded (by Hashar):
contint: drop now unused sudo rule
Change 317923 had a related patch set uploaded (by Hashar):
contint: remove python-requests