Page MenuHomePhabricator
Create Task
Maniphest T229078

Preparing Flow for Parsoid-PHP switch
Closed, ResolvedPublic0 Estimated Story Points
Actions

Description

This is a tracker task to figure out testing / qa needs and any work / changes needed to VisualEditor to switchover from Parsoid/JS to Parsoid/PHP.

T229015: Tracking: Direct live production traffic at Parsoid/PHP is the tracker task for deployment. But, we don't anticipate a deploy before mid-September 2019 at this time. But, let us handle this as if we are going to be switching over Flow to Parsoid/PHP around end September / early October.

Since Flow does not talk to RESTBase, Flow would need a config change to update its endpoints. At this time, we are planning to switch Parsoid clients one at a time. Currently, we are considering switching over Flow as the last Parsoid client. We are assuming we will have all the bugs ironed out by that time and can do a simple switchover without needing traffic partitioning. But, if it becomes necessary, Flow will need to manage a new cookie.

While Flow's HTML storage has data-parsoid DSR offsets stored there, since Flow does not use selective serialization, these offsets are ignored by Parsoid. Nevertheless, it might be appropriate to null out the old offsets (or have some suitable protection against future use) since the offsets generated by Parsoid/JS code are invalid in Parsoid/PHP land.

We will also do an early deployment to the beta cluster so that clients can do early testing there. However, Flow will nevertheless need some update / code to pick its endpoint on the Beta cluster.

But, please comment on the ticket / edit the description adding any other requirements to ensure we cover all our bases.

Details

SubjectRepoBranchLines +/-
Parsoid VRS: Add the Host headermediawiki/corewmf/1.35.0-wmf.5+4 -0
Parsoid VRS: Add the Host headermediawiki/coremaster+4 -0
[Beta] Use Parsoid/PHP for Flowoperations/mediawiki-configmaster+1 -1
beta: Point Parsoid to parsoid-php instead of parsoid-jsoperations/mediawiki-configmaster+1 -1
Flow: Configure enwiki beta to use Parsoid PHPoperations/mediawiki-configmaster+6 -1
Implement X-Parsoid-Variantmediawiki/extensions/Flowmaster+6 -2
Customize query in gerrit

Related Objects
Search...

Event Timeline

ssastry created this task.Jul 26 2019, 12:44 AM
Restricted Application added a project: Growth-Team. · View Herald TranscriptJul 26 2019, 12:44 AM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
ssastry added a parent task: T229015: Tracking: Direct live production traffic at Parsoid/PHP.Jul 26 2019, 12:45 AM
ssastry moved this task from Backlog to Non-parsing Team Work on the Parsoid-PHP board.Jul 26 2019, 12:49 AM
ssastry updated the task description. (Show Details)Jul 30 2019, 5:20 PM
Catrope moved this task from Inbox to Q2 2019-20 on the Growth-Team board.Jul 31 2019, 1:41 AM

If Parsoid doesn't make breaking changes that cause old HTML to fail to serialize back to wikitext (and Subbu told me that was not intended), then this should not cause any problems. We'd have to update the endpoint config but that's it, and we can test that in beta when the time comes.

ssastry triaged this task as High priority.Nov 4 2019, 4:02 AM

Parsoid-PHP is now in beta ( T231569#5540707 ). Can you please test Flow against it? We would ideally like to switch over all services to Parsoid-PHP before the week before Thanksgiving.

kostajh added subscribers: MMiller_WMF, Etonkovidova, kostajh.Nov 6 2019, 12:10 PM

cc @Etonkovidova ( and @MMiller_WMF: given @ssastry's proposed deployment we probably need to put this QA task into current sprint.)

Etonkovidova added a comment.Nov 6 2019, 10:44 PM

I just re-checked on betalabs and when posting to Flow-based pages got the error message (filed as T234242)

[XcNLYKwQBGoAADkzCXcAAAAE] Exception caught: Request to parsoid for "html" to "wikitext" conversion of content connected to title "Topic:U76p1teboaxb1a1b" failed: (curl error: 28) Timeout was reached

As soon as this is corrected, the testing should take ~2hours.

ssastry added a comment.Nov 6 2019, 11:03 PM
In T229078#5642732, @Etonkovidova wrote:

I just re-checked on betalabs and when posting to Flow-based pages got the error message (filed as T234242)

[XcNLYKwQBGoAADkzCXcAAAAE] Exception caught: Request to parsoid for "html" to "wikitext" conversion of content connected to title "Topic:U76p1teboaxb1a1b" failed: (curl error: 28) Timeout was reached

As soon as this is corrected, the testing should take ~2hours.

Thanks! I don't know what is broken with T234242 (I asked there) ... but, in order to test against Parsoid-PHP, a Flow developer needs to write some code updating the configuration to issue requests to Parsoid/PHP. AFAIK, that hasn't happened yet. @kostajh or @Catrope will need to do that.

kostajh claimed this task.Nov 7 2019, 10:31 AM
kostajh edited projects, added Growth-Team (Sprint 0 (Growth Team)); removed Growth-Team.
kostajh moved this task from Incoming to In Progress on the Growth-Team (Sprint 0 (Growth Team)) board.Nov 7 2019, 10:40 AM
gerritbot added a comment.Nov 7 2019, 10:49 AM

Change 549445 had a related patch set uploaded (by Kosta Harlan; owner: Kosta Harlan):
[mediawiki/extensions/Flow@master] Implement X-Parsoid-Variant

https://gerrit.wikimedia.org/r/549445

gerritbot added a project: Patch-For-Review.Nov 7 2019, 10:49 AM
gerritbot added a comment.Nov 7 2019, 11:20 AM

Change 549448 had a related patch set uploaded (by Kosta Harlan; owner: Kosta Harlan):
[operations/mediawiki-config@master] Flow: Configure enwiki beta to use Parsoid PHP

https://gerrit.wikimedia.org/r/549448

kostajh moved this task from In Progress to Code Review on the Growth-Team (Sprint 0 (Growth Team)) board.Nov 7 2019, 12:23 PM
gerritbot added a comment.Nov 7 2019, 9:23 PM

Change 549637 had a related patch set uploaded (by Catrope; owner: Catrope):
[operations/mediawiki-config@master] beta: Point Parsoid to parsoid-php instead of parsoid-js

https://gerrit.wikimedia.org/r/549637

gerritbot added a comment.Nov 7 2019, 9:24 PM

Change 549445 abandoned by Catrope:
Implement X-Parsoid-Variant

Reason:
Superseded by https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/ /549637

https://gerrit.wikimedia.org/r/549445

gerritbot added a comment.Nov 7 2019, 9:24 PM

Change 549448 abandoned by Catrope:
Flow: Configure enwiki beta to use Parsoid PHP

Reason:
Superseded by https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/ /549637

https://gerrit.wikimedia.org/r/549448

gerritbot added a comment.Nov 7 2019, 10:18 PM

Change 549637 merged by jenkins-bot:
[operations/mediawiki-config@master] beta: Point Parsoid to parsoid-php instead of parsoid-js

https://gerrit.wikimedia.org/r/549637

Maintenance_bot removed a project: Patch-For-Review.Nov 7 2019, 11:10 PM
Etonkovidova added a comment.Nov 8 2019, 12:23 AM

Checked Flow pages on betalabs - with T234242 being resolved, all seem to be working now.

gerritbot added a comment.Nov 8 2019, 4:01 PM

Change 549875 had a related patch set uploaded (by Mobrovac; owner: Mobrovac):
[operations/mediawiki-config@master] [Beta] Flow: Use Parsoid/PHP

https://gerrit.wikimedia.org/r/549875

gerritbot added a project: Patch-For-Review.Nov 8 2019, 4:01 PM
• mobrovac subscribed.Nov 19 2019, 3:43 PM

test.wikipedia.org and test2.wikipedia.org have been switched to serve Parsoid/PHP HTML only. Please test and report any issues you spot.

ssastry added a comment.Nov 19 2019, 3:45 PM
In T229078#5675507, @mobrovac wrote:

test.wikipedia.org and test2.wikipedia.org have been switched to serve Parsoid/PHP HTML only. Please test and report any issues you spot.

But, this probably doesn't affect Flow since Flow doesn't go through RESTBase, right? Flow still needs its own config patches.

gerritbot added a comment.Nov 20 2019, 3:40 PM

Change 549875 merged by jenkins-bot:
[operations/mediawiki-config@master] [Beta] Use Parsoid/PHP for Flow

https://gerrit.wikimedia.org/r/549875

Stashbot added a comment.Nov 20 2019, 3:42 PM

Mentioned in SAL (#wikimedia-operations) [2019-11-20T15:42:49Z] <mobrovac@deploy1001> Synchronized wmf-config/LabsServices.php: [BETA-ONLY] Switch Flow to use Parsoid/PHP - T229078 (duration: 00m 52s)

ssastry added a comment.Nov 20 2019, 3:49 PM

After that beta cluster sync, opening https://en.wikipedia.beta.wmflabs.org/wiki/Topic:Rjxs31tvmzxiisxu gives me this error:

An error has occurred while processing HTML/wikitext conversion.

Return to Main Page

[XdVgUqwQBHcAAG8WiAcAAAAQ] /wiki/Topic:Rjxs31tvmzxiisxu Flow\Exception\NoParserException from line 174 of /srv/mediawiki/php-master/extensions/Flow/includes/Conversion/Utils.php: Request to parsoid for "wikitext" to "html" conversion of content connected to title "Topic:Rjxs31tvmzxiisxu" failed: 404

Backtrace:

#0 /srv/mediawiki/php-master/extensions/Flow/includes/Conversion/Utils.php(66): Flow\Conversion\Utils::parsoid(string, string, string, Title)
#1 /srv/mediawiki/php-master/extensions/Flow/includes/Model/AbstractRevision.php(430): Flow\Conversion\Utils::convert(string, string, string, Title)
#2 /srv/mediawiki/php-master/extensions/Flow/includes/Parsoid/ContentFixer.php(41): Flow\Model\AbstractRevision->getContent(string)
#3 /srv/mediawiki/php-master/extensions/Flow/includes/Templating.php(158): Flow\Parsoid\ContentFixer->getContent(Flow\Model\PostRevision)
#4 /srv/mediawiki/php-master/extensions/Flow/includes/Formatter/RevisionFormatter.php(289): Flow\Templating->getContent(Flow\Model\PostRevision, string)
#5 /srv/mediawiki/php-master/extensions/Flow/includes/Formatter/TopicFormatter.php(45): Flow\Formatter\RevisionFormatter->formatApi(Flow\Formatter\TopicRow, Flow\View)
#6 /srv/mediawiki/php-master/extensions/Flow/includes/Block/Topic.php(648): Flow\Formatter\TopicFormatter->formatApi(Flow\Model\Workflow, array, Flow\View)
#7 /srv/mediawiki/php-master/extensions/Flow/includes/Block/Topic.php(553): Flow\Block\TopicBlock->renderTopicApi(array)
#8 /srv/mediawiki/php-master/extensions/Flow/includes/View.php(234): Flow\Block\TopicBlock->renderApi(array)
#9 /srv/mediawiki/php-master/extensions/Flow/includes/View.php(71): Flow\View->buildApiResponse(Flow\WorkflowLoader, array, string, array)
#10 /srv/mediawiki/php-master/extensions/Flow/includes/Actions/Action.php(112): Flow\View->show(Flow\WorkflowLoader, string)
#11 /srv/mediawiki/php-master/extensions/Flow/includes/Actions/ViewAction.php(20): Flow\Actions\FlowAction->showForAction(string, OutputPage)
#12 /srv/mediawiki/php-master/extensions/Flow/includes/Actions/Action.php(50): Flow\Actions\ViewAction->showForAction(string)
#13 /srv/mediawiki/php-master/includes/MediaWiki.php(514): Flow\Actions\FlowAction->show()
#14 /srv/mediawiki/php-master/includes/MediaWiki.php(304): MediaWiki->performAction(Article, Title)
#15 /srv/mediawiki/php-master/includes/MediaWiki.php(967): MediaWiki->performRequest()
#16 /srv/mediawiki/php-master/includes/MediaWiki.php(530): MediaWiki->main()
#17 /srv/mediawiki/php-master/index.php(46): MediaWiki->run()
#18 /srv/mediawiki/w/index.php(3): require(string)
#19 {main}
Maintenance_bot removed a project: Patch-For-Review.Nov 20 2019, 4:10 PM
gerritbot added a comment.Nov 20 2019, 4:55 PM

Change 552088 had a related patch set uploaded (by Mobrovac; owner: Mobrovac):
[mediawiki/core@master] Parsoid VRS: Add the Host header

https://gerrit.wikimedia.org/r/552088

gerritbot added a project: Patch-For-Review.Nov 20 2019, 4:55 PM
• mobrovac added a comment.Nov 20 2019, 5:11 PM

The above patch fixed the 404 issue.

gerritbot added a comment.Nov 20 2019, 5:21 PM

Change 552088 merged by jenkins-bot:
[mediawiki/core@master] Parsoid VRS: Add the Host header

https://gerrit.wikimedia.org/r/552088

gerritbot added a comment.Nov 20 2019, 5:26 PM

Change 552097 had a related patch set uploaded (by Mobrovac; owner: Mobrovac):
[mediawiki/core@wmf/1.35.0-wmf.5] Parsoid VRS: Add the Host header

https://gerrit.wikimedia.org/r/552097

gerritbot added a comment.Nov 20 2019, 6:12 PM

Change 552097 merged by jenkins-bot:
[mediawiki/core@wmf/1.35.0-wmf.5] Parsoid VRS: Add the Host header

https://gerrit.wikimedia.org/r/552097

Stashbot mentioned this in T229015: Tracking: Direct live production traffic at Parsoid/PHP.Nov 20 2019, 6:17 PM

Mentioned in SAL (#wikimedia-operations) [2019-11-20T18:17:25Z] <mobrovac@deploy1001> Synchronized php-1.35.0-wmf.5/includes/libs/virtualrest/ParsoidVirtualRESTService.php: Parsoid VRS: Add the Host header - T229015 T229078 T229074 (duration: 00m 52s)

Stashbot mentioned this in T229074: Preparing VisualEditor for Parsoid-PHP switch.Nov 20 2019, 6:17 PM
ssastry added a comment.Nov 20 2019, 6:49 PM

@Etonkovidova Can you run your Flow tests on Beta cluster again? Basic tests as well as more complex tests to make sure all content use cases continue to work.

Maintenance_bot removed a project: Patch-For-Review.Nov 20 2019, 7:10 PM
kostajh moved this task from Code Review to QA on the Growth-Team (Sprint 0 (Growth Team)) board.Nov 26 2019, 3:57 PM
ssastry added a comment.Nov 26 2019, 9:29 PM

Note that Flow is not yet on group 1 wikis. Only on group 0 wikis (so, mw.org, officewiki, test and test2 wikis). T229015 has the deployment timeline.

Etonkovidova added a comment.Nov 26 2019, 10:36 PM
In T229078#5679406, @ssastry wrote:

@Etonkovidova Can you run your Flow tests on Beta cluster again? Basic tests as well as more complex tests to make sure all content use cases continue to work.

Done on beta cluster and testwiki - all seems to be good. Will re-check after T229015.

ssastry closed this task as Resolved.Nov 27 2019, 3:55 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL