Page MenuHomePhabricator

Deploy Wikidiff2 version 1.8.2 with the timeout issue fixed
Closed, ResolvedPublic2 Story Points

Description

To do:

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 15 2019, 4:40 PM
thiemowmde updated the task description. (Show Details)May 15 2019, 4:40 PM
thiemowmde set the point value for this task to 3.
WMDE-Fisch updated the task description. (Show Details)May 17 2019, 2:14 PM
ori awarded a token.May 17 2019, 2:47 PM
MoritzMuehlenhoff removed MoritzMuehlenhoff as the assignee of this task.May 20 2019, 7:32 AM

I don't have time for this in the next 4-6 weeks, I'm removing myself for now and tagging this with ServiceOps as well.

thiemowmde added a subscriber: Tobi_WMDE_SW.

@Tobi_WMDE_SW, can you take care of this and possibly assign people who are able to deploy this, instead of @MoritzMuehlenhoff? Thanks!

Joe assigned this task to jijiki.Jun 18 2019, 9:34 AM
Joe triaged this task as High priority.
jijiki moved this task from Backlog/Radar to In Progress on the User-jijiki board.Jun 18 2019, 9:45 PM
Joe moved this task from Backlog to Doing on the serviceops board.Jun 21 2019, 7:36 AM
jijiki lowered the priority of this task from High to Medium.Jun 25 2019, 2:58 PM

@thiemowmde @WMDE-Fisch I have installed php-wikidiff2_1.8.2-1~wmf1_amd64 on deployment-mediawiki-09 and deployment-mediawiki-07 but I have not restarted php72-fpm there. You can restart it when you are ready to test it. Let me know how it went so to proceed with the production upgrade:)

jijiki updated the task description. (Show Details)Jun 25 2019, 3:00 PM

Mentioned in SAL (#wikimedia-releng) [2019-06-25T15:50:50Z] <awight> deployment-prep restart php7.2-fpm for wikidiff2 upgrade (T223391)

Mentioned in SAL (#wikimedia-cloud) [2019-06-25T15:51:49Z] <awight> restart php7.2-fpm for wikidiff2 upgrade (T223391)

Smoke testing the diffs shows that nothing seems to have been broken by the upgrade. We haven't been able to verify the fixes yet, still working on that.

WMDE-Fisch changed the point value for this task from 3 to 2.

Confirmed that the old version of wikidiff2 will fall back to not computing moved lines with more than 30 moves: https://test.wikipedia.org/w/index.php?title=User:Adamw/diff40&diff=398150&oldid=398149

Confirmed that the new version will attempt to compute moves for these 40 lines of diff: https://en.wikipedia.beta.wmflabs.org/w/index.php?title=User:Adamw/diff40&diff=394821&oldid=394820&diffmode=source

Old code showing 200 lines moved: https://test.wikipedia.org/w/index.php?title=User%3AAdamw%2Fdiff_large_chunks&type=revision&diff=398154&oldid=398153

New code showing 200 lines moved, when I expected it to fail: https://en.wikipedia.beta.wmflabs.org/w/index.php?title=User:Adamw/diff_large_chunks&diff=394823&oldid=394822&diffmode=source#movedpara_0_98_lhs

I can say, the new release seems harmless and we should go ahead with deployment, but I wasn't able to convince myself that it will solve the timeout issue.

@awight I will rollout the new version to production today

Mentioned in SAL (#wikimedia-operations) [2019-06-28T07:11:00Z] <_joe_> upgrading php-wikidiff2 on the mw canaries, only on php7 - T223391

Joe claimed this task.Jun 28 2019, 7:23 AM
Joe added subscribers: jijiki, Joe.

I did rollout the new version on the canary servers today.

If I don't see higher error rates on monday, I'll deploy to the whole fleet.

@jijiki We're a bit confused because the beta cluster reports it's running HHVM and there's no beta feature to select PHP7. Can you advise us on how to access the PHP7 backend?

Joe added a comment.Jun 28 2019, 1:20 PM

@jijiki We're a bit confused because the beta cluster reports it's running HHVM and there's no beta feature to select PHP7. Can you advise us on how to access the PHP7 backend?

You can add a PHP_ENGINE=php7 cookie to your requests.

@jijiki We're a bit confused because the beta cluster reports it's running HHVM and there's no beta feature to select PHP7. Can you advise us on how to access the PHP7 backend?

You can add a PHP_ENGINE=php7 cookie to your requests.

Thanks!

I think we can probably use the X-Wikimedia-Debug browser plugin to test on mwdebug[12]00[12], and enable PHP7 for test.wikipedia.org now that you've deployed to canaries, too?

Joe added a comment.Jun 28 2019, 1:24 PM

Yes :) It's by far the best option.

Certainly seems to be the case.

Good news, with PHP7 enabled I get the expected results where the new version of wikidiff2 refuses to calculate moves for https://test.wikipedia.org/w/index.php?title=User:Adamw/diff_large_chunks&diff=398154&oldid=398153

@jkroll ^

jijiki claimed this task.EditedJul 2 2019, 7:13 AM

@awight @jkroll should we rollout to all hosts?

Joe added a comment.Jul 2 2019, 8:34 AM

@awight @jkroll should we rollout to all hosts?

I think we should, I didn't get to it yesterday. So we should deploy everywhere and then do a rolling restart of php-fpm.

@awight @jkroll should we rollout to all hosts?

I think we should, I didn't get to it yesterday. So we should deploy everywhere and then do a rolling restart of php-fpm.

+1 please go on with the deploy from our side it should be good

Mentioned in SAL (#wikimedia-operations) [2019-07-02T10:45:07Z] <jijiki> Rollout Wikidiff 1.8.2 to codfw - T223391

Mentioned in SAL (#wikimedia-operations) [2019-07-02T10:47:43Z] <jijiki> Rollout Wikidiff 1.8.2 to eqiad - T223391

Mentioned in SAL (#wikimedia-operations) [2019-07-02T11:26:04Z] <jijiki> Run restart-php-fpm in all-mw-eqiad - T223391

Mentioned in SAL (#wikimedia-operations) [2019-07-02T14:58:41Z] <jijiki> Run restart-php-fpm in all-mw-codfw - T223391

jijiki closed this task as Resolved.Jul 2 2019, 3:02 PM

All service restarts should be complete in a few hours, please reopen if there are any issues :)