So, the open questions while making this task:

@MoritzMuehlenhoff can the php extension be updated on just a single mwdebug server / the mwdebug servers? I.e. Not have to perform an update on the whole fleet?

@Bmueller @Tobi_WMDE_SW @JStrodt_WMDE As the rollout plan, do we want to go for group0, then dewiki, then group1 and then all sites? Or do we have something else in mind?

In T177891#3674116, @Addshore wrote:

So, the open questions while making this task:

@Bmueller @Tobi_WMDE_SW @JStrodt_WMDE As the rollout plan, do we want to go for group0, then dewiki, then group1 and then all sites? Or do we have something else in mind?

No, that sounds good to me. But maybe we should wait a week after group0 to see if all goes well?

In T177891#3674116, @Addshore wrote:

So, the open questions while making this task:

@MoritzMuehlenhoff can the php extension be updated on just a single mwdebug server / the mwdebug servers?

Yeah, that's not a problem.

Before this gets deployed, the wikidiff package needs to be updated to 1.5.0, the experimental package only cherrypicked the new "detect moves" functionality, but since then there's been a 1.5.0 release, so let's update to that while we're at it.

@MoritzMuehlenhoff T176485 has just been merged, so it would be super great if we could also include https://gerrit.wikimedia.org/r/#/c/380952/ in the wikidiff2 version that will be deployed. I guess another tag/version has to be done for this?

In T177891#3678968, @Tobi_WMDE_SW wrote:

@MoritzMuehlenhoff T176485 has just been merged, so it would be super great if we could also include https://gerrit.wikimedia.org/r/#/c/380952/ in the wikidiff2 version that will be deployed.

Sure, no problem.

I guess another tag/version has to be done for this?

Ideally yes, otherwise it's always a constant source of confusion since people need to look into the debian/changelog to spot the exact changes. I think @Legoktm prepared the 1.5.0 release, adding him on CC to see whether he can tag a 1.5.1. Otherwise I'll simply cherrypick the patch.

@MoritzMuehlenhoff sounds great! Thanks

I tagged 1.5.1 and updated the packaging/changelog: https://gerrit.wikimedia.org/r/#/c/383907/

Thanks @Legoktm!
@MoritzMuehlenhoff Which would be the next steps in order to get this to production?

@Tobi_WMDE_SW : I've built and uploaded 1.5.1 to apt.wikimedia.org and upgraded the app servers in deployment-prep. Adam run some tests and it seems fine according to his tests. I'll upgrade canary app servers in production tomorrow.

We can only update wgWikiDiff2MovedParagraphDetectionCutoff once all app servers are upgraded (which will take a little, since we need to prune the HHVM byte code cache to prevent the 503s we've seen in beta). I'll keep this task updated.

In T177891#3687556, @MoritzMuehlenhoff wrote:

We can only update wgWikiDiff2MovedParagraphDetectionCutoff once all app servers are upgraded (which will take a little, since we need to prune the HHVM byte code cache to prevent the 503s we've seen in beta). I'll keep this task updated.

We could set this based on hostname though?
This could also be limited per wiki and we could start with testwiki or just group0?

if( gethostname() === 'mwdebug1001' ) {
    // Set setting
}

I'm not against just doing the version bump everywhere and moving forward from there, just a thought.

We could set this based on hostname though?
This could also be limited per wiki and we could start with testwiki or just group0?

Probably yes, but I'm not familiar with that part of wmf-config change deployment, I'll leave to people more involved with that.

@MoritzMuehlenhoff Great! Yes, please keep us updated for when the app servers caught up. I think it makes sense then to start with rolling the setting out to group0 as @Addshore suggested. Do we need another ticket for that? AFAICS the steps are documented in the description in this ticket.

@hashar @greg do you see any reason that I shouldn't set wgWikiDiff2MovedParagraphDetectionCutoff based on the return of gethostname for group0 so that we can test this while it is only deployed to mwdebug servers?

@MoritzMuehlenhoff what is the state of deployment to app servers?

@Tobi_WMDE_SW , @Addshore : wikidiff2 1.5.1 is now rolled out in production across all our mediawiki, you can proceed with the wgWikiDiff2MovedParagraphDetectionCutoff from my PoV.

In T177891#3711928, @MoritzMuehlenhoff wrote:

@Tobi_WMDE_SW , @Addshore : wikidiff2 1.5.1 is now rolled out in production across all our mediawiki, you can proceed with the wgWikiDiff2MovedParagraphDetectionCutoff from my PoV.

Thanks!

I updated the description with the 4 remaining config steps and we will probably start these next week!

@Addshore go forth and test on group0

Change 387198 had a related patch set uploaded (by Addshore; owner: Addshore):
[operations/mediawiki-config@master] Set wgWikiDiff2MovedParagraphDetectionCutoff for group0

https://gerrit.wikimedia.org/r/387198

Change 387198 merged by jenkins-bot:
[operations/mediawiki-config@master] Set wgWikiDiff2MovedParagraphDetectionCutoff for group0

https://gerrit.wikimedia.org/r/387198

So, I just tried deploying the above change.
While testing on mwdebug1002 the config switch didn't appear to enable the functionality in the extension.
I rolled back the change to make way for the rest of SWAT and may look into this further during another deploy slot this week.

The wikidiff2 version looks fine:

addshore@mwdebug1002:/srv/mediawiki/wmf-config$ php -r "var_dump(phpversion('wikidiff2'));"
string(5) "1.5.1"

The code made it to mwdebug1002.
If anyone spots anything amiss please say!

In T177891#3719471, @Addshore wrote:

So, I just tried deploying the above change.
While testing on mwdebug1002 the config switch didn't appear to enable the functionality in the extension.
I rolled back the change to make way for the rest of SWAT and may look into this further during another deploy slot this week.

The wikidiff2 version looks fine:

addshore@mwdebug1002:/srv/mediawiki/wmf-config$ php -r "var_dump(phpversion('wikidiff2'));"
string(5) "1.5.1"

The code made it to mwdebug1002.
If anyone spots anything amiss please say!

I was wondering if it could be a PHP/HHVM difference, but I ran hhvm -a and the output of the version was 1.5.1, as expected. I also noticed that in core it was looking for 0.3.0 or greater, but the version it was included in was 1.5.0, so I submitted https://gerrit.wikimedia.org/r/#/c/387836/

What's the best way to test this feature locally/in prod? Is there a sample set of text that makes it obvious to notice the difference in diffs?

@Legoktm Just make a change that moves a paragraph.
Here is one of my tests on testwiki not working while I was testing the deployment. https://test.wikipedia.org/w/index.php?diff=338364&oldid=338363&title=Page332
Here is a test on beta with the feature working https://deployment.wikimedia.beta.wmflabs.org/w/index.php?title=Test02&diff=prev&oldid=7640

When working you can see the ⚫character markers instead of +s or -s

Wondering why it worked on beta, it should have been broken since https://gerrit.wikimedia.org/r/#/c/377804/ I guess.

In T177891#3727852, @Tobi_WMDE_SW wrote:

Wondering why it worked on beta, it should have been broken since https://gerrit.wikimedia.org/r/#/c/377804/ I guess.

Well, I don't think the change @Legoktm really changes anything, but it should be / needs to be done.

@Addshore right! as long as it is > 0.3.0 it should work.

So I got it to eventually work on testwiki + mwdebug1002 - it's a caching problem. I think it should be safe to enable and let it sit for a while. We should also consider bumping DifferenceEngine::DIFF_VERSION or including the wikidiff2 version in the cache key.

Also, if you want to view a diff uncached just use show changes in action=edit or click "undo" on an edit.

Change 387978 had a related patch set uploaded (by Legoktm; owner: Legoktm):
[operations/mediawiki-config@master] Revert "Revert "Set wgWikiDiff2MovedParagraphDetectionCutoff for group0""

https://gerrit.wikimedia.org/r/387978

> Also, if you want to view a diff uncached just use show changes in action=edit or click "undo" on an edit.

I didn't try previewing in the action=edit screen.
But wouldn't a new edit not have any issue with caching?

We should also consider bumping DifferenceEngine::DIFF_VERSION or including the wikidiff2 version in the cache key.

Sounds like a good idea

In T177891#3728971, @Addshore wrote:

But wouldn't a new edit not have any issue with caching?

I had a lot of trouble with cache busting last night even after live-hacking the code, so I'm not sure. I also ran into a bug in X-Wikimedia-Debug which might have caused some problems, but action=edit always worked for me.

Change 387978 merged by jenkins-bot:
[operations/mediawiki-config@master] Revert "Revert "Set wgWikiDiff2MovedParagraphDetectionCutoff for group0""

https://gerrit.wikimedia.org/r/387978

Deployed the config change to group0.
Again when i pulled the config onto mwdebug1002 only i couldnt make the feature work.
So I also did a scap pull onto tin and made sure the config all looked fine in eval.php and did a little testing before going ahead and rolling it out everywhere.

Now we can do some testing on testwiki and look at further rollout in the coming weeks.

Woohh.. first test looking good on test.wikipedia.org
https://test.wikipedia.org/w/index.php?title=Test_page_long&type=revision&diff=338480&oldid=323544

Is this set of changes the reason that I'm seeing great big "⚫ " s for some diffs but still "-"s and "+"s for others in the same diff (e.g. https://www.mediawiki.org/w/index.php?title=Extension:3D&diff=2612241&oldid=2574782 or https://test.wikipedia.org/w/index.php?title=Test_page_long&type=revision&diff=338480&oldid=323544 ) on group0 wikis and I can't replicate this on other wikis (yet?). Was going to file a UBN.

Yes, it's only on group0 wikis for now. https://gerrit.wikimedia.org/r/#/c/386387/ switches the dots to arrows, but it didn't make the train.

In T177891#3743342, @Legoktm wrote:

Yes, it's only on group0 wikis for now. https://gerrit.wikimedia.org/r/#/c/386387/ switches the dots to arrows, but it didn't make the train.

Then I guess we can cherry pick that to the wmf branch? :)

Then I guess we can cherry pick that to the wmf branch? :)

+1 @Addshore

Yup, my issue is all fixed now, thanks!

Before we continue to roll it further out we need to tackle T179277, T166882 and T180043.

In Timeless if you hover over the arrows the dots show up underneath: https://www.mediawiki.org/w/index.php?diff=2608842&oldid=2001105&title=Requests_for_comment/ChangesList_formatting&useskin=timeless

Also the RSS feeds just show the dots since no styles are applied: https://www.mediawiki.org/w/api.php?hidebots=1&days=14&limit=50&target=Category%3AOpen_requests_for_comment&translations=filter&action=feedrecentchanges&feedformat=atom&hidecategorization=1 though I think that's probably OK, it's just a limitation of the RSS format.

In T177891#3765457, @Legoktm wrote:

Also the RSS feeds just show the dots since no styles are applied: https://www.mediawiki.org/w/api.php?hidebots=1&days=14&limit=50&target=Category%3AOpen_requests_for_comment&translations=filter&action=feedrecentchanges&feedformat=atom&hidecategorization=1 though I think that's probably OK, it's just a limitation of the RSS format.

I'm not sure if it's actually possible for this feature, but it might be avoidable if you can add appropriate replacements to FeedUtils::applyDiffStyle().

Also the RSS feeds just show the dots since no styles are applied: https://www.mediawiki.org/w/api.php?hidebots=1&days=14&limit=50&target=Category%3AOpen_requests_for_comment&translations=filter&action=feedrecentchanges&feedformat=atom&hidecategorization=1 though I think that's probably OK, it's just a limitation of the RSS format.

I'm not sure if it's actually possible for this feature, but it might be avoidable if you can add appropriate replacements to FeedUtils::applyDiffStyle().

Or we might as well just set/replace the symbols completely on PHP side and avoid having CSS do it in the first place... ^^'