Hi @MoritzMuehlenhoff, CC'ing you per T285856#7294168. Do you need anything else from us to move this forward?

@jijiki Hi, CC'ing you since you deployed the previous versions linked in the task description.

Hi @jcrespo , are you the correct person to reach out to for assistance with triaging this task? It looks like you're the contact listed on SRE Clinic Duty for this week. Please let us know if you need any more information to push this forward. Thanks! cc @Daimona

Hey, @ldelench_wmf I am indeed the first point of contact for SREs this week. Both @jijiki and @MoritzMuehlenhoff maybe temporarily unavailable or on vacations. However, it is unclear based on the ticket, what help you need from us (I don't have the context other workmates may have)?

Is this a new package that needs to be installed? Where- mediawiki servers?

Edit: I just read T236963 and this seems like a delicate mediawiki-related server process, so serviceops are the people that will know how to properly handle it. I will ping the right people to attend you.

Great, thank you for your help connecting us with serviceops @jcrespo ! @Daimona is the engineer leading this project; he will be better equipped than I to address any open questions. :)

Don't worry, pings have been sent to the right people- and the manager confirmed someone will get back to you soon. Please ping me again if it doesn't happen. Apologies for the delays, but it happend that the 2 people pinged in the first place were not available back then (August is a common vacation month).

Platform Engineering will take this, but if there are complications, we'll be back...

Thank you @ArielGlenn

@ArielGlenn Thanks for taking over this. Let us know if you need any help!

Appreciate everyone's help with this! @ArielGlenn this came up at a CommTech retro the other day - do you have an idea when this will be worked on? Knowing the rough timeframe will help us set expectations within our team for when QA can happen. Thanks!

In T285857#7343025, @ldelench_wmf wrote:

Appreciate everyone's help with this! @ArielGlenn this came up at a CommTech retro the other day - do you have an idea when this will be worked on? Knowing the rough timeframe will help us set expectations within our team for when QA can happen. Thanks!

I don't know right now, but I should know something next week, even if it's "there will be a delay".

Hi @ldelench_wmf, this is scheduled as part of our ops work. It will require some review from us before we can commit to a when. Our goal is to complete the review by the end of this week. If it's straightforward we'll do it immediately but we'll make sure to update here by EOD Monday 20th. However, the week of the 20th is a Code Jam focused week for us so if we can not complete it within this week. The deployment will have to take place in the week of the 27th.

Sorry I should also have asked: how does this impact your timelines? Do we need to escalate this?

Is there an expectation that there is going to be more regular work on Wikidiff2 planned or anymore expected work right now?

Hi @WDoranWMF, thanks for the heads up! That timeline should work. We don't expect more work on Wikidiff2 beyond this current epic.

+1 to Lauren.

@WDoranWMF of note, this does not need to be escalated if there are higher priorities, but the sooner for us the better-- we are currently blocked on getting this wish out to users and would love to do it as soon as possible (without anyone overworking or delaying important priorities)

@WDoranWMF I'd ask you to please wait for the deployment until T290731 is resolved and a new version is released (T291187). This task will then be about deploying the new version (1.12.1, I guess, or 1.13.0).

@Daimona can you let @hnowlan once everything is ready to be deployed and please update the task description with this information. We'll pause until you let us know.

In T285857#7359608, @WDoranWMF wrote:

@Daimona can you let @hnowlan once everything is ready to be deployed and please update the task description with this information. We'll pause until you let us know.

Of course, thank you. BTW, if PET would like to review the changes to wikidiff2 ahead of the deployment, here is the full list (one is still under review).

@ldelench_wmf @NRodriguez awesome, we'll our best to get it done as soon as it is unblocked.

@WDoranWMF, @hnowlan Hello, this is now unblocked and ready to go. Note that the version of wikidiff2 to be deployed is now 1.13.0, not 1.12.0. Thanks!

wikidiff 1.13.0 is now installed on the beta cluster.

@hnowlan please remember to also rebuild the corresponding docker image when rolling out to production, so that it's picked up in kubernetes as well.

Thanks @hnowlan. @dom_walden, @imaigwilo you should now be able to test the related tickets (T290731, T285793, T285776, T288676, T192526) on the beta cluster.

Has this been rolled out further since? T292762: High loading times on no.wikipedia is reporting generating diffs is slower.

In T285857#7410752, @Legoktm wrote:

Has this been rolled out further since? T292762: High loading times on no.wikipedia is reporting generating diffs is slower.

No. This can be confirmed by browsing any diff and noting that the side-locking doesn't work, or that diff markers are not in the data-marker attribute.

Hello there @Legoktm we just resolved https://phabricator.wikimedia.org/T290731#7455461 which should unblock this ticket! Hope that's helpful and thanks again!

• Beta cluster
• QAed on the beta cluster
• Prod deployment
• Docker image rebuilt

This has been tested on the beta cluster.

There is one outstanding bug in code review (T293917).

@NRodriguez I don't know if you want to put this back into Ready until we do the production deployment.

Hey @hnowlan @WDoranWMF, just want to make sure you saw the updates above. This is now ready for deployment in production.

Mentioned in SAL (#wikimedia-operations) [2021-11-02T14:15:20Z] <hnowlan> debdeploying wikidiff2-1.13.0-1 to A:mw-app-canary and A:mw-api-canary for T285857

I've deployed wikidiff2-1.13.0-1 to the canaries and will deploy to the rest of production tomorrow. For reference this is the debdeploy file used:

hnowlan@cumin1001:~$ cat 2021-11-01-wikidiff2.yaml
comment: RTL compatibility fixes T285857
fixes:
  bullseye: ''
  buster: 1.13.0-1
  stretch: ''
libraries: []
source: wikidiff2
transitions: {}
update_type: library

The deployment is just finishing up to codfw's API servers in the next few minutes, all others are complete. Please let me know if the changes are present and whether there are any issues!

PHP 7.4 images have been bumped via https://gerrit.wikimedia.org/r/plugins/gitiles/operations/docker-images/production-images/+/4de99a08bc2eb983f3aca798ff33fd22976c65e8