1.36.0-wmf.2 deployment blockers
Closed, ResolvedPublicRelease
Actions

Details

Backup Train Conductor: brennen
Release Version: 1.36.0-wmf.2
Release Date: Jul 27 2020, 12:00 AM

2020 week 31 1.36-wmf.2 Changes wmf/1.36.0-wmf.2

This MediaWiki Train Deployment is scheduled for the week of Monday, July 27th:

Monday July 27th	Tuesday, July 28th	Wednesday, July 29th	Thursday, July 30th	Friday
Backports only.	Branch `wmf.2` and deploy to Group 0 Wikis.	Deploy `wmf.2` to Group 1 Wikis.	Deploy `wmf.2` to all Wikis.	No deployments on fridays

See https://wikitech.wikimedia.org/wiki/Deployments for full schedule.

How this works

Any serious bugs affecting wmf.2 should be added as subtasks beneath this one.
- Use this form to create one.
Any open subtask(s) block the train from moving forward. This means no further deployments until the blockers are resolved.
If something is serious enough to warrant a rollback then you should bring it to the attention of deployers on the #wikimedia-operations IRC channel.
If you have a risky change in this week's train add a comment to this task using the Risky patch template
For more info about deployment blockers, see Holding the train.

Other Deployments

Previous: 1.36.0-wmf.1

Next: 1.36.0-wmf.3

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved	Release	• LarsWirzenius	T257970 1.36.0-wmf.2 deployment blockers
		Resolved	PRODUCTION ERROR	ssastry	T259311 Argument 1 passed to Wikimedia\Parsoid\Utils\DOMDataUtils::getDataMw() must be an instance of DOMElement

Event Timeline

• mmodell created this task.Jul 14 2020, 8:18 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 14 2020, 8:18 PM

thcipriani assigned this task to • LarsWirzenius.Jul 21 2020, 5:33 PM

thcipriani triaged this task as Medium priority.

thcipriani updated Other Assignee, added: brennen.

thcipriani added a project: Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1)).

thcipriani moved this task from INBOX to Deployment on the Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1)) board.

BEANS-X2 subscribed.Jul 27 2020, 7:59 AM

brennen added a project: User-brennen.Jul 27 2020, 5:46 PM

brennen moved this task from Backlog to Next on the User-brennen board.

brennen subscribed.

Mentioned in SAL (#wikimedia-operations) [2020-07-28T07:03:31Z] <liw> 1.36.0-wmf.2 was branched at rMW04e863fdf364 for T257970

taavi subscribed.Jul 28 2020, 7:05 AM

Automatic branch cut went OK. Ran the pre-deployment-window steps (scap prep, applied security patches if there were any, deployed to testwikis).

Promoted to group0.

Filed https://phabricator.wikimedia.org/T259022 but didn't make it a blocker.

Filed T259023: PHP Notice: Undefined index: content-disposition as a blocker.

Unblocked, because T259023: PHP Notice: Undefined index: content-disposition doesn't seem to be bad enough to be a blocker. Also, hopefully has a fix now, waiting for review.

brennen moved this task from Next to Doing on the User-brennen board.Jul 28 2020, 3:59 PM

thcipriani added a subtask: T259023: PHP Notice: Undefined index: content-disposition.Jul 28 2020, 4:47 PM

brennen removed a subtask: T259023: PHP Notice: Undefined index: content-disposition.Jul 28 2020, 6:09 PM

Keeping one eye on it, but now unblocked and seems as calm as should be expected for group0. Will re-assign to Lars at end-of-local-workday.

• LarsWirzenius claimed this task.Jul 29 2020, 9:00 AM

Noted that T157997: BannerExistenceException due to non-existing CentralNotice banner (after Special:LanguageStats view) seems to have happened a couple of times.

Reported T259111: PHP Notice: unserialize(): Error at offset 65519 of 65535 bytes (in LiquidThreads), but it's only happened a couple of times so far, so not as a blocker.

Promoted to group1.

Rolled back to group0 due to T259111: PHP Notice: unserialize(): Error at offset 65519 of 65535 bytes (in LiquidThreads) occurring too much.

Correction: it was T259126: Warning: Locally stored wiki page has unsupported content model (from JsonConfig) happening too much, not T258111: [BUG] Investigate Drop in User Count and Activity for Image Tags. Sorry.

• LarsWirzenius added a subtask: T259175: MediaWiki::restInPeace: transaction round 'UpdateBetaFeatureUserCountsJob::run' still running.Jul 29 2020, 6:25 PM

We reported T157997: BannerExistenceException due to non-existing CentralNotice banner (after Special:LanguageStats view) but it's not a blocker as it doesn't currently affect this week's train.

brennen removed a subtask: T259175: MediaWiki::restInPeace: transaction round 'UpdateBetaFeatureUserCountsJob::run' still running.Jul 29 2020, 7:23 PM

We've reached today's North America cutoff for deploys. Assuming a resolution on T259126, train can resume in Lars' local morning.

The fix for T259126 is backported to wmf/1.36.0-wmf.2 and the train should be good to move forward in the (EU) morning.

Ladsgroup subscribed.Jul 30 2020, 6:34 AM

Promoted back to group1, out of train window. Wanted to do this early so that if everything seems OK for a few hours, we (me or Brennen) can promote to group2 optimistic-opportunistically. If that fails, we roll back to group1 and try again on Monday if any problems found have been fixed.

This deployment's sound track was Queen's "We will rock you".

• LarsWirzenius added a subtask: T259246: Class 'CentralAuthUser' not found.Jul 30 2020, 11:09 AM

• LarsWirzenius removed a subtask: T259126: Warning: Locally stored wiki page has unsupported content model (from JsonConfig).

Reported T259247: Uncaught Psr\Log\InvalidArgumentException: Level "400" is not defined, use one of: 0, 1, 2 in /srv/mediawiki/php-1.36.0-wmf.2/vendor/monolog/monolog/src/Monolog/Logger.php:516 but not as a blocker at this time.

Urbanecm removed a subtask: T259246: Class 'CentralAuthUser' not found.Jul 30 2020, 11:33 AM

brennen added a subtask: T258609: Circular dependency when creating service! GrowthExperimentsConfigurationLoader -> GrowthExperimentsConfigurationLoader.Jul 30 2020, 7:19 PM

brennen claimed this task.Jul 30 2020, 7:31 PM

brennen added a subtask: T259311: Argument 1 passed to Wikimedia\Parsoid\Utils\DOMDataUtils::getDataMw() must be an instance of DOMElement.Jul 30 2020, 10:23 PM

Rolled out with fix for T258609 at 21:52 UTC, noticed T259311 and rolled back to group1 by 22:12. Calling it for the week, sending train status mail.

brennen removed a subtask: T258609: Circular dependency when creating service! GrowthExperimentsConfigurationLoader -> GrowthExperimentsConfigurationLoader.Jul 30 2020, 10:36 PM

It's the end of my working day Eastern Time Thursday. From the above, I'm assuming it's safe to wrap up my day and I'll get T259311 fixed tomorrow (Friday) and do the parsoid-packaging dance required so that you can bump parsoid to v0.13.0-a3 on Monday for wmf.2 and try again. cc @Arlolra, @Sbailey.

Let me know if that timeframe needs to be accelerated.

@cscott sounds good - thanks.

(For clarity: No need to accelerate.)

Jdlrobson subscribed.Jul 31 2020, 9:04 PM

Logstash looks fairly OK, though noted T257504: Argument 1 passed to Wikimedia\Parsoid\Ext\ParsoidExtensionAPI::Wikimedia\Parsoid\Ext\{closure}() must be of the type integer, null given in ParsoidExtensionAPI.php on line 849 is happening again/still. Promoting to group2.

It looks like the train was rolled without the fix for T259311, which got deployed to master but not cherry-picked to wmf.2 before the train was rolled. Cherry-pick is at https://gerrit.wikimedia.org/r/c/mediawiki/vendor/+/618068 . This is probably not super-high-importance, given that wmf.3 is at our doorstep and it also has this fix in it. We'll make sure wmf.3 has a fix for T257504 as well.

Filed T259536: Call to a member function purgeCache() on null.

Seeing several T259536 a minute. I don't know the impact there, but it's a fatal. Loathe as I am to roll back the train on a Monday, it feels like something that should probably be rolled back.

Agreed, rolling back.

brennen added a subtask: T259536: Call to a member function purgeCache() on null.Aug 3 2020, 5:07 PM

brennen removed a subtask: T259536: Call to a member function purgeCache() on null.Aug 3 2020, 5:25 PM

Ok, rolling forward again.

ssastry closed subtask T259311: Argument 1 passed to Wikimedia\Parsoid\Utils\DOMDataUtils::getDataMw() must be an instance of DOMElement as Resolved.Aug 3 2020, 6:13 PM

brennen closed this task as Resolved.Aug 4 2020, 3:09 PM

brennen moved this task from Doing to Done or Declined on the User-brennen board.

1.36.0-wmf.2 deployment blockersClosed, ResolvedPublicReleaseActions