1.36.0-wmf.27 deployment blockers
Closed, ResolvedPublicRelease
Actions

Details

Backup Train Conductor: • LarsWirzenius
Release Version: 1.36.0-wmf.27
Release Date: Jan 18 2021, 12:00 AM

2021 week 03 1.36-wmf.27 Changes wmf/1.36.0-wmf.27

This MediaWiki Train Deployment is scheduled for the week of Monday, January 18th:

Monday January 18th	Tuesday, January 19th	Wednesday, January 20th	Thursday, January 21st	Friday
Backports only.	Branch `wmf.27` and deploy to Group 0 Wikis.	Deploy `wmf.27` to Group 1 Wikis.	Deploy `wmf.27` to all Wikis.	No deployments on fridays

See https://wikitech.wikimedia.org/wiki/Deployments for full schedule.

How this works

Any serious bugs affecting wmf.27 should be added as subtasks beneath this one.
- Use this form to create one.
Any open subtask(s) block the train from moving forward. This means no further deployments until the blockers are resolved.
If something is serious enough to warrant a rollback then you should bring it to the attention of deployers on the #wikimedia-operations IRC channel.
If you have a risky change in this week's train add a comment to this task using the Risky patch template
For more info about deployment blockers, see Holding the train.

Other Deployments

Previous: 1.36.0-wmf.26

Next: 1.36.0-wmf.28

	Subject	Repo	Branch	Lines +/-
	Branch commit for wmf/1.36.0-wmf.27	mediawiki/core	wmf/1.36.0-wmf.27	+971 -3

Customize query in gerrit

Related Objects
Search...

Status	Subtype	Assigned	Task
Resolved	Release	brennen	T271341 1.36.0-wmf.27 deployment blockers
Resolved		Daimona	T272248 AbuseFilter-triggered API errors on Beta Commons: FilterRunner::checkFilter() must be an instance of ExistingFilter, instance of Filter given
Resolved	PRODUCTION ERROR	abi_	T272428 Error 1146: Table 'mediawikiwiki.translate_cache' doesn't exist
Resolved	PRODUCTION ERROR	Daimona	T272505 FilterLookup: No version of filter [x] closest to [y] found

Event Timeline

thcipriani created this task.Jan 6 2021, 4:54 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 6 2021, 4:54 PM

Risky Patch! 🚂🔥

Change: https://gerrit.wikimedia.org/r/c/mediawiki/vendor/+/621953 / https://gerrit.wikimedia.org/r/c/mediawiki/core/+/644834
What it does/What it's risky:
- Monolog is critical and not well tested, a poor combination. The last time we upgraded it (which was a much riskier change) broke logging and resulted in a rollback. From manual testing it looks OK, but production isn't exactly like Beta Cluster.
Test Plan:
- Manual checking that production logstash contents look correct.
Places to Watch for Breakage
- Logstash MediaWiki Errors Dashboard: https://logstash.wikimedia.org/app/kibana#/dashboard/mediawiki-errors
- Grafana Production Logging Dashboard: https://grafana.wikimedia.org/d/000000102/production-logging
Revert Plan: rollback
Wikis Affected: All
IRC Contact: @Reedy or @James_F
UBN Task Projects/tags: Wikimedia-Logstash

thcipriani assigned this task to brennen.Jan 13 2021, 12:50 AM

thcipriani triaged this task as Medium priority.

thcipriani updated Other Assignee, added: • LarsWirzenius.

thcipriani added a project: Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)).

thcipriani moved this task from INBOX to Deployment on the Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)) board.

RhinosF1 subscribed.Jan 16 2021, 10:18 AM

AbuseFilter seems to be having issues on the Beta cluster – doesn’t look like something that would be Beta-specific, so probably worth keeping an eye on: T272248: AbuseFilter-triggered API errors on Beta Commons: FilterRunner::checkFilter() must be an instance of ExistingFilter, instance of Filter given

Jdforrester-WMF added a subtask: T272248: AbuseFilter-triggered API errors on Beta Commons: FilterRunner::checkFilter() must be an instance of ExistingFilter, instance of Filter given.Jan 17 2021, 11:39 PM

Daimona closed subtask T272248: AbuseFilter-triggered API errors on Beta Commons: FilterRunner::checkFilter() must be an instance of ExistingFilter, instance of Filter given as Resolved.Jan 17 2021, 11:58 PM

Ladsgroup subscribed.Jan 18 2021, 12:39 PM

Change 656987 had a related patch set uploaded (by DannyS712; owner: trainbranchbot):
[mediawiki/core@wmf/1.36.0-wmf.27] Branch commit for wmf/1.36.0-wmf.27

https://gerrit.wikimedia.org/r/656987

gerritbot added a project: Patch-For-Review.Jan 19 2021, 2:12 AM

brennen added a project: User-brennen.Jan 19 2021, 4:22 PM

brennen moved this task from Backlog to Doing on the User-brennen board.Jan 19 2021, 4:40 PM

Mentioned in SAL (#wikimedia-operations) [2021-01-19T16:46:29Z] <brennen> 1.36.0-wmf.27 was branched at fbb516d8e33924c6cb66c93bb6d42907558c31f3 for T271341

Change 656987 merged by jenkins-bot:
[mediawiki/core@wmf/1.36.0-wmf.27] Branch commit for wmf/1.36.0-wmf.27

https://gerrit.wikimedia.org/r/656987

Mentioned in SAL (#wikimedia-operations) [2021-01-19T17:59:03Z] <brennen> starting deploy-promote to testwikis for 1.36.0-wmf.27 (T271341)

brennen added a subtask: T272428: Error 1146: Table 'mediawikiwiki.translate_cache' doesn't exist.Jan 19 2021, 10:00 PM

As we're coming up on 15:00 Pacific, blocking on T272428 for the (US) day. Can resume at earliest tomorrow during @liw's daytime if that situation is sorted.

• Nikerabbit closed subtask T272428: Error 1146: Table 'mediawikiwiki.translate_cache' doesn't exist as Resolved.Jan 20 2021, 1:43 PM

Mentioned in SAL (#wikimedia-operations) [2021-01-20T15:18:06Z] <brennen> 1.36.0-wmf.27 train unblocked, proceeding to group0 (T271341)

brennen added a subtask: T272505: FilterLookup: No version of filter [x] closest to [y] found.Jan 20 2021, 4:17 PM

brennen added a subtask: T272508: PropertyInfoSnakUrlExpander: Bad value for parameter $snak->getDataValue(): must be a DataValues\StringValue.Jan 20 2021, 4:53 PM

brennen closed subtask T272505: FilterLookup: No version of filter [x] closest to [y] found as Resolved.Jan 20 2021, 6:41 PM

Mentioned in SAL (#wikimedia-operations) [2021-01-20T20:06:29Z] <brennen> 1.36.0-wmf.27 (T271341) train status as of deploy window: currently blocked at group0 on T272508

Mentioned in SAL (#wikimedia-operations) [2021-01-20T20:23:11Z] <brennen> 1.36.0-wmf.27 (T271341) train: proceeding to group1

brennen removed a subtask: T272508: PropertyInfoSnakUrlExpander: Bad value for parameter $snak->getDataValue(): must be a DataValues\StringValue.Jan 20 2021, 8:23 PM

End-of-day status: Quiet on group1 for the last several hours.

brennen mentioned this in T272635: LinkBatch::addResultToCache: encountered invalid title.Jan 21 2021, 8:34 PM

brennen mentioned this in T272637: ConvertibleTimestamp::getTimestamp: The timestamp cannot be represented in the specified format.Jan 21 2021, 9:17 PM

Jdlrobson added a subtask: T272638: 0.5 million errors relating to error caused by rollback "TypeError: null is not an object (evaluating 't[e.title]') on mobile domain".Jan 21 2021, 9:35 PM

brennen removed a subtask: T272638: 0.5 million errors relating to error caused by rollback "TypeError: null is not an object (evaluating 't[e.title]') on mobile domain".Jan 22 2021, 12:53 AM

End-of-day summary:

I deployed to all wikis at the usual time, finishing at 20:05:54 UTC
Around 21:20 I noticed a spike of errors in logs. These would turn out to be, I believe, from T270334, but at the time I decided to roll back out of an abundance of caution. Rollback finished at 21:27.
Rollback triggered T272638.
Brief incident report at https://www.mediawiki.org/wiki/Reading/Web/Notable_incidents#January

Currently stable on all wikis.

brennen closed this task as Resolved.Jan 22 2021, 6:55 PM

brennen moved this task from Doing to Done or Declined on the User-brennen board.Mar 11 2021, 7:10 PM

1.36.0-wmf.27 deployment blockersClosed, ResolvedPublicReleaseActions