1.35.0-wmf.23 deployment blockers
Closed, ResolvedPublicRelease
Actions

Assigned To

Authored By

	• mmodell
	Sep 25 2019, 10:14 PM

Details

Backup Train Conductor: hashar
Release Version: 1.35.0-wmf.23
Release Date: Mar 9 2020, 12:00 AM

2020 week 11 1.35-wmf.23 Changes wmf/1.35.0-wmf.23

This MediaWiki Train Deployment is scheduled for the week of Monday, March 9th:

Monday March 9th	Tuesday, March 10th	Wednesday, March 11th	Thursday, March 12th	Friday
Backports only.	Branch `wmf.23` and deploy to Group 0 Wikis.	Deploy `wmf.23` to Group 1 Wikis.	Deploy `wmf.23` to all Wikis.	No deployments on fridays

See https://wikitech.wikimedia.org/wiki/Deployments for full schedule.

How this works

Any serious bugs affecting wmf.23 should be added as subtasks beneath this one.
- Use this form to create one.
Any open subtask(s) block the train from moving forward. This means no further deployments until the blockers are resolved.
If something is serious enough to warrant a rollback then you should bring it to the attention of deployers on the #wikimedia-operations IRC channel.
If you have a risky change in this week's train add a comment to this task using the Risky patch template
For more info about deployment blockers, see Holding the train.

Other Deployments

Previous: 1.35.0-wmf.22

Next: 1.35.0-wmf.24

Related Objects
Search...

Status	Subtype	Assigned	Task
Resolved	Release	hashar	T233872 1.35.0-wmf.24 deployment blockers
Resolved	Release	brennen	T233871 1.35.0-wmf.23 deployment blockers
Resolved	PRODUCTION ERROR	brennen	T247446 CPT want the wmf.23 train blocked for group1
Resolved	PRODUCTION ERROR	Krinkle	T247562 Warning: Memcached::setMulti(): failed to set key global:segment:...

Event Timeline

• mmodell created this task.Sep 25 2019, 10:14 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 25 2019, 10:14 PM

thcipriani assigned this task to brennen.Jan 16 2020, 10:13 PM

thcipriani triaged this task as Medium priority.

thcipriani updated Other Assignee, added: hashar.

thcipriani added a project: Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)).

thcipriani moved this task from INBOX to Deployment on the Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)) board.

• Mholloway subscribed.Mar 3 2020, 6:42 PM

brennen added a project: User-brennen.Mar 4 2020, 7:19 PM

brennen moved this task from Backlog to Next on the User-brennen board.

brennen moved this task from Next to Doing on the User-brennen board.Mar 10 2020, 3:13 PM

Mentioned in SAL (#wikimedia-operations) [2020-03-10T16:50:04Z] <brennen> starting branch cut for wmf/1.35.0-wmf.23 - T233871

Mentioned in SAL (#wikimedia-operations) [2020-03-10T17:18:33Z] <brennen> 1.35.0-wmf.23 was branched at 8e3738cc2f0665d19c1ff758a1f16eebae0039dd for T233871

Group0 is currently on wmf.23.

scap sync "testwiki to php-1.35.0-wmf.23 and rebuild l10n cache" took a long time to complete:

...
20:39:06 Finished scap-cdb-rebuild (duration: 91m 00s)
20:39:06 Started sync_wikiversions
20:39:06 Compiled /srv/mediawiki-staging/wikiversions.json to /srv/mediawiki-staging/wikiversions.php
sync_wikiversions: 100% (ok: 445; fail: 0; left: 0)  
20:39:13 Finished sync_wikiversions (duration: 00m 07s)
20:39:13 Running refreshMessageBlobs.php for each wiki
20:39:13 Check php-fpm cache...
20:39:22 Finished scap: testwiki to php-1.35.0-wmf.23 and rebuild l10n cache (duration: 163m 37s)

I manually generated the deploy notes with ./makedeploynotes.py 1.35.0-wmf.22 1.35.0-wmf.23 | tee > ~/1.35.0-wmf.23.txt

brennen mentioned this in T247426: Investigate duration of scap sync for 1.35.0-wmf.23 train deploy.Mar 11 2020, 5:08 PM

brennen closed subtask T247446: CPT want the wmf.23 train blocked for group1 as Resolved.Mar 11 2020, 8:21 PM

brennen mentioned this in T247458: PHP Notice: Undefined index: wgKartographerLiveData.Mar 11 2020, 8:43 PM

brennen mentioned this in T247466: SimpleCacheWithBagOStuff: Cache key contains characters that are not allowed.Mar 11 2020, 9:33 PM

Krinkle added a subtask: T247458: PHP Notice: Undefined index: wgKartographerLiveData.Mar 12 2020, 1:25 AM

Krinkle added a subtask: T247466: SimpleCacheWithBagOStuff: Cache key contains characters that are not allowed.Mar 12 2020, 1:27 AM

Krinkle added a subtask: T247484: Lots of "EventBus: Unable to deliver all events".Mar 12 2020, 1:53 AM

@Krinkle do T247484 or T247466 seem serious enough to prompt a rollback at this stage, or just block at group1?

For T247484, I defer to Analytics (EventBus) and CPT (MW-API) in terms of impact of those logs being incomplete. I suspect it might be okay for a day on group1.

For T247466, I suspect not as non-Wikipedia wikis interact very little with wikidata as I understand it but I could be wrong. However if we find more cases on Commons and Wikidata where pages can't be opened or edited, then I think we should rollback indeed.

Krinkle removed a subtask: T247484: Lots of "EventBus: Unable to deliver all events".Mar 12 2020, 4:05 PM

brennen mentioned this in T247546: DataUpdateAdapter::doUpdate: Cannot execute query.Mar 12 2020, 5:47 PM

brennen added a subtask: T247546: DataUpdateAdapter::doUpdate: Cannot execute query.Mar 12 2020, 6:02 PM

brennen removed a subtask: T247466: SimpleCacheWithBagOStuff: Cache key contains characters that are not allowed.Mar 12 2020, 6:59 PM

brennen removed a subtask: T247458: PHP Notice: Undefined index: wgKartographerLiveData.Mar 12 2020, 7:02 PM

brennen removed a subtask: T247546: DataUpdateAdapter::doUpdate: Cannot execute query.

• nnikkhoui added a subtask: T247553: Fail-safe exception from ReplicaMasterAwareRecordIdsAcquirer via Wikibase DataUpdateAdapter::doUpdate.Mar 12 2020, 7:16 PM

brennen removed a subtask: T247553: Fail-safe exception from ReplicaMasterAwareRecordIdsAcquirer via Wikibase DataUpdateAdapter::doUpdate.Mar 12 2020, 8:19 PM

brennen added a subtask: T247562: Warning: Memcached::setMulti(): failed to set key global:segment:....Mar 12 2020, 8:30 PM

thcipriani added a parent task: T233872: 1.35.0-wmf.24 deployment blockers.Mar 16 2020, 11:06 PM

Still blocked as of this writing, not likely to move forward during this US workday. I'll see where things are at at 06:00 local when the cat wakes me up for breakfast.

It is no more blocked!!!! I will push 1.35.0-wmf.23 to the rest of the wikis.

[14:06:27] <logmsgbot> !log hashar@deploy1001 rebuilt and synchronized wikiversions files: all wikis to 1.35.0-wmf.23

Logstash does not show any new concerning messages.

During/after the deployment memcache had:

5 minutes spike of get requests from 550k > 650k
30 minutes raise of set requests from 20k to 25k

I guess that is just caches being populated.

I am claiming 1.35.0-wmf.23 to be fully deployed.

brennen moved this task from Doing to Done or Declined on the User-brennen board.Mar 18 2020, 4:11 PM

phuedx mentioned this in T248010: Vector sidebar missing on test.wikipedia.org and weird footer.Mar 18 2020, 7:51 PM

1.35.0-wmf.23 deployment blockersClosed, ResolvedPublicReleaseActions