Page MenuHomePhabricator
Create Task
Maniphest T305218

1.39.0-wmf.12 deployment blockers
Closed, ResolvedPublic5 Estimated Story PointsRelease
Actions

Details

Backup Train Conductor
hashar
Release Version
1.39.0-wmf.12
Release Date
May 16 2022, 12:00 AM

2022 week 20 1.39-wmf.12 Changes wmf/1.39.0-wmf.12

This MediaWiki Train Deployment is scheduled for the week of Monday, May 16th:

Monday May 16thTuesday, May 17thWednesday, May 18thThursday, May 19thFriday
Backports only.Branch wmf.12 and deploy to Group 0 Wikis.Deploy wmf.12 to Group 1 Wikis.Deploy wmf.12 to all Wikis.No deployments on fridays

How this works

  • Any serious bugs affecting wmf.12 should be added as subtasks beneath this one.
  • Any open subtask(s) block the train from moving forward. This means no further deployments until the blockers are resolved.
  • If something is serious enough to warrant a rollback then you should bring it to the attention of deployers on the #wikimedia-operations IRC channel.
  • If you have a risky change in this week's train add a comment to this task using the Risky patch template
  • For more info about deployment blockers, see Holding the train.

Related Links

Other Deployments

Previous: 1.39.0-wmf.11
Next: 1.39.0-wmf.13
SubjectRepoBranchLines +/-
parser: Avoid pushing the whole content to ParserObserver debug logmediawiki/coremaster+6 -6
parser: Avoid pushing the whole content to ParserObserver debug logmediawiki/corewmf/1.39.0-wmf.12+6 -6
Customize query in gerrit

Related Objects
Search...

Event Timeline

thcipriani created this task.Apr 1 2022, 12:52 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 1 2022, 12:52 AM
taavi added a subtask: T307518: Internal error on changing user's global rights (CentralAuthGroupMembershipProxy::getWikiId() undefined).May 4 2022, 5:04 AM
taavi removed a subtask: T307518: Internal error on changing user's global rights (CentralAuthGroupMembershipProxy::getWikiId() undefined).
Jdforrester-WMF subscribed.May 9 2022, 1:38 PM
Risky Patch! 🚂🔥
thcipriani assigned this task to jnuche.May 10 2022, 9:39 PM
thcipriani triaged this task as Medium priority.
thcipriani updated Other Assignee, added: hashar.
thcipriani set the point value for this task to 5.
thcipriani added a project: Release-Engineering-Team (Priority Backlog 📥).
Ladsgroup subscribed.May 12 2022, 6:48 AM
Risky Patch! 🚂🔥
  • Change: https://gerrit.wikimedia.org/r/c/mediawiki/core/+/767769
  • Summary: Changing connection management to the databases
  • Test plan: Errors related to database should not go up.
  • Places to monitor:
  • Revert plan: Probably rolling back train would be ok. Quick fix in cases would be to replace getConnection with getConnectionInternal and the patch better not to be reverted.
  • Affected wikis: all
  • IRC contact: Amir1, Krinkle
  • UBN Task Projects/tags: DBA Performance-Team
  • Would you like to backport this change rather than ride the train?: No because I want it to be tested in group0 first.
hashar subscribed.May 16 2022, 4:10 PM
gerritbot added a comment.May 17 2022, 7:20 AM

Change 792557 had a related patch set uploaded (by Jaime Nuche; author: Jaime Nuche):

[operations/mediawiki-config@master] testwikis wikis to 1.39.0-wmf.12 refs T305218

https://gerrit.wikimedia.org/r/792557

gerritbot added a project: Patch-For-Review.May 17 2022, 7:20 AM

Change 792557 merged by jenkins-bot:

[operations/mediawiki-config@master] testwikis wikis to 1.39.0-wmf.12 refs T305218

https://gerrit.wikimedia.org/r/792557

Stashbot added a comment.May 17 2022, 7:22 AM

Mentioned in SAL (#wikimedia-operations) [2022-05-17T07:22:07Z] <jnuche@deploy1002> Started scap: testwikis wikis to 1.39.0-wmf.12 refs T305218

Stashbot added a comment.May 17 2022, 7:39 AM

Mentioned in SAL (#wikimedia-operations) [2022-05-17T07:39:23Z] <jnuche@deploy1002> Started scap: testwikis wikis to 1.39.0-wmf.12 refs T305218

Stashbot added a comment.May 17 2022, 7:53 AM

Mentioned in SAL (#wikimedia-operations) [2022-05-17T07:53:58Z] <jnuche@deploy1002> Finished scap: testwikis wikis to 1.39.0-wmf.12 refs T305218 (duration: 14m 35s)

gerritbot added a comment.May 17 2022, 8:03 AM

Change 792563 had a related patch set uploaded (by Jaime Nuche; author: Jaime Nuche):

[operations/mediawiki-config@master] group0 wikis to 1.39.0-wmf.12 refs T305218

https://gerrit.wikimedia.org/r/792563

gerritbot added a comment.May 17 2022, 8:04 AM

Change 792563 merged by jenkins-bot:

[operations/mediawiki-config@master] group0 wikis to 1.39.0-wmf.12 refs T305218

https://gerrit.wikimedia.org/r/792563

Stashbot added a comment.May 17 2022, 8:06 AM

Mentioned in SAL (#wikimedia-operations) [2022-05-17T08:06:02Z] <jnuche@deploy1002> rebuilt and synchronized wikiversions files: group0 wikis to 1.39.0-wmf.12 refs T305218

gerritbot added a comment.May 18 2022, 8:09 AM

Change 792976 had a related patch set uploaded (by Jaime Nuche; author: Jaime Nuche):

[operations/mediawiki-config@master] group1 wikis to 1.39.0-wmf.12 refs T305218

https://gerrit.wikimedia.org/r/792976

gerritbot added a comment.May 18 2022, 8:20 AM

Change 792976 merged by jenkins-bot:

[operations/mediawiki-config@master] group1 wikis to 1.39.0-wmf.12 refs T305218

https://gerrit.wikimedia.org/r/792976

gerritbot added a comment.May 18 2022, 8:23 AM

Change 792978 had a related patch set uploaded (by Jaime Nuche; author: Jaime Nuche):

[operations/mediawiki-config@master] group1 wikis to 1.39.0-wmf.12 refs T305218

https://gerrit.wikimedia.org/r/792978

gerritbot added a comment.May 18 2022, 8:24 AM

Change 792978 merged by jenkins-bot:

[operations/mediawiki-config@master] group1 wikis to 1.39.0-wmf.12 refs T305218

https://gerrit.wikimedia.org/r/792978

Stashbot added a comment.May 18 2022, 8:26 AM

Mentioned in SAL (#wikimedia-operations) [2022-05-18T08:26:06Z] <jnuche@deploy1002> rebuilt and synchronized wikiversions files: group1 wikis to 1.39.0-wmf.12 refs T305218

Stashbot added a comment.May 18 2022, 8:27 AM

Mentioned in SAL (#wikimedia-operations) [2022-05-18T08:27:01Z] <jnuche@deploy1002> Synchronized php: group1 wikis to 1.39.0-wmf.12 refs T305218 (duration: 00m 53s)

jnuche added a subtask: T308640: Error: Call to undefined method CirrusSearch\Connection::getPageType().May 18 2022, 8:46 AM
Ladsgroup closed subtask T308640: Error: Call to undefined method CirrusSearch\Connection::getPageType() as Resolved.May 18 2022, 9:09 AM
jnuche added a subtask: T308663: LogicException: This ParserOutput contains no text!.May 18 2022, 1:47 PM
Ladsgroup closed subtask T308663: LogicException: This ParserOutput contains no text! as Resolved.May 18 2022, 3:37 PM
TheresNoTime added a subtask: T308691: Fatal exception of type "CannotCreateActorException" when trying to export file from zhwikibooks to commons.May 18 2022, 5:55 PM
Krinkle subscribed.EditedMay 18 2022, 8:50 PM

There is a huge spike of "MediaWiki\\Parser\\ParserObserver::notifyParse: Possibly redundant parse!","type":"mediawiki","channel":"DuplicateParse" errors in production. These are not visible on the new-errors dashboard due to the large message payload being truncated, and thus not indexed by OpenSearch, but they are instead visible on the mediawiki-errors dashboard.

The start of this correlates exactly with the 09:00 UTC rollout of group1 wikis to 1.39.0-wmf.12.

I believe this is train-blocking for reasons of exessive logging and making telemetry unstable/unusable. Secondarily, the warning itselfg may also be significant and train-blocking, depending on what investigation finds. Tertiarily, the warning should be fixed post-hoc to not exceed the allowed payload size, but that is not urgent.

Screenshot (739×1 px, 111 KB)
Screenshot (663×1 px, 97 KB)
Ladsgroup added a subscriber: matmarex.May 18 2022, 9:24 PM

This one was probably caused by yours truly: https://gerrit.wikimedia.org/r/c/mediawiki/core/+/791702

I think I know what part caused it, we probably should replace it with hash

gerritbot added a comment.May 18 2022, 9:30 PM

Change 793120 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[mediawiki/core@master] ParserObserver: Avoid pushing the whole content to the debug log

https://gerrit.wikimedia.org/r/793120

gerritbot added a comment.May 18 2022, 10:09 PM

Change 792665 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[mediawiki/core@wmf/1.39.0-wmf.12] parser: Avoid pushing the whole content to ParserObserver debug log

https://gerrit.wikimedia.org/r/792665

matmarex added a subtask: T308626: TypeError: Cannot read properties of undefined (reading 'nodeType').May 18 2022, 10:19 PM
gerritbot added a comment.May 18 2022, 10:26 PM

Change 792665 merged by jenkins-bot:

[mediawiki/core@wmf/1.39.0-wmf.12] parser: Avoid pushing the whole content to ParserObserver debug log

https://gerrit.wikimedia.org/r/792665

Stashbot added a comment.May 18 2022, 10:30 PM

Mentioned in SAL (#wikimedia-operations) [2022-05-18T22:30:03Z] <ladsgroup@deploy1002> Synchronized php-1.39.0-wmf.12/includes/parser/ParserObserver.php: Backport: [[gerrit:792665|parser: Avoid pushing the whole content to ParserObserver debug log (T305218)]] (duration: 00m 52s)

gerritbot added a comment.May 18 2022, 10:30 PM

Change 793120 merged by jenkins-bot:

[mediawiki/core@master] parser: Avoid pushing the whole content to ParserObserver debug log

https://gerrit.wikimedia.org/r/793120

Ladsgroup added a comment.May 18 2022, 10:32 PM

The json truncated thing is fixed now:

image.png (380×979 px, 42 KB)

matmarex closed subtask T308626: TypeError: Cannot read properties of undefined (reading 'nodeType') as Resolved.May 18 2022, 10:55 PM
ReleaseTaggerBot added a project: MW-1.39-notes (1.39.0-wmf.12; 2022-05-16).May 18 2022, 11:00 PM
jnuche closed subtask T308691: Fatal exception of type "CannotCreateActorException" when trying to export file from zhwikibooks to commons as Resolved.May 19 2022, 1:25 PM
gerritbot added a comment.May 19 2022, 1:50 PM

Change 793476 had a related patch set uploaded (by Jaime Nuche; author: Jaime Nuche):

[operations/mediawiki-config@master] all wikis to 1.39.0-wmf.12 refs T305218

https://gerrit.wikimedia.org/r/793476

gerritbot added a comment.May 19 2022, 1:51 PM

Change 793476 merged by jenkins-bot:

[operations/mediawiki-config@master] all wikis to 1.39.0-wmf.12 refs T305218

https://gerrit.wikimedia.org/r/793476

Stashbot added a comment.May 19 2022, 1:52 PM

Mentioned in SAL (#wikimedia-operations) [2022-05-19T13:52:49Z] <jnuche@deploy1002> rebuilt and synchronized wikiversions files: all wikis to 1.39.0-wmf.12 refs T305218

thiemowmde reopened subtask T308691: Fatal exception of type "CannotCreateActorException" when trying to export file from zhwikibooks to commons as Open.May 23 2022, 6:00 AM
Jdforrester-WMF closed subtask T308691: Fatal exception of type "CannotCreateActorException" when trying to export file from zhwikibooks to commons as Resolved.May 23 2022, 11:14 PM
dancy closed this task as Resolved.Aug 10 2022, 3:54 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL