Page MenuHomePhabricator
Create Task
Maniphest T173520

Fatal error: Stack overflow in [files] for wmf.14
Closed, ResolvedPublicPRODUCTION ERROR
Actions

Description

Rolling group1 to wmf.14 resulted in stack overflows coming from several files.

Fatal error: Stack overflow in /srv/mediawiki/php-1.30.0-wmf.14/vendor/monolog/monolog/src/Monolog/Logger.php on line 292
Fatal error: Stack overflow in /srv/mediawiki/php-1.30.0-wmf.14/includes/libs/objectcache/WANObjectCache.php on line 251
Fatal error: Stack overflow in /srv/mediawiki/php-1.30.0-wmf.14/includes/libs/objectcache/WANObjectCache.php on line 306
Fatal error: Stack overflow in /srv/mediawiki/php-1.30.0-wmf.14/includes/libs/objectcache/MemcachedBagOStuff.php on line 182
Fatal error: Stack overflow in /srv/mediawiki/php-1.30.0-wmf.14/vendor/monolog/monolog/src/Monolog/Logger.php on line 532
Fatal error: Stack overflow in /srv/mediawiki/php-1.30.0-wmf.14/extensions/WikimediaMessages/WikimediaMessages.hooks.php on line 97
Fatal error: Stack overflow in /srv/mediawiki/php-1.30.0-wmf.14/includes/cache/MessageCache.php on line 991

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
Revert to db7507246665e69384c1d92af2aedc62263a5116Thciprianimediawiki/extensions/ProofreadPagewmf/1.30.0-wmf.16+738 -934
Avoids a stack overflowTptmediawiki/extensions/ProofreadPagemaster+4 -2
Avoids a stack overflowThciprianimediawiki/extensions/ProofreadPagewmf/1.30.0-wmf.16+4 -2
Revert to db7507246665e69384c1d92af2aedc62263a5116Thciprianimediawiki/extensions/ProofreadPagewmf/1.30.0-wmf.15+738 -934
Revert to db7507246665e69384c1d92af2aedc62263a5116Thciprianimediawiki/extensions/ProofreadPagewmf/1.30.0-wmf.14+738 -934
Avoids a stack overflowTptmediawiki/extensions/ProofreadPagemaster+4 -2
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedReleasethciprianiT170632 1.30.0-wmf.14 deployment blockers
 ResolvedRelease• demonT170634 1.30.0-wmf.16 deployment blockers
 ResolvedPRODUCTION ERRORaaronT173520 Fatal error: Stack overflow in [files] for wmf.14

Event Timeline

thcipriani created this task.Aug 17 2017, 5:14 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 17 2017, 5:14 PM
thcipriani triaged this task as Unbreak Now! priority.Aug 17 2017, 5:15 PM
thcipriani added a parent task: T170632: 1.30.0-wmf.14 deployment blockers.

UBN! as blocking train.

Restricted Application added subscribers: Liuxinyu970226, Jay8g, TerraCodes. · View Herald TranscriptAug 17 2017, 5:15 PM
Stashbot subscribed.Aug 17 2017, 5:20 PM

Mentioned in SAL (#wikimedia-operations) [2017-08-17T17:20:06Z] <thcipriani@tin> rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis back to wmf.13 now T173520

greg mentioned this in T173462: LinksUpdate::acquirePageLock: Cannot flush pre-lock snapshot because writes are pending.Aug 17 2017, 5:32 PM
gerritbot subscribed.Aug 17 2017, 6:21 PM

Change 372427 had a related patch set uploaded (by Thcipriani; owner: Thcipriani):
[mediawiki/extensions/ProofreadPage@wmf/1.30.0-wmf.14] Revert to db7507246665e69384c1d92af2aedc62263a5116

https://gerrit.wikimedia.org/r/372427

gerritbot added a project: Patch-For-Review.Aug 17 2017, 6:21 PM
gerritbot added a comment.Aug 17 2017, 6:56 PM

Change 372427 merged by jenkins-bot:
[mediawiki/extensions/ProofreadPage@wmf/1.30.0-wmf.14] Revert to db7507246665e69384c1d92af2aedc62263a5116

https://gerrit.wikimedia.org/r/372427

Stashbot added a comment.Aug 17 2017, 7:01 PM

Mentioned in SAL (#wikimedia-operations) [2017-08-17T19:01:27Z] <thcipriani@tin> Started scap: [[gerrit:372427|ProofReadPage Revert to db7507246665e69384c1d92af2aedc62263a5116]] T173520

Stashbot added a comment.Aug 17 2017, 7:07 PM

Mentioned in SAL (#wikimedia-operations) [2017-08-17T19:07:40Z] <thcipriani@tin> Finished scap: [[gerrit:372427|ProofReadPage Revert to db7507246665e69384c1d92af2aedc62263a5116]] T173520 (duration: 06m 13s)

thcipriani lowered the priority of this task from Unbreak Now! to High.Aug 17 2017, 7:26 PM
thcipriani added a subscriber: Tpt.

@aaron found a commit in proofreadpage that looked like the root cause of this issue. Since reverting and resyncing, the stack overflows seem to have subsided. We will still need to address the root issue in master.

Adding @Tpt; @aaron, could you explain the cause of this?

thcipriani added a comment.Aug 17 2017, 7:30 PM

From @Tpt via gerrit:

I am in vacation with at best a bad 3G connection so it would be much appreciated if some wmf people could work on it.

So we'll need to ensure this is fixed and/or reverted in master for wmf.15.

Krinkle edited projects, added: ProofreadPage; removed: Patch-For-Review.Aug 17 2017, 8:24 PM
Tpt added a comment.Aug 18 2017, 1:27 PM

@thcipriani do you have the full stack trace (I do not have access to the production logs)? It would hopefully make solving this problem easier.

Ladsgroup added a comment.Aug 18 2017, 1:30 PM

He made P5891 yesterday. Does it work for you? We don't have anything else beside that.

gerritbot added a comment.Aug 18 2017, 8:19 PM

Change 371649 had a related patch set uploaded (by Tpt; owner: Tpt):
[mediawiki/extensions/ProofreadPage@master] Avoids a stack overflow

https://gerrit.wikimedia.org/r/371649

gerritbot added a project: Patch-For-Review.Aug 18 2017, 8:19 PM
Tpt added a comment.Aug 18 2017, 8:23 PM

@Ladsgroup Thank you! I've made a change that should fix this problem. https://gerrit.wikimedia.org/r/371649

gerritbot added a comment.Aug 18 2017, 8:27 PM

Change 372565 had a related patch set uploaded (by Tpt; owner: Tpt):
[mediawiki/extensions/ProofreadPage@master] Avoids a stack overflow

https://gerrit.wikimedia.org/r/372565

gerritbot added a comment.Aug 18 2017, 8:28 PM

Change 371649 abandoned by Tpt:
Avoids a stack overflow

Reason:
Conflict in change-ids. See https://gerrit.wikimedia.org/r/#/c/372565 for a clean change

https://gerrit.wikimedia.org/r/371649

gerritbot added a comment.Aug 23 2017, 7:22 PM

Change 373351 had a related patch set uploaded (by Thcipriani; owner: Thcipriani):
[mediawiki/extensions/ProofreadPage@wmf/1.30.0-wmf.15] Revert to db7507246665e69384c1d92af2aedc62263a5116

https://gerrit.wikimedia.org/r/373351

gerritbot added a comment.Aug 23 2017, 7:27 PM

Change 373351 merged by jenkins-bot:
[mediawiki/extensions/ProofreadPage@wmf/1.30.0-wmf.15] Revert to db7507246665e69384c1d92af2aedc62263a5116

https://gerrit.wikimedia.org/r/373351

thcipriani added a comment.Aug 23 2017, 7:52 PM

Re-cherry picked the change to the wmf.14 branch to wmf.15.

@Tpt I added @aaron as a reviewer to your updated patch since he spotted the issue in the first place. Ideally we'll need to get something merged into master soon (preferably before next branch cut) to avoid cherry-picking that same patch to wmf.16.

thcipriani added a parent task: T170634: 1.30.0-wmf.16 deployment blockers.Aug 23 2017, 7:54 PM

Added as blocking task to wmf.16 so that it doesn't slip off the radar.

Stashbot added a comment.Aug 23 2017, 8:08 PM

Mentioned in SAL (#wikimedia-operations) [2017-08-23T20:08:21Z] <thcipriani@tin> Started scap: [[gerrit:373351|Revert ProofreadPage to db7507246665e69384c1d92af2aedc62263a5116 for wmf.15]] T173520

Stashbot added a comment.Aug 23 2017, 8:12 PM

Mentioned in SAL (#wikimedia-operations) [2017-08-23T20:12:33Z] <thcipriani@tin> Finished scap: [[gerrit:373351|Revert ProofreadPage to db7507246665e69384c1d92af2aedc62263a5116 for wmf.15]] T173520 (duration: 04m 11s)

gerritbot added a comment.Aug 30 2017, 8:07 PM

Change 374879 had a related patch set uploaded (by Thcipriani; owner: Thcipriani):
[mediawiki/extensions/ProofreadPage@wmf/1.30.0-wmf.16] Revert to db7507246665e69384c1d92af2aedc62263a5116

https://gerrit.wikimedia.org/r/374879

gerritbot added a comment.Aug 30 2017, 8:10 PM

Change 374882 had a related patch set uploaded (by Thcipriani; owner: Tpt):
[mediawiki/extensions/ProofreadPage@wmf/1.30.0-wmf.16] Avoids a stack overflow

https://gerrit.wikimedia.org/r/374882

gerritbot added a comment.Aug 30 2017, 8:27 PM

Change 374882 merged by jenkins-bot:
[mediawiki/extensions/ProofreadPage@wmf/1.30.0-wmf.16] Avoids a stack overflow

https://gerrit.wikimedia.org/r/374882

gerritbot added a comment.Aug 30 2017, 8:59 PM

Change 372565 merged by jenkins-bot:
[mediawiki/extensions/ProofreadPage@master] Avoids a stack overflow

https://gerrit.wikimedia.org/r/372565

ReleaseTaggerBot added a project: MW-1.30-release-notes (WMF-deploy-2017-09-05 (1.30.0-wmf.17)).Aug 30 2017, 9:00 PM
Stashbot added a comment.Aug 30 2017, 9:03 PM

Mentioned in SAL (#wikimedia-operations) [2017-08-30T21:03:18Z] <thcipriani@tin> Synchronized php-1.30.0-wmf.16/extensions/ProofreadPage/ProofreadPage.body.php: [[gerrit:374882|ProofreadPage: Avoids a stack overflow]] T173520 (duration: 00m 47s)

Jdforrester-WMF subscribed.Sep 1 2017, 10:29 PM

Is this now Resolved? If so, can https://gerrit.wikimedia.org/r/#/c/374879/ be abandoned? If not, what needs to happen next?

Tpt added a comment.Sep 2 2017, 6:09 PM

Yes, it seems resolved. The change that should fix this error have been merged and deployed.

Could someone with access to the production logs confirm this task is resolved?

aaron closed this task as Resolved.Sep 2 2017, 7:10 PM
aaron claimed this task.
gerritbot added a comment.Sep 2 2017, 9:08 PM

Change 374879 abandoned by Jforrester:
Revert to db7507246665e69384c1d92af2aedc62263a5116

https://gerrit.wikimedia.org/r/374879

Liuxinyu970226 unsubscribed.Sep 15 2017, 3:17 AM
• mmodell changed the subtype of this task from "Task" to "Production Error".Aug 28 2019, 11:10 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits