Fatal error: Stack overflow in [files] for wmf.14
Closed, ResolvedPublic

Description

Rolling group1 to wmf.14 resulted in stack overflows coming from several files.

Fatal error: Stack overflow in /srv/mediawiki/php-1.30.0-wmf.14/vendor/monolog/monolog/src/Monolog/Logger.php on line 292
Fatal error: Stack overflow in /srv/mediawiki/php-1.30.0-wmf.14/includes/libs/objectcache/WANObjectCache.php on line 251
Fatal error: Stack overflow in /srv/mediawiki/php-1.30.0-wmf.14/includes/libs/objectcache/WANObjectCache.php on line 306
Fatal error: Stack overflow in /srv/mediawiki/php-1.30.0-wmf.14/includes/libs/objectcache/MemcachedBagOStuff.php on line 182
Fatal error: Stack overflow in /srv/mediawiki/php-1.30.0-wmf.14/vendor/monolog/monolog/src/Monolog/Logger.php on line 532
Fatal error: Stack overflow in /srv/mediawiki/php-1.30.0-wmf.14/extensions/WikimediaMessages/WikimediaMessages.hooks.php on line 97
Fatal error: Stack overflow in /srv/mediawiki/php-1.30.0-wmf.14/includes/cache/MessageCache.php on line 991
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 17 2017, 5:14 PM
thcipriani triaged this task as Unbreak Now! priority.

UBN! as blocking train.

Restricted Application added subscribers: Liuxinyu970226, Jay8g, TerraCodes. · View Herald TranscriptAug 17 2017, 5:15 PM

Mentioned in SAL (#wikimedia-operations) [2017-08-17T17:20:06Z] <thcipriani@tin> rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis back to wmf.13 now T173520

Change 372427 had a related patch set uploaded (by Thcipriani; owner: Thcipriani):
[mediawiki/extensions/ProofreadPage@wmf/1.30.0-wmf.14] Revert to db7507246665e69384c1d92af2aedc62263a5116

https://gerrit.wikimedia.org/r/372427

Change 372427 merged by jenkins-bot:
[mediawiki/extensions/ProofreadPage@wmf/1.30.0-wmf.14] Revert to db7507246665e69384c1d92af2aedc62263a5116

https://gerrit.wikimedia.org/r/372427

Mentioned in SAL (#wikimedia-operations) [2017-08-17T19:01:27Z] <thcipriani@tin> Started scap: [[gerrit:372427|ProofReadPage Revert to db7507246665e69384c1d92af2aedc62263a5116]] T173520

Mentioned in SAL (#wikimedia-operations) [2017-08-17T19:07:40Z] <thcipriani@tin> Finished scap: [[gerrit:372427|ProofReadPage Revert to db7507246665e69384c1d92af2aedc62263a5116]] T173520 (duration: 06m 13s)

thcipriani lowered the priority of this task from Unbreak Now! to High.Aug 17 2017, 7:26 PM
thcipriani added a subscriber: Tpt.

@aaron found a commit in proofreadpage that looked like the root cause of this issue. Since reverting and resyncing, the stack overflows seem to have subsided. We will still need to address the root issue in master.

Adding @Tpt; @aaron, could you explain the cause of this?

From @Tpt via gerrit:

I am in vacation with at best a bad 3G connection so it would be much appreciated if some wmf people could work on it.

So we'll need to ensure this is fixed and/or reverted in master for wmf.15.

Tpt added a comment.Aug 18 2017, 1:27 PM

@thcipriani do you have the full stack trace (I do not have access to the production logs)? It would hopefully make solving this problem easier.

He made P5891 yesterday. Does it work for you? We don't have anything else beside that.

Change 371649 had a related patch set uploaded (by Tpt; owner: Tpt):
[mediawiki/extensions/ProofreadPage@master] Avoids a stack overflow

https://gerrit.wikimedia.org/r/371649

Tpt added a comment.Aug 18 2017, 8:23 PM

@Ladsgroup Thank you! I've made a change that should fix this problem. https://gerrit.wikimedia.org/r/371649

Change 372565 had a related patch set uploaded (by Tpt; owner: Tpt):
[mediawiki/extensions/ProofreadPage@master] Avoids a stack overflow

https://gerrit.wikimedia.org/r/372565

Change 371649 abandoned by Tpt:
Avoids a stack overflow

Reason:
Conflict in change-ids. See https://gerrit.wikimedia.org/r/#/c/372565 for a clean change

https://gerrit.wikimedia.org/r/371649

Change 373351 had a related patch set uploaded (by Thcipriani; owner: Thcipriani):
[mediawiki/extensions/ProofreadPage@wmf/1.30.0-wmf.15] Revert to db7507246665e69384c1d92af2aedc62263a5116

https://gerrit.wikimedia.org/r/373351

Change 373351 merged by jenkins-bot:
[mediawiki/extensions/ProofreadPage@wmf/1.30.0-wmf.15] Revert to db7507246665e69384c1d92af2aedc62263a5116

https://gerrit.wikimedia.org/r/373351

Re-cherry picked the change to the wmf.14 branch to wmf.15.

@Tpt I added @aaron as a reviewer to your updated patch since he spotted the issue in the first place. Ideally we'll need to get something merged into master soon (preferably before next branch cut) to avoid cherry-picking that same patch to wmf.16.

Added as blocking task to wmf.16 so that it doesn't slip off the radar.

Mentioned in SAL (#wikimedia-operations) [2017-08-23T20:08:21Z] <thcipriani@tin> Started scap: [[gerrit:373351|Revert ProofreadPage to db7507246665e69384c1d92af2aedc62263a5116 for wmf.15]] T173520

Mentioned in SAL (#wikimedia-operations) [2017-08-23T20:12:33Z] <thcipriani@tin> Finished scap: [[gerrit:373351|Revert ProofreadPage to db7507246665e69384c1d92af2aedc62263a5116 for wmf.15]] T173520 (duration: 04m 11s)

Change 374879 had a related patch set uploaded (by Thcipriani; owner: Thcipriani):
[mediawiki/extensions/ProofreadPage@wmf/1.30.0-wmf.16] Revert to db7507246665e69384c1d92af2aedc62263a5116

https://gerrit.wikimedia.org/r/374879

Change 374882 had a related patch set uploaded (by Thcipriani; owner: Tpt):
[mediawiki/extensions/ProofreadPage@wmf/1.30.0-wmf.16] Avoids a stack overflow

https://gerrit.wikimedia.org/r/374882

Change 374882 merged by jenkins-bot:
[mediawiki/extensions/ProofreadPage@wmf/1.30.0-wmf.16] Avoids a stack overflow

https://gerrit.wikimedia.org/r/374882

Change 372565 merged by jenkins-bot:
[mediawiki/extensions/ProofreadPage@master] Avoids a stack overflow

https://gerrit.wikimedia.org/r/372565

Mentioned in SAL (#wikimedia-operations) [2017-08-30T21:03:18Z] <thcipriani@tin> Synchronized php-1.30.0-wmf.16/extensions/ProofreadPage/ProofreadPage.body.php: [[gerrit:374882|ProofreadPage: Avoids a stack overflow]] T173520 (duration: 00m 47s)

Is this now Resolved? If so, can https://gerrit.wikimedia.org/r/#/c/374879/ be abandoned? If not, what needs to happen next?

Tpt added a comment.Sat, Sep 2, 6:09 PM

Yes, it seems resolved. The change that should fix this error have been merged and deployed.

Could someone with access to the production logs confirm this task is resolved?

aaron closed this task as Resolved.Sat, Sep 2, 7:10 PM
aaron claimed this task.