PHP Notice: Undefined offset: 2 in WikimediaEvents\PageSplitter\PageSplitterInstrumentation->getBucket
Closed, ResolvedPublic3 Estimated Story PointsPRODUCTION ERROR
Actions

Assigned To

Authored By

	dancy
	Apr 27 2022, 5:09 PM

Description

Error

mwversion: 1.39.0-wmf.8
reqId: 290860cd-eb20-4f4b-9617-05022c59536c
Find reqId in Logstash

normalized_message

[{reqId}] {exception_url}   PHP Notice: Undefined offset: 2

exception.trace

from /srv/mediawiki/php-1.39.0-wmf.8/extensions/WikimediaEvents/includes/PageSplitter/PageSplitterInstrumentation.php(82)
#0 /srv/mediawiki/php-1.39.0-wmf.8/extensions/WikimediaEvents/includes/PageSplitter/PageSplitterInstrumentation.php(82): MWExceptionHandler::handleError(integer, string, string, integer, array)
#1 /srv/mediawiki/php-1.39.0-wmf.8/extensions/WikimediaEvents/includes/WebABTest/WebABTestArticleIdStrategy.php(91): WikimediaEvents\PageSplitter\PageSplitterInstrumentation->getBucket(double)
#2 /srv/mediawiki/php-1.39.0-wmf.8/skins/Vector/includes/Hooks.php(530): WikimediaEvents\WebABTest\WebABTestArticleIdStrategy->getBucket()
#3 /srv/mediawiki/php-1.39.0-wmf.8/skins/Vector/includes/Hooks.php(569): Vector\Hooks::getTocClasses(Vector\SkinVector22, GlobalVarConfig)
#4 /srv/mediawiki/php-1.39.0-wmf.8/includes/HookContainer/HookContainer.php(338): Vector\Hooks::onOutputPageBodyAttributes(OutputPage, Vector\SkinVector22, array)
#5 /srv/mediawiki/php-1.39.0-wmf.8/includes/HookContainer/HookContainer.php(137): MediaWiki\HookContainer\HookContainer->callLegacyHook(string, array, array, array)
#6 /srv/mediawiki/php-1.39.0-wmf.8/includes/HookContainer/HookRunner.php(2660): MediaWiki\HookContainer\HookContainer->run(string, array, array)
#7 /srv/mediawiki/php-1.39.0-wmf.8/includes/OutputPage.php(3345): MediaWiki\HookContainer\HookRunner->onOutputPageBodyAttributes(OutputPage, Vector\SkinVector22, array)
#8 /srv/mediawiki/php-1.39.0-wmf.8/includes/skins/SkinMustache.php(63): OutputPage->headElement(Vector\SkinVector22)
#9 /srv/mediawiki/php-1.39.0-wmf.8/includes/skins/SkinTemplate.php(161): SkinMustache->generateHTML()
#10 /srv/mediawiki/php-1.39.0-wmf.8/includes/OutputPage.php(2809): SkinTemplate->outputPage()
#11 /srv/mediawiki/php-1.39.0-wmf.8/includes/MediaWiki.php(926): OutputPage->output(boolean)
#12 /srv/mediawiki/php-1.39.0-wmf.8/includes/MediaWiki.php(939): MediaWiki::{closure}()
#13 /srv/mediawiki/php-1.39.0-wmf.8/includes/MediaWiki.php(566): MediaWiki->main()
#14 /srv/mediawiki/php-1.39.0-wmf.8/index.php(50): MediaWiki->run()
#15 /srv/mediawiki/php-1.39.0-wmf.8/index.php(46): wfIndexMain()
#16 /srv/mediawiki/w/index.php(3): require(string)
#17 {main}

Impact

Distracting log noise

Notes

These seem to have started Apr 26 2022 20:57:02. There have been 386 hits since then.

Details

Request URL: https://eu.wikipedia.org/wiki/Pineapple_Express

Related Changes in Gerrit:

Subject	Repo	Branch	Lines +/-
Fix undefined offset error	mediawiki/extensions/WikimediaEvents	master	+5 -0
PageSplitter: Fix underlying bug in PageRandomGenerate	mediawiki/extensions/WikimediaEvents	master	+8 -12
Fix undefined offset error	mediawiki/extensions/WikimediaEvents	wmf/1.39.0-wmf.10	+5 -0
Test for undefined offset error	mediawiki/extensions/WikimediaEvents	wmf/1.39.0-wmf.10	+59 -0
Test for undefined offset error	mediawiki/extensions/WikimediaEvents	master	+59 -0
Fix undefined offset error	mediawiki/extensions/WikimediaEvents	wmf/1.39.0-wmf.9	+5 -0

Customize query in gerrit

Related Objects

Mentioned In: T306559: Schema QA: mediawiki_web_ab_test_enrollment
T305216: 1.39.0-wmf.10 deployment blockers
Mentioned Here: T306559: Schema QA: mediawiki_web_ab_test_enrollment
T305216: 1.39.0-wmf.10 deployment blockers
T301584: Add rich snippet instrument to WikimediaEvents
rEWMV0d7d748d3a3e: Add rich snippet instrument

Event Timeline

dancy created this task.Apr 27 2022, 5:09 PM

dancy triaged this task as Unbreak Now! priority.

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 27 2022, 5:09 PM

The class PageSplitterInstrumentation was added for T301584 some weeks ago - https://gerrit.wikimedia.org/r/c/mediawiki/extensions/WikimediaEvents/+/763469 / 0d7d748d3a3ea7e7e9c34f52c6e98eddbcee164d

@cjming Hi Clare. What say you on this error?

dancy renamed this task from PHP Notice: Undefined offset: 2 to PHP Notice: Undefined offset: 2 in WikimediaEvents\PageSplitter\PageSplitterInstrumentation->getBucket.Apr 28 2022, 8:48 PM

dancy added a parent task: T305216: 1.39.0-wmf.10 deployment blockers.

dancy updated the task description. (Show Details)

In T307019#7888243, @dancy wrote:

@cjming Hi Clare. What say you on this error?

hi @dancy - thanks for reporting -- just getting back from a break - digging into it

Change 788377 had a related patch set uploaded (by Clare Ming; author: Clare Ming):

[mediawiki/extensions/WikimediaEvents@master] Fix undefined offset error

https://gerrit.wikimedia.org/r/788377

gerritbot added a project: Patch-For-Review.May 2 2022, 4:52 PM

Jdlrobson added a project: Web-Team-Backlog-Archived (Kanbanana-FY-2021-22).May 2 2022, 9:10 PM

Jdlrobson moved this task from Needs Analysis to Upcoming on the Web-Team-Backlog-Archived (Kanbanana-FY-2021-22) board.May 3 2022, 2:51 PM

LGoto moved this task from Upcoming to Ready for Development on the Web-Team-Backlog-Archived (Kanbanana-FY-2021-22) board.May 3 2022, 5:13 PM

LGoto set the point value for this task to 3.

cjming claimed this task.May 3 2022, 5:27 PM

cjming moved this task from Ready for Development to Doing on the Web-Team-Backlog-Archived (Kanbanana-FY-2021-22) board.

Mentioned in SAL (#wikimedia-operations) [2022-05-03T18:04:23Z] <brennen> train 1.39.0-wmf.10 (T305216): train is still blocked on T307019, although in practice that blocker doesn't prevent us from going ahead safely. i'm going unavoidably afk for a couple of hours; plan to move train to group1 on my return.

cjming removed cjming as the assignee of this task.May 3 2022, 7:41 PM

cjming moved this task from Doing to Code Review on the Web-Team-Backlog-Archived (Kanbanana-FY-2021-22) board.

Discussed with cjming; going to remove this one as a train blocker since train won't materially affect level of errors and it can be backported once reviewed.

brennen removed a parent task: T305216: 1.39.0-wmf.10 deployment blockers.May 3 2022, 9:02 PM

Change 788804 had a related patch set uploaded (by Jdlrobson; author: Jdlrobson):

[mediawiki/extensions/WikimediaEvents@master] Test for undefined offset error

https://gerrit.wikimedia.org/r/788804

Change 788377 merged by jenkins-bot:

[mediawiki/extensions/WikimediaEvents@master] Fix undefined offset error

https://gerrit.wikimedia.org/r/788377

ReleaseTaggerBot added a project: MW-1.39-notes (1.39.0-wmf.12; 2022-05-16).May 3 2022, 11:00 PM

Change 788849 had a related patch set uploaded (by Clare Ming; author: Clare Ming):

[mediawiki/extensions/WikimediaEvents@wmf/1.39.0-wmf.10] Fix undefined offset error

https://gerrit.wikimedia.org/r/788849

Change 788850 had a related patch set uploaded (by Clare Ming; author: Clare Ming):

[mediawiki/extensions/WikimediaEvents@wmf/1.39.0-wmf.9] Fix undefined offset error

https://gerrit.wikimedia.org/r/788850

Change 788850 abandoned by Clare Ming:

[mediawiki/extensions/WikimediaEvents@wmf/1.39.0-wmf.9] Fix undefined offset error

Reason:

backporting to wmf10 only

https://gerrit.wikimedia.org/r/788850

Change 788804 merged by jenkins-bot:

[mediawiki/extensions/WikimediaEvents@master] Test for undefined offset error

https://gerrit.wikimedia.org/r/788804

Change 788851 had a related patch set uploaded (by Clare Ming; author: Jdlrobson):

[mediawiki/extensions/WikimediaEvents@wmf/1.39.0-wmf.10] Test for undefined offset error

https://gerrit.wikimedia.org/r/788851

Change 788851 abandoned by Jdlrobson:

[mediawiki/extensions/WikimediaEvents@wmf/1.39.0-wmf.10] Test for undefined offset error

Reason:

Just a test. No need to backport.

https://gerrit.wikimedia.org/r/788851

Jdlrobson moved this task from Code Review to Ready for Signoff on the Web-Team-Backlog-Archived (Kanbanana-FY-2021-22) board.May 4 2022, 3:08 AM

@cjming - do we know whether this will affect the way the bucketing works for the A/B test? Would it be possible to write out what it was affecting and how?

In T307019#7902832, @ovasileva wrote:

@cjming - do we know whether this will affect the way the bucketing works for the A/B test? Would it be possible to write out what it was affecting and how?

hi @ovasileva - I will look into it and report back here

Change 788849 merged by jenkins-bot:

[mediawiki/extensions/WikimediaEvents@wmf/1.39.0-wmf.10] Fix undefined offset error

https://gerrit.wikimedia.org/r/788849

Mentioned in SAL (#wikimedia-operations) [2022-05-04T20:07:54Z] <cjming@deploy1002> Synchronized php-1.39.0-wmf.10/extensions/WikimediaEvents/includes/PageSplitter/PageSplitterInstrumentation.php: Backport: [[gerrit:788849|Fix undefined offset error (T307019)]] (duration: 00m 50s)

Maintenance_bot removed a project: Patch-For-Review.May 4 2022, 8:30 PM

ReleaseTaggerBot edited projects, added MW-1.39-notes (1.39.0-wmf.10; 2022-05-02); removed MW-1.39-notes (1.39.0-wmf.12; 2022-05-16).May 4 2022, 9:01 PM

LGoto assigned this task to • nray.May 5 2022, 5:05 PM

fwiw - it seems since we backported fix yesterday, errors have dropped off. None in last 8 hours - https://logstash.wikimedia.org/goto/c43e963c735bbcb77874e901e304eaf1

There is still some discussion on the patch in Gerrit about correct approach to fix issue. I still want to dig deeper into it since it's still a bit fuzzy to me.

In T307019#7902832, @ovasileva wrote:

@cjming - do we know whether this will affect the way the bucketing works for the A/B test? Would it be possible to write out what it was affecting and how?

hi @ovasileva - I wrote up a longish comment over at T306559#7908751 about my findings re: TOC A/B test bucketing. The tl;dr with respect to this bug is that the fix here resolves the issue where both new sticky toc AND legacy toc were being displayed at the same time on certain pages. For example, prior to backport, I was seeing both TOCs on https://eu.wikipedia.org/wiki/Pineapple_Express. Now I see only the new TOC on that page which means for that page, my web_session_id is in the treatment bucket.

However I'm still not sure exactly how these double-TOC pages were getting bucketed - I suspect they were in neither bucket based on the logic in the PageSplitter instrument. In terms of numbers though, they were a very small fraction of the total that has been logged so far in the event.mediawiki_web_ab_test_enrollment table.

FWIW, I am seeing some lower numbers of duplicate sessions between 5/4/22 and 5/5/22 for hewiki (group1 wiki which got the backport at UTC 20:00 on 5/4) as well as for euwiki (group2 wiki which got the backport at UTC 8:00 on 5/5) between earlier and later hours on 5/5. But I'm not sure that decrease can be attributed to this bug fix. And it isn't enough of a decrease to resolve the dupes altogether for either wiki. In any event, I'll keep an eye on logstash for a few more days as well as run more queries in Hue after some more time has passed and report back here with any new info.

Here is the logstash dashboard for this error (~3500 total from 4/26/22 to 5/5/22) which thankfully dropped off after the backport:

Screen Shot 2022-05-05 at 11.56.44 PM.png (1×3 px, 340 KB)

cjming mentioned this in T306559: Schema QA: mediawiki_web_ab_test_enrollment.May 6 2022, 6:36 AM

dancy unsubscribed.May 6 2022, 3:06 PM

Change 789933 had a related patch set uploaded (by Krinkle; author: Krinkle):

[mediawiki/extensions/WikimediaEvents@master] PageSplitter: Fix underlying bug in PageRandomGenerate

https://gerrit.wikimedia.org/r/789933

gerritbot added a project: Patch-For-Review.May 7 2022, 6:58 PM

Krinkle moved this task from Untriaged to Apr 2022 on the Wikimedia-production-error board.May 9 2022, 5:50 PM

@cjming Should this be moved back to code review given there is an outstanding patch?

Change 789933 merged by jenkins-bot:

[mediawiki/extensions/WikimediaEvents@master] PageSplitter: Fix underlying bug in PageRandomGenerate

https://gerrit.wikimedia.org/r/789933

In T307019#7921689, @nray wrote:

@cjming Should this be moved back to code review given there is an outstanding patch?

hi @nray -- missed your message -- just reviewed and merged

Maintenance_bot removed a project: Patch-For-Review.May 12 2022, 12:31 AM

ReleaseTaggerBot edited projects, added MW-1.39-notes (1.39.0-wmf.12; 2022-05-16); removed MW-1.39-notes (1.39.0-wmf.10; 2022-05-02).May 12 2022, 1:00 AM

• nray closed this task as Resolved.May 12 2022, 5:25 PM

• nray reassigned this task from • nray to cjming.

• nray subscribed.

	F35105690: Screen Shot 2022-05-05 at 11.56.44 PM.png
	May 6 2022, 6:30 AM

PHP Notice: Undefined offset: 2 in WikimediaEvents\PageSplitter\PageSplitterInstrumentation->getBucketClosed, ResolvedPublic3 Estimated Story PointsPRODUCTION ERRORActions