15:47:49   Minerva AB-test
15:47:49     ✖ Bucketing test
15:47:49       HeadlessChrome 73.0.3683 (Linux 0.0.0)
15:47:49     test control group is about 25% (30.8%)
15:47:49     Expected: true
15:47:49     Actual: false

This seems to be doing something stochastic, which is not cool in a test and especially not in a gated repo. From tests/qunit/skins.minerva.scripts/AB.test.js,

assert.strictEqual(
    ( userBuckets.unsampled / maxUsers > 0.4 ) &&
    ( userBuckets.unsampled / maxUsers < 0.6 ),
    true, 'test unsampled group is about 50% (' + userBuckets.unsampled / 10 + '%)' );

assert.strictEqual(
    ( userBuckets.control / maxUsers > 0.2 ) &&
    ( userBuckets.control / maxUsers < 0.3 ),
    true, 'test control group is about 25% (' + userBuckets.control / 10 + '%)' );

assert.strictEqual(
    ( userBuckets.treatment / maxUsers > 0.2 ) &&
    ( userBuckets.treatment / maxUsers < 0.3 ),
    true, 'test new treatment group is about 25% (' + userBuckets.treatment / 10 + '%)' );

The right approach is to manipulate the random number generator, mock the internals of the class, whatever it takes, but failing some percentage of the time is a problem for everyone.

Details

Related Changes in Gerrit:

	Subject	Repo	Branch	Lines +/-
	Embrace headless node-qunit tests	mediawiki/skins/MinervaNeue	master	+4 K -875

Customize query in gerrit

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		None	T225162 Fix flapping tests in gated extensions as of 2019-06-06
		Resolved	PRODUCTION ERROR	Jdlrobson	T225100 "Minerva AB test" blocking merge in unrelated repos

Event Timeline

awight created this task.Jun 5 2019, 2:32 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 5 2019, 2:32 PM

Looks like we can just pass a handful of static session IDs into the function and make sure they each map to the expected bucket.

awight added subscribers: Jdrewniak, • Niedzielski.Jun 5 2019, 2:36 PM

@awight, thanks for reporting. Is this failing frequently?

• Niedzielski added a project: Web-Team-Backlog-Archived.Jun 5 2019, 2:46 PM

In T225100#5237063, @Niedzielski wrote:

@awight, thanks for reporting. Is this failing frequently?

No, it's not particularly urgent, but according to my napkin math it would have failed something like 0.1% - 1% of the time. This adds up quickly if many gated extensions are doing similar things, so I think it's worth fixing in the medium term. With constant session IDs, you could get stronger guarantees about code correctness, with no chance for spurious failure.

awight added a parent task: T225162: Fix flapping tests in gated extensions as of 2019-06-06.Jun 5 2019, 10:09 PM

hashar added a project: Wikimedia-production-error (ARCHIVED -- Shared Build Failure).Jun 6 2019, 6:39 AM

In T225100#5237063, @Niedzielski wrote:

@awight, thanks for reporting. Is this failing frequently?

Flaky tests should be disabled. Breaking 1% of merges, cherry picks, and deployments is actively unproductive, especially with those actions already taking 15-20 minutes, and a 1% failure rate very rapidly increases the time it takes to do these actions given that it often depends on several commits passing together in the parallel Zuul gating mechanism.

If it doesn't fail frequently, that might inform the priority as to when it should be fixed, but it should be disabled first regardless unless a fix is imminent (e.g. less than a day or so?)

Is this still an issue?

ovasileva lowered the priority of this task from High to Medium.Jun 25 2019, 4:18 PM

ovasileva moved this task from Incoming to Needs Prioritization on the Web-Team-Backlog-Archived board.

Change 520651 had a related patch set uploaded (by Jdlrobson; owner: Jdlrobson):
[mediawiki/skins/MinervaNeue@master] WIP: Embrace packageFiles and headless node-qunit tests

https://gerrit.wikimedia.org/r/520651

gerritbot added a project: Patch-For-Review.Jul 3 2019, 11:12 PM

Fixed in https://gerrit.wikimedia.org/r/#/c/mediawiki/skins/MinervaNeue/+/514495/
Sorry for the delay here.

Change 523755 had a related patch set uploaded (by Jdlrobson; owner: Jdlrobson):
[mediawiki/skins/MinervaNeue@master] Embrace packageFiles

https://gerrit.wikimedia.org/r/523755

• mmodell changed the subtype of this task from "Task" to "Production Error".Aug 28 2019, 11:06 PM

Change 520651 abandoned by Jdlrobson:
Embrace headless node-qunit tests

Reason:
not working on this

https://gerrit.wikimedia.org/r/520651

Restricted Application added a subscriber: Masumrezarock100. · View Herald TranscriptDec 18 2019, 12:06 AM

"Minerva AB test" blocking merge in unrelated reposClosed, ResolvedPublicPRODUCTION ERRORActions

Description

Details

Related ObjectsSearch...

Event Timeline

"Minerva AB test" blocking merge in unrelated repos
Closed, ResolvedPublicPRODUCTION ERROR
Actions

Related Objects
Search...