Background

After enabling the sticky header edit button A/B test for pilot wikis T311144, we realized the experiment also needs to account for the use case wherein the control group does not see a sticky header at all (vs the treatment group that sees the sticky header with edit buttons).

Currently config overrides the A/B test logic that shows the sticky header with or without the edit buttons. We need the A/B test logic to override config that shows or does not show the sticky header (with or without the edit buttons).

For example the following $wgVectorStickyHeader config sets sticky header to be visible for logged in users but the A/B test config indicates that all sampled users should NOT see the sticky header.
Expected behavior: not to see the sticky header
Observed behavior: seeing the sticky header

$wgVectorStickyHeader = [
	"logged_in" =>  true,
	"logged_out" => false,
];

$wgVectorStickyHeaderEdit = [
	"logged_in" => true,
	"logged_out" => false,
];

$wgVectorWebABTestEnrollment = [
	"name" => "vector.sticky_header_edit",
	"enabled" => true,
	"buckets" => [
		"unsampled" => [
			"samplingRate" => 0
		],
		"noStickyHeaderControl" => [
			"samplingRate" => 1
		],
		"stickyHeaderEditButtonTreatment" => [
			"samplingRate" => 0
		],
	],
];

Inversely, the following $wgVectorStickyHeader config sets sticky header to be NOT visible for logged in users but the A/B test config indicates that all sampled users should see the sticky header.
Expected behavior: to see the sticky header Because of server-side feature manager for sticky header, I would expect no sticky header. The $wgVectorStickyHeader config needs to be true for logged-in users in order to run the experiment for treatment groups to see the sticky header with edit buttons so the applicable config override by the A/B test config is noted above.
Observed behavior: not seeing the sticky header

$wgVectorStickyHeader = [
	"logged_in" =>  false,
	"logged_out" => false,
];

$wgVectorStickyHeaderEdit = [
	"logged_in" => true,
	"logged_out" => false,
];

$wgVectorWebABTestEnrollment = [
	"name" => "vector.sticky_header_edit",
	"enabled" => true,
	"buckets" => [
		"unsampled" => [
			"samplingRate" => 0
		],
		"noStickyHeaderControl" => [
			"samplingRate" => 0
		],
		"stickyHeaderEditButtonTreatment" => [
			"samplingRate" => 1
		]
	],
];

Description

Adjust the sticky header A/B test logic to override config so that sampling rates of the control and treatment groups take precedence.

Acceptance criteria

• Update vector.sticky_header_edit to have a 3rd treatment group
• When running the sticky header edit button A/B test (currently applicable to idwiki + viwiki), ensure that the control group does not see sticky header and the treatment 1 group sees sticky header with edit buttons and treatment 2 group sees sticky header without edit buttons << treatment buckets are switched in latest config so that treatment 1 is sticky header without edit buttons and treatment 2 is sticky header with edit buttons
• Indicate what the default config should be for $wgVectorStickyHeader and $wgVectorStickyHeaderEdit for the relevant wikis that will run this experiment.

default config:

$wgVectorWebABTestEnrollment = [
	"name" => "vector.sticky_header_edit",
	"enabled" => true,
	"buckets" => [
		"unsampled" => [
			"samplingRate" => 0
		],
		"noStickyHeaderControl" => [
			"samplingRate" => 0.34
		],
		"stickyHeaderNoEditButtonTreatment1" => [
			"samplingRate" => 0.33
		],
		"stickyHeaderEditButtonTreatment2" => [
			"samplingRate" => 0.33
		]
	],
];

QA steps

• You may need to create a few accounts on beta cluster to see each of the following configurations:
  • control: no sticky header
  • treatment 1: sticky header without edit buttons
  • treatment 2: sticky header with edit buttons
• log into a page on beta cluster i.e. https://en.wikipedia.beta.wmflabs.org/wiki/Albert_Einstein and scroll down to trigger sticky header
  • It might take a few tries, but you should (using different accounts) be able to reproduce the experience of no sticky header at all, sticky header without edit buttons, and sticky header with edit buttons.