Document guidelines, scenarios, and recommendations on exposure logging
Closed, ResolvedPublic5 Estimated Story Points
Actions

Assigned To

Authored By

	mpopov
	Jan 15 2026, 8:52 PM

Description

NOTE: T414729: Add Experiment#sendExposure method to SDKs includes basic instructions on usage when conducting experiments, this task is about more extensive documentation

Once exposure logging is implemented, we should provide developers with more thorough guidance for logging exposure in their experiments to ensure accuracy and trustworthiness of the data collected.

Notes

For everyone (all user traffic) experiments (that use edge uniques for enrollment) with server-side changes, we can borrow ideas from this approach:

In T414732#11570245, @mpopov wrote:

Marco from Reader Growth shared their server-side implementation with me. In summary:

The code makes use of a helper function isInAnyTreatmentGroup which checks if the request is assigned to any group for a given experiment

If the request is assigned to any group and satisfies other conditions (e.g. main namespace, skin), it loads ext.readerExperiments.imageBrowsing module and also adds <div id="ext-readerExperiments-imageBrowsing"></div> to the response.

Therefore, all users enrolled in the experiment would have this client-side module loaded.

The module then checks for enrollment/assignment and

If user is in control group, do nothing to div#ext-readerExperiments-imageBrowsing

If user is in treatment group, populate div#ext-readerExperiments-imageBrowsing

With this pattern, they could easily call Experiment#logExposure() on the client-side and have it be accurate (only log exposure when there is actually exposure to any variation).

So they effectively have a solution to this task simply by only loading & running client-side code using server-side logic. As long as they continue to use this pattern for implementing their experiments, they're all good.

We can provide a recommendation for how to log exposure on the client-side, namely by using server-side logic to conditionally load client-side instrumentation which would include an exposure logging call.

In T414732#11582833, @phuedx wrote:

Some thoughts here:

We already add classes to the <body> element when the user has been enrolled in an experiment. This is already a signal from the server

Adding additional markup can break existing styles. Developers will be aware of this though

As I mentioned elsewhere, there's a lot of nuance around RL modules and their cost. Defining a new RL module adds a cost to all pageviews. RL modules and are minified in the context of the other modules that are being loaded at the same time, so sometimes it's more performant to include a new file in an old module (see the ext.wikimediaEvents RL module, which holds a number of experiment-related instruments)

There isn't going to be a one-size-fits-all solution. The general guidance of "Only log exposure when the user is actually exposed to the experiment" with a library of examples might have to be good enough

There are some cases where we could automate this, e.g. with special markup, but I'm not convinced that this will be a good developer experience for anything other than a simple experiment. There will come a time when enough experiments have been run that we can extract patterns from their implementations but we're not there yet

Acceptance criteria

https://wikitech.wikimedia.org/wiki/Test_Kitchen/Experiment_exposure_logging includes examples/scenarios, do's & don't's, recommended patterns / best practices to help developers:
- maximize true positives (logging exposure when exposure actually took place)
- minimize false positives (logging exposure when there wasn't any exposure) and false negatives (failing to log exposure when exposure should have been logged)
https://wikitech.wikimedia.org/wiki/Test_Kitchen/Conduct_an_experiment links to https://wikitech.wikimedia.org/wiki/Test_Kitchen/Experiment_exposure_logging

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		mpopov	T411677 SDS 2.2.5 Exposure Logging
		Resolved		mpopov	T414735 Document guidelines, scenarios, and recommendations on exposure logging

Event Timeline

mpopov created this task.Jan 15 2026, 8:52 PM

KReid-WMF triaged this task as High priority.Jan 20 2026, 5:57 PM

KReid-WMF moved this task from Incoming to READY TO GROOM on the Test Kitchen board.

mpopov mentioned this in T414729: Add Experiment#sendExposure method to SDKs.Jan 22 2026, 8:46 PM

mpopov updated the task description. (Show Details)Jan 23 2026, 3:54 PM

KReid-WMF edited projects, added Test Kitchen (Experiment Platform Sprint 19); removed Test Kitchen.Jan 29 2026, 4:52 PM

KReid-WMF set the point value for this task to 5.

mpopov mentioned this in T414732: Ability to log exposure client-side using server-side trigger/flag.Jan 30 2026, 5:53 PM

mpopov updated the task description. (Show Details)Jan 30 2026, 6:03 PM

mpopov updated the task description. (Show Details)Feb 4 2026, 3:25 PM

mpopov added a subscriber: phuedx.

I think exposure logging guidance probably warrants its own guide/page which is then linked to from the conduct an experiment guide.

mpopov updated the task description. (Show Details)Feb 4 2026, 10:25 PM

mpopov lowered the priority of this task from High to Medium.Feb 11 2026, 6:18 PM

KReid-WMF edited projects, added Test Kitchen (Experiment Platform Sprint 20); removed Test Kitchen (Experiment Platform Sprint 19).Feb 19 2026, 4:11 PM

KReid-WMF edited projects, added Test Kitchen (Experiment Platform Sprint 21); removed Test Kitchen (Experiment Platform Sprint 20).Mar 12 2026, 3:10 PM

mpopov updated the task description. (Show Details)Mar 16 2026, 3:19 PM

Michael subscribed.Mar 17 2026, 1:15 PM

mpopov renamed this task from Update documentation with guidance on exposure logging to Document guidelines, scenarios, and recommendations on exposure logging.Mar 17 2026, 1:44 PM

mpopov updated the task description. (Show Details)Mar 21 2026, 8:08 PM

mpopov claimed this task.Mar 23 2026, 2:10 PM

mpopov added a project: Product-Analytics (Kanban).

mpopov moved this task from Ready for Development to In Progress on the Test Kitchen (Experiment Platform Sprint 21) board.

mpopov moved this task from Next 2 weeks to Doing on the Product-Analytics (Kanban) board.Mar 23 2026, 2:22 PM

mpopov updated the task description. (Show Details)Mar 23 2026, 2:25 PM

mpopov moved this task from In Progress to Needs Review on the Test Kitchen (Experiment Platform Sprint 21) board.Mar 24 2026, 10:49 PM