After an audit discussion with @MNeisler around the current Account creation rate metric possible problems, we identified a potential pitfall being how the query is currently written. As it is now, there's no guarantee that the subjects for which we counted their experiment_exposure match the subjects for which we counted their successful registration via account_created event. That's because all users that try to register an account within the experiment via any link in our site, not just the anon warning, will have an enrollment attempt and potentially be part of the experiment even if they were never exposed to the anon warning.

The data engineering proposal was to limit the cohort of users for which we count the account registrations to the ones that have indeed been exposed to the warning. Using this approach (sanity checks and feedback welcome),

same as we observed for all users that created accounts within the experiment:

While the exposures look balanced, control: 87660| treatment: 87166, both the exposed and not exposed users seem to have the same pattern here. There's still T421152 open with an unclear impact. Another explanation is that our accountCreated query param signal is flawed for some reason in favor of the control group. We could and probably should run an AA test for this metric see if we're missing something but I'm afraid it will only give a confirmation on the issue rather than additional information. Ideas welcome, cc @KStoller-WMF @Michael @mpopov

Just chatted with Megan about this. Here's what the query should be:

WITH t_exposed AS (
  SELECT DISTINCT
    experiment.subject_id
  FROM {table}
  WHERE {where_boilerplate}
    AND action = 'experiment_exposure'
)
SELECT
  IF(experiment.assigned = 'control', 'control', 'treatment') AS variation,
  t_exposed.subject_id,
  CAST(SUM(IF(action = 'account_created', 1, 0)) > 0 AS INT) AS outcome
FROM t_exposed
LEFT JOIN {table} t_src ON t_exposed.subject_id = t_src.experiment.subject_id
WHERE {where_boilerplate}
GROUP BY 1, 2

The results from currently collected data are:

(No SRM)

I guess the two possible explanations are:

• Either there is a bug in the instrumentation where account_created events aren't fired as often as they should specifically for the treatment group
• Or… treatment (1.4%) really is performing 15% worse than control (1.6%), which seems unexpected but it's certainly possible to encounter counter-intuitive results.

So I guess it comes down to: how much trust do you have in the instrumentation?

Change #1269004 had a related patch set uploaded (by Sergio Gimeno; author: Sergio Gimeno):

[mediawiki/extensions/WikimediaEvents@master] loggedOutWarning: instrument browser navigation and tab close

https://gerrit.wikimedia.org/r/1269004

bearloga opened https://gitlab.wikimedia.org/repos/product-analytics/test-kitchen/experiment-analytics-configs/-/merge_requests/55

Update metrics used in logged-out warning experiment

bearloga merged https://gitlab.wikimedia.org/repos/product-analytics/test-kitchen/experiment-analytics-configs/-/merge_requests/55

Update metrics used in logged-out warning experiment

Updated Account creation rate query in the catalog and reloaded the definition, new results up in the dashboard (same as what I shared in the comment earlier).

Change #1269004 merged by jenkins-bot:

[mediawiki/extensions/WikimediaEvents@master] loggedOutWarning: instrument browser navigation and tab close

https://gerrit.wikimedia.org/r/1269004

Change #1280226 had a related patch set uploaded (by Sergio Gimeno; author: Sergio Gimeno):

[mediawiki/extensions/WikimediaEvents@wmf/1.46.0-wmf.26] loggedOutWarning: instrument browser navigation and tab close

https://gerrit.wikimedia.org/r/1280226

Change #1280226 merged by jenkins-bot:

[mediawiki/extensions/WikimediaEvents@wmf/1.46.0-wmf.26] loggedOutWarning: instrument browser navigation and tab close

https://gerrit.wikimedia.org/r/1280226

Mentioned in SAL (#wikimedia-operations) [2026-05-05T12:40:47Z] <sgimeno@deploy1003> Started scap sync-world: Backport for [[gerrit:1280226|loggedOutWarning: instrument browser navigation and tab close (T421518)]]

Mentioned in SAL (#wikimedia-operations) [2026-05-05T12:41:37Z] <sgimeno@deploy1003> sgimeno: Backport for [[gerrit:1280226|loggedOutWarning: instrument browser navigation and tab close (T421518)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Mentioned in SAL (#wikimedia-operations) [2026-05-05T12:44:43Z] <sgimeno@deploy1003> Finished scap sync-world: Backport for [[gerrit:1280226|loggedOutWarning: instrument browser navigation and tab close (T421518)]] (duration: 03m 56s)

Waiting on T425604: mediawiki.product_metrics.contributors.experiments - '.action_context' should be string to analyze data.

With all the validation issues fixed I was able to run some manual analysis with the reworked CTR instrumentation. The new added metrics and queries can be reviewed in [[ https://gitlab.wikimedia.org/sgimeno/experiment-analytics-configs/-/commit/447df61105300655954712a972183bfdcf0b903b | [DNM] Logged-out anon warning: add re-worked CTRs ]]. I used a query checker template with the following metrics and look at a day snapshot of today, (start 2026-05-14, end 2026-05-15); these where the results:

As we can see the overall CTR is now higher than 1. That may indicate some re-counting precision error but overall the numbers make a lot of sense:

• ~57% of users click on back
• ~13% of users close the tab
• ~20% of users close the editor using the editor button

This is about the 90% of the page interaction, the remaining click through goes to the page links we had already weighted around 10-12% of the page interaction. With this numbers I believe we can conclude the investigation, but @KStoller-WMF has the last call.

Note: I made a mistake on the mode_switch interaction for which I didn't create artificial impressions, I'm trying to work out this problem by using the impressions of another interface but I don't think the mode switch CTR will alter much what we already know. For the snapshot analysis there are around 40K subjects per group and I could only observe around 40 mode switches, so this CTR will probably be very low.