<div class="markdown-cell"><h1>checking which cohort users have no mentor on record</h1><br><br>wikipedia has a feature where new users can get assigned a mentor (an experienced editor) at sign-up. the assignment happens server-side and is stored in a database table called <code>growthexperiments_mentor_mentee</code>. importantly, the backend assigns a mentor <strong>regardless of whether the user has the mentor ui module turned on</strong> — <code>mentorshipState</code> controls only whether the ui shows, not whether the backend writes the row. so every user in the cohort should have a row in the table.<br><br>for some users there's no row at all. i want to figure out who they are, why the records are missing, and whether the rate of missing records changes over time.<br><br>three sources of truth used below:<br><br>1. <strong><code>inputs/z_mentorship_state.tsv</code></strong> — for every user, what value their <code>mentorshipState</code> user property carries. this is the <strong>updated</strong> version (2026-06-08+) of wikipedia's official released file (<code>dataset.tsv</code> from <code>https://analytics.wikimedia.org/published/datasets/one-off/growth/growthexperiments-mentorship-enabled-T420387/</code>), renamed locally. columns: <code>userId</code>, <code>mentorshipState</code>. the column now carries three raw values instead of the original two-way enabled/disabled split:<br> - <code>unset</code> — the property was never written for this user (ui defaults to on if everything else is in place)<br> - <code>'0'</code> — the user explicitly opted out<br> - <code>'50'</code> — proactive-assignment flag (a 2024-era addition; backend kicks an async job to assign a mentor)<br><br>2. <strong><code>inputs/cov_mentor_mentee_assignment_20260530.sql.gz</code></strong> — a snapshot of the actual mentor↔mentee database table, taken on 2026-05-30. downloaded from <code>https://dumps.wikimedia.org/other/growthmentorship/</code>. each row is a (mentee_id, mentor_role, mentor_id, mentee_is_active) tuple. the <code>mentor_role</code> column can take two values:<br> - <code>primary</code> — the mentor formally assigned to this mentee. by default, questions get routed here.<br> - <code>backup</code> — a fill-in mentor written by the system when the primary mentor sets themselves "away" temporarily. a mentee can have both rows simultaneously, only one, or neither.<br><br>3. <strong><code>inputs/excl_mentor_claim_log_raw.jsonl</code></strong> — every public log entry on <code>Special:Log/growthexperiments</code>, fetched directly from the mediawiki api. these record every time a mentor was reassigned (<code>action='setmentor'</code>) or proactively claimed by a new mentor (<code>action='claimmentee'</code>). using this log, the snapshot can be "played backward" in time to figure out who the mentor was at any earlier point.<br><br>the mentorship module was rolled out gradually: 10% of new users initially, then 25%, 50%, 75%, then 100%. the boundary dates of each step come from the wikipedia mediawiki-config commit history and are listed in the next cell. the relevant identifying window is <strong>2019-09-20 to 2025-02-16</strong>.</div>
<div class="markdown-cell"><h2>Setup</h2><br><br>This notebook performs a data-quality audit of the mentorship-assignment data used in the 2SLS analysis. The audit answers a single question: for every user in Martin's mentorship-state table who does not have a corresponding row in the server-side mentor-assignment snapshot, what is the reason that row is absent?<br><br>The next cell imports libraries, defines absolute paths to the three input files, and declares the rollout phase table. The inputs are:<br><br>- <code>DATASET</code> (<code>inputs/z_mentorship_state.tsv</code>) — Martin's per-user file with two columns: <code>userId</code> and <code>mentorshipState</code>.<br>- <code>SNAPSHOT</code> (<code>inputs/cov_mentor_mentee_assignment_*.sql.gz</code>) — a MariaDB dump of the GrowthExperiments <code>cov_mentor_mentee</code> table. Each row records that a particular mentee was assigned a particular mentor server-side.<br>- <code>REG_FILE</code> (<code>build/registrations.jsonl</code>) — local en.wiki registration metadata parsed from the public dump: registration timestamp (<code>reg_ts</code>) and whether the account was self-created (<code>is_self</code>).<br><br>The <code>ROLLOUT</code> table records the production rollout phases of the mentorship feature. Phases are defined by gerrit commit dates and are used in later cells to assign each user a phase based on their registration date.<br></div>
<div class="markdown-cell"><h2>Step 1 — Load the mentorship-state table and compute the <code>no_row</code> count per state</h2><br><br>This cell reads <code>z_mentorship_state.tsv</code> into a DataFrame called <code>ds</code>, loads the mentor-assignment snapshot from a cached pickle (or rebuilds it from the SQL dump on first run), and computes — for each value of <code>mentorshipState</code> — how many users do and do not have at least one row in the snapshot.<br><br>Three values of <code>mentorshipState</code> appear in this file:<br><br>- <code>0</code> — the user was placed in the disabled arm of the mentorship A/B test (UI not shown).<br>- <code>unset</code> — no value was written to <code>user_properties</code> for this preference. Per Martin's release script (<code>notebooks/generate_dataset_for_release.ipynb</code>, function <code>convert_up_property</code>), <code>unset</code> is mapped to the enabled bucket. The reason <code>unset</code> dominates the early period is that the A/B-assignment code only runs when the user is also shown the Newcomer Homepage; users not shown the homepage have no value written and remain <code>unset</code>.<br>- <code>50</code> — present for only a handful of users. The precise meaning of <code>50</code> is not documented in this repository. Martin's release script maps it to the same enabled bucket as <code>1</code> and <code>NaN</code>. This notebook reports counts for <code>50</code> separately throughout, in case its semantics turn out to differ.<br><br>The printed totals are:<br><br>- Dataset size: <strong>4,981,433</strong> users.<br>- Per-state counts: <code>0</code> 2,457,315; <code>unset</code> 2,524,086; <code>50</code> 32.<br>- The snapshot covers <strong>5,758,864</strong> mentees with at least one assignment row.<br>- Crosstab <code>mentorshipState × has_snapshot_row</code> gives the <code>no_row</code> count per state: state <code>0</code> → 11,600 (0.47%); state <code>50</code> → 4 (12.50%); state <code>unset</code> → 540,773 (21.42%). The total <code>no_row</code> count across all states is <strong>552,377</strong>. All subsequent cells refer to these 552,377 users as the <code>no_row</code> population.<br></div>
<div class="markdown-cell"><h2>Step 2 — When does each <code>mentorshipState</code> value appear in time, and where is <code>no_row</code> concentrated?</h2><br><br>Before assigning causal reasons to the 552,377 <code>no_row</code> users, this cell describes the temporal distribution of <code>mentorshipState</code> values and of <code>no_row</code> users. It loads <code>build/registrations.jsonl</code> (cached), attaches each user's local en.wiki registration timestamp (<code>reg_ts</code>) and registration month (<code>reg_month</code>) to <code>ds</code>, and prints three monthly tables:<br><br>1. <strong>Counts per month</strong>, broken down by <code>mentorshipState</code>. This shows when each state value first appears.<br>2. <strong><code>no_row</code> count per month</strong>, broken down by <code>mentorshipState</code>. This shows the absolute volume of missing-row users month by month.<br>3. <strong><code>no_row</code> count per month as a percentage of that month's total registrants</strong>. This shows the rate at which <code>no_row</code> occurs over time.<br><br>Two facts read from the output are used later:<br><br>- State <code>0</code> does not appear until <strong>2021-09</strong> (first non-zero count 7,521). State <code>50</code> first appears in 2021-04 with a single row. Before 2021-09 every registrant in <code>ds</code> is <code>unset</code>. This is consistent with state <code>0</code> only being written after the mentorship A/B was activated.<br>- The monthly <code>no_row</code> rate is highest in the pre-rollout months and decays after 2021-06. Months from 2025-03 onward have very small total registrant counts because the dataset's right edge is the date Martin produced the file.<br><br>The remainder of the notebook decomposes the 552,377 <code>no_row</code> users into mutually exclusive reasons.<br></div>
<div class="markdown-cell"><h2>Step 3 — Enumerate the candidate reasons a user can be <code>no_row</code>, and resolve those answerable from existing fields</h2><br><br>The mentor-assignment snapshot writes a row only when the server-side assignment hook fires for a user. There are five candidate reasons for a user to be in <code>ds</code> but absent from the snapshot:<br><br>1. <strong>Pre-rollout</strong> — the user registered before mentorship was active on en.wiki (effective date <strong>2021-06-01</strong>). The assignment hook never ran for them.<br>2. <strong>Auto-created</strong> — the local en.wiki account was created by CentralAuth or by another user (not self-registration). The <code>onLocalUserCreated</code> code path is not the same as <code>onAccountCreated</code>, and auto-created accounts can miss the assignment.<br>3. <strong>Indefinitely blocked at registration</strong> — when a registration is blocked, the GrowthExperiments code drops the assignment row entirely.<br>4. <strong>No mentor available</strong> — the auto-assign mentor pool was empty at the moment of registration.<br>5. <strong>Deleted account</strong> — the local user row was deleted after registration, so neither the snapshot nor <code>build/registrations.jsonl</code> records the user.<br><br>Reasons 1 and 2 can be answered directly from fields already present in <code>ds</code> (<code>reg_ts</code>, <code>is_self</code>). Reasons 3 and 4 require an external API call to determine block status (cell 7). Reason 5 is detectable as <code>reg_ts</code> being missing.<br><br>This cell tags every <code>no_row</code> user with three boolean flags — <code>cause_pre_rollout</code> (<code>reg_ts < 2021-06-01</code>), <code>is_self == False</code> (auto-created), and <code>reg_ts_missing</code> (no local registration row) — and prints the counts for each.<br><br>Output among the 552,377 <code>no_row</code> users:<br><br>- Reason 1, pre-rollout: <strong>524,323</strong> (94.92%).<br>- Reason 2, auto-created (<code>is_self == False</code>): <strong>0</strong> (0.00%).<br>- Aside, <code>reg_ts</code> missing (preliminarily labelled "deleted"): <strong>6,451</strong> (1.17%).<br>- Remainder after excluding reasons 1 and 2: <strong>28,054</strong> (5.08%).<br><br>The label "deleted" attached to the 6,451 is preliminary; cell 13 re-checks it against CentralAuth and finds it misleading. The 28,054 remainder is the group that needs block-status information to be classified between reasons 3 and 4.<br></div>
<div class="markdown-cell"><h2>Step 3b — Export the post-rollout self-registered remainder for the block-status API query</h2><br><br>The previous cell left 28,054 <code>no_row</code> users unexplained after removing pre-rollout and auto-created. Of those, 6,451 have no <code>reg_ts</code> (preliminary "deleted"). The remaining <strong>21,603</strong> users are post-rollout, self-registered, and have a known <code>reg_ts</code>. These are the users for whom the block-status check is meaningful.<br><br>This cell collects their userIds, resolves each to a username from <code>build/registrations.jsonl</code> (the MediaWiki blocks API takes usernames, not userIds), and writes the (uid, name) pairs to <code>analysis/diagnose_missing_mentors/remainder_to_block.tsv</code>. The next cell consumes that file.<br><br>Printed output confirms: remainder <strong>21,603</strong>; names resolved <strong>21,603</strong>; output file written.<br></div>
<div class="markdown-cell">Of the 552,377 no_row users (all states: 0 + 50 + unset), this cell has so far identified:<br>- pre-rollout: 524,323<br>- deleted account (reg_ts missing): 6,451<br><br>The remaining <strong>21,603</strong> (post-rollout, self-registered) still need block lookup — exported next and resolved into reason 3 (indefinitely blocked) vs reason 4 (residual).</div>
<div class="markdown-cell"><h2>Step 4 — Fetch block status from the en.wiki API and split the remainder into reason 3 vs reason 4</h2><br><br>This cell queries the public en.wiki MediaWiki API (<code>action=query&list=blocks</code>) for each of the 21,603 users exported above, asking whether the username is currently indefinitely blocked. The results are written one record per line to <code>analysis/diagnose_missing_mentors/block_status.jsonl</code>. The fetch is resume-safe: usernames already present in the output file are skipped.<br><br>A user is classified as <strong>reason 3 (indefinitely blocked)</strong> if the API returns a block record whose <code>expiry == "infinite"</code>. The other users in the 21,603 are classified as <strong>reason 4 (no mentor available, or unknown residual)</strong> by elimination — they are post-rollout, self-registered, not deleted, and not currently indefinitely blocked, so the assignment hook should have fired. The most likely explanation is that the mentor pool was empty at their registration moment; other unknown failure modes remain possible.<br><br>Printed counts of the 21,603 remainder:<br><br>- Reason 3, indefinitely blocked: <strong>20,974</strong> (97.09%).<br>- Reason 4, no mentor available or unknown residual: <strong>629</strong> (2.91%).<br><br>Combined with reasons 1, 2 and the preliminary 6,451 "deleted" count, the five reasons sum to 524,323 + 0 + 20,974 + 629 + 6,451 = 552,377, which matches the <code>no_row</code> total exactly.<br></div>
<div class="markdown-cell"><h2>Step 5 — Did any <code>no_row</code> users actually post a mentor question?</h2><br><br>The cells above classify every <code>no_row</code> user by reason without distinguishing between users who would have used the mentorship feature and users who would not. This cell narrows the focus: among the 552,377 <code>no_row</code> users, how many actually posted a question through the GrowthExperiments mentor-questions interface? Question-askers who did not have a mentor row are the cohort whose treatment status is most likely to be misclassified in the 2SLS analysis.<br><br>This cell:<br><br>- Loads the <code>growthexperiments-mentor-questions</code> event table (one row per question event).<br>- Resolves the asker's username to a <code>userId</code> via <code>build/registrations.jsonl</code>.<br>- Intersects the askers with the <code>no_row</code> users in <code>ds</code>.<br>- Saves the intersection to <code>analysis/diagnose_missing_mentors/asked_but_no_row.tsv</code>.<br><br>Printed counts:<br><br>- Unique mentees who asked at least one question: <strong>36,430</strong>.<br>- Resolved to userId: <strong>35,102</strong> of 36,430.<br>- Askers who appear in <code>ds</code>: <strong>18,593</strong>.<br>- Askers in <code>ds</code> who are <code>no_row</code>: <strong>385</strong>. Of these, 380 are <code>unset</code>, 4 are <code>50</code>, and 1 is <code>0</code>.<br><br>The 385 question-asking <code>no_row</code> users are the population analyzed in the next two cells.<br></div>
<div class="input-area"><pre># See if any of those who didn't have a mentor row (no_row) had actually ASKED A QUESTION (and thus were more likely to be impacted by missing mentorship, if they were in the treatment group).
# ---- 1. all mentee usernames who asked a question ----
QFILE = ROOT / "inputs/d_mentor_questions.jsonl"
askers = set()
with open(QFILE) as f:
for line in f:
o = json.loads(line)
m = o.get("mentee")
if m:
askers.add(m)
print(f"unique mentees who asked a question: {len(askers):,}")
<div class="markdown-cell"><h2>Step 6 — Classify the 385 question-asking <code>no_row</code> users by reason</h2><br><br>This cell takes the 385 question-asking <code>no_row</code> users and assigns each one a reason from the five-reason classification. It joins the askers with <code>reg_ts</code>, <code>is_self</code>, and the block-status output produced in cell 7, then applies the same rules used in cell 4 and cell 7. The output is written to <code>asked_but_no_row_classified.tsv</code>.<br><br>Printed counts among the 385:<br><br>- Reason 1, pre-rollout: <strong>6</strong>.<br>- Reason 3, indefinitely blocked: <strong>373</strong>.<br>- Reason 4, residual: <strong>6</strong>.<br><br>Cross-tabulated by <code>mentorshipState</code>:<br><br>- Reason 1 × state: 6 in <code>unset</code>, 0 in <code>0</code> or <code>50</code>.<br>- Reason 3 × state: 368 in <code>unset</code>, 4 in <code>50</code>, 1 in <code>0</code>.<br>- Reason 4 × state: 6 in <code>unset</code>, 0 in <code>0</code> or <code>50</code>.<br><br>The dominant reason among question-asking <code>no_row</code> users is indefinite block (373 of 385, 96.9%). The next cell drills into the textual block reasons for these 373 users.<br></div>
<div class="input-area"><pre># See why are those people missing mentor rows, and how many of them are indefinitely blocked (reason 3) vs potentially having no mentor available (reason 4, by elimination).
# indefinitely-blocked uids from the block fetch
OUT = ROOT / "analysis/diagnose_missing_mentors/block_status.jsonl"
<div class="markdown-cell"><h2>Step 7 — First look at indefinite-block reasons (restricted to the 373 question-asker subset)</h2><br><br>This cell plots the indefinite-block reasons for the 373 question-asking indefinitely-blocked users identified above, split by <code>mentorshipState</code>. The block reason text is read from <code>block_status.jsonl</code> and bucketed into categories (such as <code>spam/promo</code>, <code>checkuser</code>, <code>sockpuppet</code>, <code>vandalism</code>, <code>not-here</code>, <code>username</code>, <code>disruption</code>, <code>harassment</code>, <code>block-evasion</code>, <code>other</code>) using string matching on the reason field.<br><br>The output is saved to <code>blocked_reason_by_state.png</code> and printed as a count table. The largest categories within the 373 are <code>spam/promo</code> (111 in <code>unset</code>), <code>checkuser</code> (63 in <code>unset</code>, plus 3 in <code>50</code> and 1 in <code>0</code>), and <code>sockpuppet</code> (59 in <code>unset</code>).<br><br>This view covers only the 373 question-asker subset of indefinitely-blocked users, which is small. Cell 14 produces the same breakdown for the full 20,974 indefinitely-blocked <code>no_row</code> population and is the authoritative version. This cell is retained to show the block-reason composition specifically among users who actually engaged with the mentor-question interface.<br></div>
<div class="markdown-cell"><h2>Step 8 — Authoritative five-reason × <code>mentorshipState</code> classification of all 552,377 <code>no_row</code> users</h2><br><br>This cell produces the authoritative reason-by-state table for the entire <code>no_row</code> population. Each user is assigned exactly one of the five reasons, in this precedence order: <code>5_deleted</code> if <code>reg_ts</code> is missing; otherwise <code>1_pre_rollout</code> if <code>reg_ts < 2021-06-01</code>; otherwise <code>2_autocreated</code> if <code>is_self == False</code>; otherwise <code>3_indef_blocked</code> if the user is in the indef-blocked set from cell 7; otherwise <code>4_residual</code>. The result is cross-tabulated against <code>mentorshipState</code>.<br><br>Printed reason × <code>mentorshipState</code> totals:<br><br>| reason | 0 | 50 | unset | total |<br>|---|---|---|---|---|<br>| 1_pre_rollout | 0 | 0 | 524,323 | 524,323 |<br>| 2_autocreated | 0 | 0 | 0 | 0 |<br>| 3_indef_blocked | 11,409 | 4 | 9,561 | 20,974 |<br>| 4_residual | 165 | 0 | 464 | 629 |<br>| 5_deleted | 26 | 0 | 6,425 | 6,451 |<br><br>A sum check inside the cell confirms the column sums match <code>no_row</code> per state (0: 11,600; 50: 4; unset: 540,773; total 552,377). The figure is saved to <code>no_row_reason_by_state.png</code> and the table to <code>no_row_reason_by_state.tsv</code>.<br><br>Two observations from this table are used downstream: reason 1 (pre-rollout) is entirely within <code>unset</code>, consistent with state <code>0</code> not appearing in the data before 2021-09; indefinitely-blocked users are concentrated in state <code>0</code> (11,409) and state <code>unset</code> (9,561), with only 4 in state <code>50</code>.<br></div>
<div class="markdown-cell"><h2>Step 9 — Re-check the "5_deleted" bucket against the CentralAuth API</h2><br><br>The label <code>5_deleted</code> was assigned in cell 4 to the 6,451 <code>no_row</code> users whose <code>reg_ts</code> is missing. "Missing local registration timestamp" is not the same as "account deleted": a user can have no local en.wiki user row but still have a live global (CentralAuth) account whose home wiki is another project. This cell tests that hypothesis by querying Meta's CentralAuth API (<code>action=query&meta=globaluserinfo</code>) for each of the 6,451 userIds, one at a time (<code>guiid</code> is not batchable), with a 0.4 s pacing between requests. The fetch is resume-safe and writes one JSON record per line to <code>analysis/diagnose_missing_mentors/deleted_global_info.jsonl</code>.<br><br>Printed results:<br><br>- Total records: <strong>6,451</strong>.<br>- Accounts that do not exist globally (<code>missing == True</code>): <strong>0</strong>.<br>- Accounts that exist but are globally locked: <strong>55</strong>.<br>- Hidden accounts: <strong>0</strong>.<br>- API errors or records without a registration timestamp: <strong>11</strong>.<br><br>Registration date distribution among the 6,440 records with a known global registration timestamp:<br><br>- Pre-rollout (<code>reg < 2021-06-01</code>): <strong>6,440</strong> (100%).<br>- In the identifying window (2022-03-07 to 2025-02-16): <strong>0</strong>.<br><br>Top home wikis (out of all 6,451): <code>enwiki</code> 3,154 (48.9%), <code>eswiki</code> 464 (7.2%), <code>frwiki</code> 252 (3.9%), <code>ruwiki</code> 213 (3.3%), <code>ptwiki</code> 204 (3.2%), <code>dewiki</code> 178 (2.8%), with the remainder spread across many other projects.<br><br>Conclusion printed at the bottom of the cell: none of the 6,451 users are actually deleted. They all have a queryable global account; all 6,440 with a known registration date registered before 2021-06-01; zero fall in the identifying window. The <code>5_deleted</code> label is therefore misleading and should be read as "pre-rollout global accounts with no local en.wiki user row" — many because their home wiki is not en.wiki. This misclassification has no effect on the cohort used for 2SLS estimation, because none of these users fall in the identifying window.<br></div>
<div class="markdown-cell"><h2>Step 10 — Block-reason breakdown for all 20,974 indefinitely-blocked <code>no_row</code> users, split by state</h2><br><br>This cell repeats the block-reason analysis from cell 11 on the full population of 20,974 indefinitely-blocked <code>no_row</code> users (rather than the 373 question-askers). Block-reason text is read from <code>block_status.jsonl</code> and bucketed using the same string-matching rules as cell 11. The outputs are saved to <code>block_reason_by_state_all.png</code>, <code>block_reason_by_state_all.tsv</code>, and <code>block_reason_share_by_state.tsv</code>.<br><br>Printed totals (top categories, absolute counts by <code>mentorshipState</code>):<br><br>| category | 0 | 50 | unset | total |<br>|---|---|---|---|---|<br>| spam/promo | 4,614 | 0 | 3,643 | 8,257 |<br>| checkuser | 1,672 | 3 | 1,626 | 3,301 |<br>| sockpuppet | 1,249 | 0 | 1,108 | 2,357 |<br>| vandalism | 1,251 | 0 | 1,104 | 2,355 |<br>| not-here | 891 | 0 | 666 | 1,557 |<br>| other | 603 | 0 | 474 | 1,077 |<br>| username | 519 | 0 | 357 | 876 |<br>| disruption | 452 | 1 | 388 | 841 |<br><br>Within-state shares are also printed. The composition is similar in state <code>0</code> and state <code>unset</code>: in both, the top three categories (spam/promo, checkuser, sockpuppet) account for roughly two-thirds of indefinite blocks. State <code>50</code> has only 4 indefinitely-blocked users and is too small to interpret.<br><br>This cell concludes the analysis of the 552,377 <code>no_row</code> users. The remaining cells are a separate completeness audit of registration timestamps, prompted by a finding made while building the for-Martin output file.<br></div>
<div class="markdown-cell"><h3>Where the <code>reg_ts MISSING</code> number comes from</h3><br><br>The next cell will print <code>users with reg_ts MISSING : 7,378</code>. That count is <strong>every uid in <code>z_mentorship_state.tsv</code> whose lookup in <code>build/registrations.jsonl</code> returned NaN</strong>, regardless of whether the uid is in the snapshot. It splits into:<br><br>- <strong>no_row & reg_ts NaN</strong> — the 6,451 "deleted"-tagged group from the 5-reason no_row breakdown above.<br>- <strong>has_row & reg_ts NaN</strong> — 927 users with a real mentor row but no local enwiki reg_ts (audited in the new cell below).<br><br>The pre-check cell below prints both sub-counts so the 7,378 doesn't appear out of nowhere.</div>
<div class="markdown-cell"><h2>Step 11 — Realized rollout share per phase, with bounds for users whose <code>reg_ts</code> is missing</h2><br><br>This cell measures how the realized Z=1 share in <code>ds</code> compares to the share implied by each rollout phase's <code>mentorship_pct</code> and <code>homepage_pct</code>. The Z mapping used here is <code>Z=1</code> if <code>mentorshipState ∈ {unset, 50}</code>, else <code>Z=0</code>.<br><br>For each phase, the cell computes:<br><br>- <strong><code>n_known</code></strong>: number of users with a known <code>reg_ts</code> whose registration falls in that phase.<br>- <strong><code>realized_baseline_%</code></strong>: realized Z=1 share among <code>n_known</code>.<br>- <strong><code>nominal_Z1_all_users</code></strong>: expected Z=1 share under the rollout configuration. For phases with <code>homepage_pct == 1.0</code>, this equals <code>mentorship_pct</code>. For the early phase <code>homepage_25pct_mentor_20pct</code>, the expected share is <code>(1 − homepage_pct) × 1 + homepage_pct × mentorship_pct</code> = 0.75 × 1 + 0.25 × 0.20 = 0.80, because users not shown the Newcomer Homepage never have the A/B-assignment code run and remain <code>unset</code> (Z=1 under this mapping).<br>- <strong>Bounds (<code>lower_all_miss_as_Z0</code>, <code>upper_all_miss_as_Z1</code>)</strong>: realized share recomputed under the two extreme assumptions about users whose <code>reg_ts</code> is missing (all assumed Z=0 vs all assumed Z=1).<br><br>The cell first prints the count of users with <code>reg_ts</code> missing — <strong>7,378</strong> (Z=1: 6,771; Z=0: 607). The two previous cells decomposed this 7,378 into 6,451 (<code>no_row & reg_ts NaN</code>) plus 927 (<code>has_row & reg_ts NaN</code>).<br><br>Per-phase realized vs nominal (identifying phases only):<br><br>| phase | window | nominal Z1 | realized baseline | diff (pp) |<br>|---|---|---|---|---|<br>| p10 | 2022-03-07 to 2023-07-10 | 10.0% | 11.753% | +1.75 |<br>| p25 | 2023-07-11 to 2023-10-04 | 25.0% | 26.145% | +1.14 |<br>| p50 | 2023-10-05 to 2025-02-02 | 50.0% | 50.574% | +0.57 |<br>| p75 | 2025-02-03 to 2025-02-16 | 75.0% | 73.899% | -1.10 |<br><br>Realized shares match the nominal targets to within 1.75 percentage points across all identifying phases. The bounds for missing-<code>reg_ts</code> users are narrow (less than 0.5 pp for p10, p25, p50), which shows that the missing-<code>reg_ts</code> cohort is too small to move the realized share appreciably.<br><br>The output table is saved to <code>rollout_realized_with_bounds.tsv</code>.<br></div>
<div class="markdown-cell"><h3>Audit: <code>reg_ts MISSING</code> is two different groups</h3><br><br>The 7,378 <code>reg_ts.isna()</code> users above are <strong>not</strong> all "deleted accounts." They split into:<br><br>| group | in snapshot? | meaning | count |<br>|---|---|---|---|<br>| A | no_row & reg_ts NaN | originally tagged <code>5_deleted</code> in the no_row breakdown; later found to be pre-rollout <strong>global</strong> CentralAuth accounts whose enwiki local registration log is missing | 6,451 |<br>| B | has_row & reg_ts NaN | has a real mentor row, but <code>build/registrations.jsonl</code> has no local enwiki reg_ts for them | <strong>927</strong> |<br><br>Group A is the one analyzed throughout this notebook (5 no_row reasons).<br>Group B was <strong>not</strong> broken out anywhere; it was surfaced only when building the for-Martin file (<code>analysis/for_martin/users_name_state_regts.tsv</code>).<br><br>The next cell audits group B: by state, by snapshot's <code>mentor_assigned_ts</code> month (a proxy for when they entered), and exports the list.</div>