@cchen and I had a conversation with @SBisson last week, and based on that conversation, I think it is better to split the baseline measurements based on the change being done and measured appropriately. A single measure, for example just article quality may not work as there are multiple changes. The changes being done within the CX workflow are: (@SBisson: please correct if I am wrong)

• Changes in suggestions to newcomers for article expansions.
• Changes in suggestions to newcomers for new article creations.
• More emphasis on article expansions rather than creations; change in user interface to place expansions over new page creations (note: this change impacts all users).

For three various changes, the measurement approach will be different. I will be listing options from my knowledge of the tool and the objective area, and @cchen can add further details and finalize before starting to measure the baselines.

Change in suggestions to newcomers for article expansions

If newcomers are expanding existing articles, the quality of the articles may be improved, as measured through the standard quality criteria. Currently, newcomers would be expanding the articles already, whether new suggestions are better can be measured in the following way:

• revert rate of article expansions made by newcomers using CX/SX.
  • success can be reduced revert rate for article expansions as compared to current
• increased quality of articles expanded by newcomers using CX/SX.
  • this would be tricky to measure as a baseline since we don't know which articles would be suggested to users, so that we can calculate baseline quality vs. what is improved.
  • it is tricky to generalize as well - one way to think about is, average increase in article quality for expansions by newcomers - pre/post.
    • if currently, on average, increase in article quality for expansions by newcomers is 0.2, then with new suggestions success could be something like 0.25. However, if the intervention is focused on surfacing smaller blocks of content so that it is easier, then the average increase would actually be lower than before.
  • given all the above, it maybe better to use revert rate for baselines and measurement, and then article quality increase can be a secondary metric to report, and further understand the impact.

Changes in suggestions to newcomers for new article creations

The metrics could be:

• deletion rate of new page translations by newcomers
• if recommendations surface smaller articles such that it is easier for newcomers to translate, then standard quality criteria will not suit this use case. Essentially, these articles will likely miss the length and number of sections criteria to start with.

Emphasis on article expansions rather than creations

If UI changes are implemented within the dashboard, then we should see more users selecting translation_type as section rather than article in the CX events. The baseline and the measurements can be proportion of users who initiate article translations vs. section translations. It would be better to also include some guardrails, such as transition rate to dashboard_translation_start step should not drop too much, across all user experience levels.

The target group of users should be those who open the dashboard directly, i.e. dashboard_open event type having event_source as direct. All baselines should be calculated based on sessions initiated by directly opening the dashboard. Also, "newcomers" should be defined, for example, 0-99 edits. @SBisson: please let us know if you're using a different definition for the recommendations system.

Here are the baselines using data between April 1, 2025, and July 1, 2025. Please refer to this notebook for detailed calculations.

Some notes for the baseline calculations:

• For 'new editors', we are looking at users in the '0-99' edit bucket.
• Previous research on variables affecting the deletion rate of articles, the size of the target language Wikipedia is an important variable in the quality of the translation. For the baselines, we will break down the deletion rate/revert rate into 5 target Wikipedia rank bins: 1-5, 6-10, 11-20, 20-50, 51-max.
• The target group of interest is new editors who open the dashboard directly, and ideally revert and deletion rates would be calculated only for sessions initiated this way. However, due to data issues—specifically missing events (T395493) and missing revision IDs (T402975)—we do not have sufficient historical data to reliably track translations initiated via direct dashboard access. Therefore, our current calculations of revert and deletion rates use all sessions by new editors rather than being limited to dashboard-initiated sessions.

Change in suggestions to newcomers for article expansions

• Revert rate of article translations made by new editors
• Success can be reduced revert rate for article expansions as compared to the current.
• Baselines:

Changes in suggestions to newcomers for new article creations

• Deletion rate of new page translations by new editors
• Success can be reduced deletion rate for new page translations as compared to the current.
• Baselines:

Emphasis on article expansions rather than creations

• Proportion of users who initiate article translations vs. section translations
• We should see more users selecting translation_type as a section rather than an article
• We are looking at users who open the dashboard directly and then initiate translations
• Baselines: The proportion of new editors who initiate two types of translations is 84.24% for article translations and 15.75% for section translations.

Notebook for tracking baselines: https://gitlab.wikimedia.org/conniecc1/we2.1.2-kr-2025/-/blob/main/baselines-tracking.ipynb

Change start_date, end_date and snapshot for updated data.