Page MenuHomePhabricator

[DE 1.1.9] Define constructive edit-based metrics in GrowthBook
Closed, ResolvedPublic2 Estimated Story Points

Description

DE1.1 KR: By the end of Q2, achieve a 50% increase in the proportion of distinct registered users with ≤100 cumulative edits who publish at least one constructive edit on mobile web through controlled experiments, compared to editors in the control group.

Once we have edit_saved events produced by T431493: [DE 1.1.9] An instrument that produces edit_survived events in an Iceberg table (T433379), we can use those to define:

  • Constructive edit rate
  • Proportion of users who publish at least one constructive edit
    • This is the metric for DE1.1 KR
    • The KR text targets a very specific audience and we will use GrowthBook's Dimensions to dig deeper into experiment results and see how this metric is impacted for the target segment of users (see Slack thread for more context and details)

Acceptance criteria

  • Create a Web Edits fact table with columns:
    • subject_id, timestamp (of the edit_saved event)
    • page_namespace_id (numeric)
    • is_constructive (boolean)
  • Define metrics on top of the Web Edits fact table:
    • Constructive edit rate
    • Proportion of users who publish at least one constructive edit

Event Timeline

mpopov triaged this task as Medium priority.Jul 28 2026, 2:56 PM
mpopov set the point value for this task to 2.
JVanderhoop-WMF raised the priority of this task from Medium to High.Jul 28 2026, 3:39 PM

Defined:


Originally this task included defining a "Permanent account constructive activation rate" but we currently cannot test queries to define any fact tables as we don't have any recent experiments which produces both account_created and edit_saved events which the fact table's query requires. I have removed it from acceptance criteria / scope.

I am including my query draft for future reference:

WITH
  t_account_created AS (
    SELECT
      experiment.subject_id AS subject_id,
      timestamp AS account_created_timestamp -- when the account was created
    FROM
      wmf_experiments.experiment_event_v1
    WHERE
      timestamp BETWEEN from_iso8601_timestamp('{{ startDateISO }}') AND from_iso8601_timestamp('{{ endDateISO }}')
      AND experiment.enrolled LIKE '{{ experimentId }}'
      AND action = 'account_created'
      AND NOT performer.is_temp -- permanent accounts only
  ),
  t_edits_saved AS (
    -- Note: Article edits only!
    SELECT
      experiment.subject_id AS subject_id,
      timestamp AS edit_saved_timestamp, -- when the edit was saved
      mediawiki.database AS wiki_id,
      page.revision_id AS rev_id
    FROM
      wmf_experiments.experiment_event_v1
    WHERE
      timestamp BETWEEN from_iso8601_timestamp('{{ startDateISO }}') AND from_iso8601_timestamp('{{ endDateISO }}')
      AND experiment.enrolled LIKE '{{ experimentId }}'
      AND action = 'edit_saved'
      AND page.namespace_id = 0
  ),
  t_edits_filtered AS (
    SELECT
      tac.subject_id AS subject_id,
      tes.edit_saved_timestamp,
      tes.wiki_id,
      tes.rev_id,
      COALESCE(
        -- (article) edit was saved AFTER account was created
        (
          tes.edit_saved_timestamp > tac.account_created_timestamp
        )
        -- but DURING the first 24 hours since the account was created
        AND (
          tes.edit_saved_timestamp < tac.account_created_timestamp + INTERVAL '24' HOUR
        ),
        false -- if no (article) edit was saved at all
      ) AS edit_saved_in_first_24h
    FROM
      t_account_created tac
      LEFT JOIN t_edits_saved tes ON tac.subject_id = tes.subject_id
  ),
  t_edits_filtered_status AS (
    SELECT
      t_edits_filtered.subject_id,
      t_edits_filtered.edit_saved_timestamp,
      COALESCE(
        (NOT mwhi.revision_is_identity_reverted)
        OR (
          mwhi.revision_seconds_to_identity_revert > (48 * 3600)
        ),
        false
      ) AS is_constructive_24h
    FROM
      t_edits_filtered
      LEFT JOIN wmf_mediawiki.mediawiki_history_incremental_v1 mwhi ON (
        mwhi.event_timestamp BETWEEN from_iso8601_timestamp('{{ startDateISO }}') AND from_iso8601_timestamp('{{ endDateISO }}')
        AND mwhi.event_entity = 'revision'
        AND t_edits_filtered.wiki_id = mwhi.wiki_id
        AND t_edits_filtered.rev_id = mwhi.revision_id
      )
    WHERE
      t_edits_filtered.edit_saved_in_first_24h
  )
SELECT
  tac.subject_id,
  tac.account_created_timestamp AS timestamp,
  SUM(IF(is_constructive_24h, 1, 0)) > 0 AS is_constructively_activated_24h
FROM
  t_account_created tac
  LEFT JOIN t_edits_filtered_status tefs ON (tac.subject_id = tefs.subject_id)
GROUP BY
  1, -- subject_id
  2 -- timestamp (of account_created event)