@GGalofre-WMF

Based on a review of existing events, we do not currently have sufficient existing instrumentation to answer the questions identified above. Events will need to be added to VisualEditorFeatureUse to track user engagement with the pre-populated article structure. In the meantime, we may need to rely on qualitative insights and potentially manual review of a sample of articles created by users in the treatment group.

Below is a summary of the information we currently track and the gaps that would require new events to be instrumented.

How do editors interact with the pre-populated article structure?

• What we track: After a user enters the Visual Editor, we send an event when someone makes a first change to an article in EditAttemptStep (action = firstChange). We also track engagement with features in the VE toolbar (ie add a citation, add a link, add a bulleted list) in VisualEditorFeatureUse.
• Gaps: We do not send an event to clarify if text was added or deleted from the pre-populated structure.

Based on this, we can only answer more broadly “Did an editor start to make a change after being shown a pre-populated article structure? This could be added as a point in the funnel analysis T432601 to understand how often users drop off before making a change once they're in the editor. But it will not provide details into specific engagement patterns with the pre-populated article structure while the user is in the Visual Editor.

Section-level interaction data if available (which sections were touched, expanded, or deleted)

• What we track: The only section-level data we track within EditAttemptStep is if a user clicks the section edit button to initiate editing for a section of the page. This only occurs for existing articles before the user is in the Visual Editor.

• Gaps: We don't track clicks to section headers once a user is in the editor. We also don't have events to track if sections get filled, left empty, or deleted while the user is in the Visual Editor.

A way to compare the article structure as pre-populated at session start against the saved article content, to measure what was kept, edited, or removed.

• What we track: For published edits, we save revision content for all pages. For example, wmf_content.mediawiki_content_current_v1 is dataset available in the Data Lake that provides the full content of the latest revisions from all Wikimedia wikis.
• Gaps: We do not have an easy way to obtain aggregate data on changes outside of a manual review of revisions for a sample set of treatment users. Since the pre-populated article structure is generated at session start, that initial state is never written to the revision database. As a result, this diff data would not be stored in mediawiki_content_diff. We would need session-level instrumentation generated during the session itself to track changes.

Edit session data for treatment editors who received a matched outline, filtered to the post-May 27 clean dataset

• What we track: Experiment data including events to identify users bucketed into the treatment group are stored in the product_metrics_web_base table. This data can be joined with EditAttemptStep to obtain relevant edit session data but we would need a unique identifer to join the two data sets.
• Gaps: We do not currently have a unique identifier to join the two datasets. We've achieved this in past Editing experiments by populating the funnel_entry_token interaction data for the experiment with the event.editing_session_id in EditAttemptStep.