Differentiate between edits to existing sections and the creation of new sections
Closed, ResolvedPublic
Actions

Description

This task is about implementing a new event that will enable the software to distinguish edits to existing sections from edits associated with the creation of new sections.

Background

Current behavior

In T269486#6746575, @DLynch wrote:

Init events with type=section and mechanism=url are going to be a mixture of new and existing sections. Intuitively I'd expect mechanism=url to be comparatively uncommon, on the assumption that most ways we'd create a new section involve just clicking the link on the page.

Implications of the "current behavior"

In T269486#6760483, @MNeisler wrote:

...we are unable to identify new section events created using the existing workflow because the init_mechanism behavior is inconsistent. This is needed if we want to make any comparisons between the two workflows and to accurately calculate metrics such as Adoption metric #4 in T263053 (how many people used the New Discussion Tool to add X proportions of new topics ).

Requirements

Add instrumentation that enables us to distinguish edits to existing sections from edits associated with the creation of new sections so that we can know things like what percentage of new talk page sections did people use the New Discussion Tool to create. See: #4 in T263053.

Open questions

1. What should the new event(s) needed to satisfy the ===Requirements above be called?

Done

The instrumentation needed to satisfy the ===Requirements is implemented
Verify the instrumentation needed to satisfy the ===Requirements is working as expected
Update the necessary schema with the instrumentation changes needed satisfy the ===Requirements

Details

Subject	Repo	Branch	Lines +/-
Logging init mechanism consistency	mediawiki/extensions/VisualEditor	master	+3 -3
Logging init mechanism for new section direct navigation	mediawiki/extensions/WikiEditor	master	+8 -1
Bump EditAttemptStep schema version to 21024286	mediawiki/extensions/WikimediaEvents	master	+1 -1

Customize query in gerrit

Related Objects
Search...

Status	Subtype	Assigned	Task
Open		None	T276497 Scale DiscussionTools to all projects
Open		None	T251207 [Epic] Scale DiscussionTools to all Wikipedias
Open		None	T233446 [Epic] New Discussion Tool
Resolved		None	T275256 Make New Topic Tool (desktop) available as opt-out feature at all wikis
Resolved		• LZaman	T271964 [Release ticket] Make New Discussion Tool available as opt-out feature at partner wikis
Resolved		ppelberg	T269471 Make New Discussion Tool available as Beta Feature at partner wikis
Resolved		ppelberg	T272544 Differentiate between edits to existing sections and the creation of new sections
			Restricted Task
Resolved		ppelberg	T275257 Offer New Discussion Tool as Beta Feature at all Wikipedias
Resolved		Ryasmeen	T275813 Post-deployment QA (New Discussion Tool)
Resolved		matmarex	T275827 Deploy config change to scale New Discussion Tool's availability as a beta feature
Resolved		Esanders	T275923 Pressing "cancel" on an empty new topic widget triggers the title validation and doesn't close the widget
Resolved		matmarex	T276967 Turn off Reply Tool A/B test
Resolved		• Whatamidoing-WMF	T275828 Make volunteers aware of New Discussion Tool's availability as beta feature
Resolved		• Whatamidoing-WMF	T275830 Verify integrity of New Discussion Tool edits at ar, cs and hu.wiki
Resolved		ppelberg	T273082 Offer Reply and New Discussion Tools as Beta Features at "Phase 4" Wikipedias.
Resolved		matmarex	T273146 Make the Reply and New Discussion Tools available as opt-in Beta Features at en.wiki
Resolved		• Whatamidoing-WMF	T273147 Consult with volunteers at en.wiki about offering the Reply Tool as a Beta Feature
Resolved		• Whatamidoing-WMF	T273233 Update en.wiki Talk pages project page
Resolved		matmarex	T276493 Make the Reply Tool and New Discussion Tools available as opt-in Beta Features at ru.wiki
Resolved		• Whatamidoing-WMF	T275255 Consult with volunteers at ru.wiki about offering the Reply Tool as a Beta Feature
Resolved		matmarex	T276494 Make the Reply and New Discussion Tools available as Beta Features at de.wiki
Resolved		• Whatamidoing-WMF	T275254 Consult with volunteers at de.wiki about offering the Reply Tool as a Beta Feature
Duplicate		None	T277506 Enable New Discussion Tool as opt-out setting at office.wiki
Declined		MNeisler	T263054 New Discussion Tool: calculate workflow engagement metrics
Resolved		matmarex	T285117 Add support for the InputBox extension
Resolved		Urbanecm_WMF	T269310 Add support for preloaded text
Resolved		Ryasmeen	T269033 Add edit notice element to New Discussion Tool
Resolved		ppelberg	T270454 Implement edit notice logic
Resolved		Esanders	T272188 VE API: Return structured information about edit notices so they can be filtered by clients
Resolved		None	T272656 Solicit feedback from page maintainers about preload opt-in approach
Resolved		None	T270797 Create way for individual pages to opt-in to using the New Discussion Tool for preloads
Resolved		ppelberg	T274746 Create an easy way to link people to an open New Discussion Tool
Resolved		ppelberg	T282204 Handle URLs for starting new sections
Resolved		• Whatamidoing-WMF	T287804 Create deployment plan for scaling New Topic Tool as opt-out at all projects
Resolved		matmarex	T287901 Revise empty Title field warning message display logic
Resolved		matmarex	T288319 "Start a discussion" empty state message should not appear on empty talk pages when corresponding page (or user) doesn't exist
Resolved		ppelberg	T290517 [Release Ticket] Introduce New Topic Tool (desktop) as opt-out at A/B test wikis
Resolved		mpopov	T277825 Run A/B test to evaluate impact of New Topic Tool
Resolved		MNeisler	T290204 [SPIKE] Estimate New Discussion Tool A/B Test Feasibility
Resolved		DLynch	T291308 Make config change to start New Discussion Tool A/B Test
Resolved		• Whatamidoing-WMF	T291306 Identify wikis for New Discussion Tool A/B test
Resolved		Ryasmeen	T291307 Implement New Discussion Tool A/B test bucketing
Resolved		ppelberg	T291536 [SPIKE] How can we ensure that we are able to identify and distinguish prior users of the New Discussion Tool in the AB test?
Resolved		ppelberg	T292120 [SPIKE] Determine how we can identify blocked save attempts
Resolved		DLynch	T293962 Introduce a "hint" that informs people how they can enable the legacy section=new experience
Resolved		ppelberg	T294491 [SPIKE] How many people have turned off the New Discussion Tool since it's been available as a Beta Feature
Resolved		DLynch	T294167 Implement way to identify people who have opened the New Discussion Tool before
Resolved		ppelberg	T297847 Ensure New Discussion Tool Hint Is Fully Translated
Resolved		ppelberg	T301496 Add distinct event to talk_page_edit for when a new section is added
Resolved		matmarex	T288741 Talk Page Empty State: write English copy for translation
Resolved	BUG REPORT	matmarex	T291085 redlink=1 not respected in zhwiki
Resolved		ppelberg	T291535 Inform participating wikis of start of New Discussion Tool A/B Test
Resolved		DLynch	T291873 Deploy config change to "turn off" New Discussion Tool A/B test
Resolved		• Whatamidoing-WMF	T295098 [SPIKE] What – if any – reactions have people had to the start of the New Discussion Tool A/B test?
Resolved		ppelberg	T297714 [SPIKE] What is the makeup of people who have adjusted the state of the Discussion tools beta feature?
Resolved	Spike	MNeisler	T301495 [SPIKE] Investigate what init type is associated with the sessions where a ready event is being emitted, but an anonymous user token is not being logged
Resolved		ppelberg	T301497 Ensure the A/B test bucket is logged when a logged out user edits a full page in wikitext
Resolved		DLynch	T301499 [SPIKE] What could be causing a ready event to be emitted, but an anonymous user token NOT to be logged
Resolved		• Whatamidoing-WMF	T304658 Announce New Topic Tool A/B test findings to wikis participating in test
Resolved		ppelberg	T306481 Add more granular experience level buckets to New Topic Tool A/B test report
Declined		MNeisler	T306579 Investigate rate at which Senior Contributors abandon New Topic Tool edits
Resolved		DLynch	T307410 [Config Change] Enable the New Topic Tool as opt-out at A/B test wikis
Declined		None	T295097 [Communications] Ensure Senior Contributors are aware of the New Discussion Tool being made available as opt-out
Resolved		ppelberg	T309368 [Config Change] Enable New Topic Tool as opt-out at Phase 3 wikis (desktop)
Declined		None	T309369 Inform volunteers at Phase 3 wikis of upcoming New Topic Tool (desktop) deployment
Resolved		matmarex	T297990 Clarify how the "Enable quick topic adding" setting affects creating new talk pages (separate setting?)
Resolved		matmarex	T310392 [Config Change] Enable New Topic Tool as opt-out at Phase 4 wikis (desktop)
Resolved		Frostly	T297974 "Title" is unclear, and should probably say "Subject" or "Topic" instead
Resolved		matmarex	T311023 Enable new topic tool by default on enwiki
Resolved		Ryasmeen	T313699 [Config Change] Enable New Topic Tool as opt-out at de.wiki (desktop)

Event Timeline

ppelberg created this task.Jan 20 2021, 8:58 PM

ppelberg mentioned this in T269486: Determine what – if any – new events need to be added to the existing section=new workflow.Jan 20 2021, 9:04 PM

ppelberg updated the task description. (Show Details)Jan 20 2021, 9:24 PM

ppelberg moved this task from Incoming to Ready to Be Worked On on the Editing-team (Kanban Board) board.

ppelberg added subscribers: DLynch, MNeisler.

ppelberg assigned this task to DLynch.Jan 21 2021, 6:09 PM

DLynch moved this task from Ready to Be Worked On to Doing on the Editing-team (Kanban Board) board.Jan 26 2021, 6:48 AM

Just to note, this is explicitly a change to the existing logging in VisualEditor / WikiEditor to get the semantics of init_mechanism tweaked so that (a) they're consistent, and (b) you can always tell whether there's a new section being created.

I don't think I'll need to make any changes to DiscussionTools for this, unless and until we implement a takeover for the section=new URL.

ppelberg edited projects, added Editing-team (Tracking); removed Editing-team (Kanban Board).Jan 27 2021, 8:56 PM

ppelberg edited projects, added Editing-team (Kanban Board); removed Editing-team (Tracking).

ppelberg moved this task from Backlog to Triaged on the DiscussionTools board.Jan 28 2021, 4:22 PM

ppelberg triaged this task as High priority.Jan 28 2021, 6:19 PM

Change 660069 had a related patch set uploaded (by DLynch; owner: David Lynch):
[mediawiki/extensions/WikimediaEvents@master] Bump EditAttemptStep schema version to 21024286

https://gerrit.wikimedia.org/r/660069

gerritbot added a project: Patch-For-Review.Jan 30 2021, 1:52 AM

Change 660075 had a related patch set uploaded (by DLynch; owner: DLynch):
[mediawiki/extensions/WikiEditor@master] Logging init mechanism for new section direct navigation

https://gerrit.wikimedia.org/r/660075

Change 660077 had a related patch set uploaded (by DLynch; owner: DLynch):
[mediawiki/extensions/VisualEditor@master] Logging init mechanism consistency

https://gerrit.wikimedia.org/r/660077

These patches add a new possible value for init_mechanism: url-new. It will be logged when direct navigation occurs to a URL that triggers an editor for either a new page or a new section.

The VE patch also makes its use of init_mechanism=new consistent with WikiEditor/the-schema-docs. It was previously not using new when you clicked the "Create" tab after navigating to a non-existent page. (You could still distinguish this case by looking for page_id=0.)

DLynch moved this task from Doing to Code Review on the Editing-team (Kanban Board) board.Jan 30 2021, 2:05 AM

Change 660069 merged by jenkins-bot:
[mediawiki/extensions/WikimediaEvents@master] Bump EditAttemptStep schema version to 21024286

https://gerrit.wikimedia.org/r/660069

ReleaseTaggerBot added a project: MW-1.36-notes (1.36.0-wmf.29; 2021-02-02).Jan 30 2021, 6:00 AM

Change 660077 merged by jenkins-bot:
[mediawiki/extensions/VisualEditor@master] Logging init mechanism consistency

https://gerrit.wikimedia.org/r/660077

Change 660075 merged by jenkins-bot:
[mediawiki/extensions/WikiEditor@master] Logging init mechanism for new section direct navigation

https://gerrit.wikimedia.org/r/660075

Maintenance_bot removed a project: Patch-For-Review.Feb 1 2021, 5:10 PM

META
Assigning this over to @MNeisler considering the necessary patches have been merged.

Although, QA is currently blocked considering these new events being logged depend on 1.36.0-wmf.29 being fully rolled out which has not yet happened. See: T271343.

ppelberg moved this task from Backlog to Analytics on the Editing-team (Tracking) board.Feb 3 2021, 8:10 PM

MNeisler added a project: Product-Analytics (Kanban).Feb 3 2021, 9:27 PM

MNeisler moved this task from Next 2 weeks to Blocked on the Product-Analytics (Kanban) board.Feb 4 2021, 7:36 PM

DLynch mentioned this in T273554: Make config change to enable Reply Tool A/B test .Feb 8 2021, 7:41 PM

In T272544#6801681, @ppelberg wrote:

Although, QA is currently blocked considering these new events being logged depend on 1.36.0-wmf.29 being fully rolled out which has not yet happened. See: T271343.

Update, this ticket is now ready for QA. considering 1.36.0-wmf.30 has arrived at all Wikipedias.

MNeisler moved this task from Blocked to Next 2 weeks on the Product-Analytics (Kanban) board.Feb 19 2021, 8:45 PM

MNeisler moved this task from Next 2 weeks to Doing on the Product-Analytics (Kanban) board.Feb 22 2021, 11:59 PM

ppelberg moved this task from To Triage to Triaged on the VisualEditor board.Feb 23 2021, 4:12 PM

@ppelberg
I completed QA of the new instrumentation changes and confirmed that the events are logging as expected and we can now differentiate between edits to existing sections vs the creation of new section using current VE/Wikieditor workflows.

See summary of checks and findings below:
(Based on data from 15 Feb 2021 through 23 Feb 2021 logged in EditAtttemptStep)

We are logging both init_mechanism = 'url-new' and init_mechanism = 'url' events for page and section init_types. This will allow us to now distinguish between new and existing section or page VisualEditor/WikiEditor edits that occur from direct navigation to a URL.
The number of logged events for each init_mechanism type seem reasonable given the likelihood of occurrence. The majority of edits to create a new section (78%) (as indicated by init_mechanism = 'new' or 'url-new and init_type = section) are created by clicking on a link to a page vs direct navigation to a URL that triggers an editor for a new section. The majority of edits from direct navigation to a url (98.9%) are page (init_type = page) vs section edits.
Discussion tool events (as indicated by event.integration = 'discussiontools) are only recorded for init_mechnaism = click events. This is expected as these instrumentation changes were not implemented for DiscussionTools. We can still distinguish new vs existing events for discussion tools based on current instrumentation. See T265099 for details

Edit Attempts by Init Mechanism and Type
Note: Data reflects all init events across all namespaces and wikis.

init_mechanism	init_type	integration	n_events
click	page	discussiontools	9418
click	page	page	235317
click	section	discussiontools	459
click	section	page	181493
new	page	page	156614
new	section	page	7989
url	page	page	112989
url	section	page	165317
url-new	page	page	223159
url-new	section	page	2285

New and existing sections edit attempts to talk pages
Note: Data reflects all init events across all talk namespaces and wikis

section_status	init_mechanism	n_events
existing	click	7319
existing	url	3226
new	new	6097
new	url-new	1543

The number of logged events for each section edit type seem reasonable given the liklihood of occurrence.
- Since deployment of the instrumentation changes, a slight majority of section edits on talk pages have been to existing sections (58%).
- Most edits (79.8%) to create new sections on talk pages are completed by clicking the link on the page vs direct navigation to a url as expected.

Further details documented in the QA notebook.

MNeisler reassigned this task from MNeisler to ppelberg.Feb 24 2021, 10:47 PM

ppelberg edited projects, added Editing-team (Kanban Board); removed Editing-team (Tracking).Feb 24 2021, 10:48 PM

ppelberg moved this task from Code Review to Ready for Sign Off on the Editing-team (Kanban Board) board.

This looks great – thank you, @MNeisler.

happy5214 moved this task from Backlog to Closed on the WikiEditor board.Jul 17 2021, 5:21 AM