Page MenuHomePhabricator
Create Task
Maniphest T265099

Differentiate between events emitted from the Reply Tool and the New Discussion Tool
Closed, ResolvedPublic
Actions

Description

This task is about implementing a way for Editing Engineering and Product Analytics to differentiate between events that are emitted from the Reply Tool and events that are emitted from the New Discussion Tool.

Background

Currently, EditAttemptStep contains one integration value for DiscussionTool features. This value is called discussiontools.

To date, we've been able to assume all integration-discussiontools events are emitted from the Reply Tool.

Trouble is, once the New Discussion Tool is deployed, it will no longer be clear what editor integration – the Reply Tool or the New Discussion Tool – is responsible for the integration-discussiontools events we will be seeing.

Implementation details

Use the EditAttemptStep schema's existing init_type field to differentiate between events emitted from the Reply Tool and the New Discussion Tool.

Once implemented, events from the Reply Tool and New Discussion Tool should be logged as follows:

  • Reply Tool events: event.action = 'init', event.integration = 'discussiontools', event.init_type = 'page'
  • New Discussion Tool events: event.action = 'init', event.integration = 'discussiontools', event.init_type = 'section'

The above is the outcome of what was discussed in T265099#6561327, T265099#6571916 and T265099#6604344.

Done

  • The changes detailed in the ===Implementation details section are implemented so that we (Editing and Product Analytics) are able to differentiate between events emitted from the Reply Tool and events emitted from the New Discussion Tool.

Details

SubjectRepoBranchLines +/-
Give new-section a specific init_type to distinguish itmediawiki/extensions/DiscussionToolsmaster+9 -1
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 OpenNoneT276497 Scale DiscussionTools to all projects
 OpenNoneT251207 [Epic] Scale DiscussionTools to all Wikipedias
 OpenNoneT233446 [Epic] New Discussion Tool
 OpenNoneT275256 Make New Topic Tool (desktop) available as opt-out feature at all wikis
 Resolved• LZamanT271964 [Release ticket] Make New Discussion Tool available as opt-out feature at partner wikis
 Resolved• ppelbergT269471 Make New Discussion Tool available as Beta Feature at partner wikis
 Resolved• ppelbergT265099 Differentiate between events emitted from the Reply Tool and the New Discussion Tool
Restricted Task
 Resolved• ppelbergT275257 Offer New Discussion Tool as Beta Feature at all Wikipedias
 ResolvedRyasmeenT275813 Post-deployment QA (New Discussion Tool)
 ResolvedmatmarexT275827 Deploy config change to scale New Discussion Tool's availability as a beta feature
 ResolvedEsandersT275923 Pressing "cancel" on an empty new topic widget triggers the title validation and doesn't close the widget
 ResolvedmatmarexT276967 Turn off Reply Tool A/B test
 Resolved• Whatamidoing-WMFT275828 Make volunteers aware of New Discussion Tool's availability as beta feature
 Resolved• Whatamidoing-WMFT275830 Verify integrity of New Discussion Tool edits at ar, cs and hu.wiki
 Resolved• ppelbergT273082 Offer Reply and New Discussion Tools as Beta Features at "Phase 4" Wikipedias.
 ResolvedmatmarexT273146 Make the Reply and New Discussion Tools available as opt-in Beta Features at en.wiki
 Resolved• Whatamidoing-WMFT273147 Consult with volunteers at en.wiki about offering the Reply Tool as a Beta Feature
 Resolved• Whatamidoing-WMFT273233 Update en.wiki Talk pages project page
 ResolvedmatmarexT276493 Make the Reply Tool and New Discussion Tools available as opt-in Beta Features at ru.wiki
 Resolved• Whatamidoing-WMFT275255 Consult with volunteers at ru.wiki about offering the Reply Tool as a Beta Feature
 ResolvedmatmarexT276494 Make the Reply and New Discussion Tools available as Beta Features at de.wiki
 Resolved• Whatamidoing-WMFT275254 Consult with volunteers at de.wiki about offering the Reply Tool as a Beta Feature
 DuplicateNoneT277506 Enable New Discussion Tool as opt-out setting at office.wiki
 DeclinedMNeislerT263054 New Discussion Tool: calculate workflow engagement metrics
ResolvedmatmarexT285117 Add support for the InputBox extension
 ResolvedUrbanecm_WMFT269310 Add support for preloaded text
ResolvedRyasmeenT269033 Add edit notice element to New Discussion Tool
 Resolved• ppelbergT270454 Implement edit notice logic
 ResolvedEsandersT272188 VE API: Return structured information about edit notices so they can be filtered by clients
 ResolvedNoneT272656 Solicit feedback from page maintainers about preload opt-in approach
 ResolvedNoneT270797 Create way for individual pages to opt-in to using the New Discussion Tool for preloads
 Resolved• ppelbergT274746 Create an easy way to link people to an open New Discussion Tool
 Resolved• ppelbergT282204 Handle URLs for starting new sections
 Resolved• Whatamidoing-WMFT287804 Create deployment plan for scaling New Topic Tool as opt-out at all projects
 ResolvedmatmarexT287901 Revise empty Title field warning message display logic
 ResolvedmatmarexT288319 "Start a discussion" empty state message should not appear on empty talk pages when corresponding page (or user) doesn't exist
 Resolved• ppelbergT290517 [Release Ticket] Introduce New Topic Tool (desktop) as opt-out at A/B test wikis
 ResolvedmpopovT277825 Run A/B test to evaluate impact of New Topic Tool
 ResolvedMNeislerT290204 [SPIKE] Estimate New Discussion Tool A/B Test Feasibility
 ResolvedDLynchT291308 Make config change to start New Discussion Tool A/B Test
 Resolved• Whatamidoing-WMFT291306 Identify wikis for New Discussion Tool A/B test
 ResolvedRyasmeenT291307 Implement New Discussion Tool A/B test bucketing
 Resolved• ppelbergT291536 [SPIKE] How can we ensure that we are able to identify and distinguish prior users of the New Discussion Tool in the AB test?
 Resolved• ppelbergT292120 [SPIKE] Determine how we can identify blocked save attempts
 ResolvedDLynchT293962 Introduce a "hint" that informs people how they can enable the legacy section=new experience
 Resolved• ppelbergT294491 [SPIKE] How many people have turned off the New Discussion Tool since it's been available as a Beta Feature
 ResolvedDLynchT294167 Implement way to identify people who have opened the New Discussion Tool before
 Resolved• ppelbergT297847 Ensure New Discussion Tool Hint Is Fully Translated
 Resolved• ppelbergT301496 Add distinct event to talk_page_edit for when a new section is added
 ResolvedmatmarexT288741 Talk Page Empty State: write English copy for translation
 ResolvedBUG REPORTmatmarexT291085 redlink=1 not respected in zhwiki
 Resolved• ppelbergT291535 Inform participating wikis of start of New Discussion Tool A/B Test
 ResolvedDLynchT291873 Deploy config change to "turn off" New Discussion Tool A/B test
 Resolved• Whatamidoing-WMFT295098 [SPIKE] What – if any – reactions have people had to the start of the New Discussion Tool A/B test?
 Resolved• ppelbergT297714 [SPIKE] What is the makeup of people who have adjusted the state of the Discussion tools beta feature?
 ResolvedSpikeMNeislerT301495 [SPIKE] Investigate what init type is associated with the sessions where a ready event is being emitted, but an anonymous user token is not being logged
 Resolved• ppelbergT301497 Ensure the A/B test bucket is logged when a logged out user edits a full page in wikitext
 ResolvedDLynchT301499 [SPIKE] What could be causing a ready event to be emitted, but an anonymous user token NOT to be logged
 Resolved• Whatamidoing-WMFT304658 Announce New Topic Tool A/B test findings to wikis participating in test
 Resolved• ppelbergT306481 Add more granular experience level buckets to New Topic Tool A/B test report
 DeclinedMNeislerT306579 Investigate rate at which Senior Contributors abandon New Topic Tool edits
ResolvedDLynchT307410 [Config Change] Enable the New Topic Tool as opt-out at A/B test wikis
 OpenNoneT295097 [Communications] Ensure Senior Contributors are aware of the New Discussion Tool being made available as opt-out
Resolved• ppelbergT309368 [Config Change] Enable New Topic Tool as opt-out at Phase 3 wikis (desktop)
 OpenNoneT309369 Inform volunteers at Phase 3 wikis of upcoming New Topic Tool (desktop) deployment
 ResolvedmatmarexT297990 Clarify how the "Enable quick topic adding" setting affects creating new talk pages (separate setting?)
 ResolvedmatmarexT310392 [Config Change] Enable New Topic Tool as opt-out at Phase 4 wikis (desktop)
 ResolvedFrostlyT297974 "Title" is unclear, and should probably say "Subject" or "Topic" instead
 OpenNoneT310215 Some English Wikipedia templates with signatures in the middle don't work well with reply and new topic tool
 ResolvedmatmarexT311023 Enable new topic tool by default on enwiki
 ResolvedRyasmeenT313699 [Config Change] Enable New Topic Tool as opt-out at de.wiki (desktop)

Event Timeline

• ppelberg created this task.Oct 9 2020, 12:06 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 9 2020, 12:06 AM
• ppelberg moved this task from Incoming to Upcoming on the Editing-team (Kanban Board) board.Oct 9 2020, 12:06 AM
• ppelberg added a project: DiscussionTools.
JTannerWMF assigned this task to DLynch.Oct 14 2020, 4:39 PM
MNeisler mentioned this in T258981: Publish New Discussion Tool measurement plan.Oct 14 2020, 7:48 PM
• ppelberg moved this task from Upcoming to Ready to Be Worked On on the Editing-team (Kanban Board) board.Oct 19 2020, 3:45 PM
DLynch added a comment.Oct 19 2020, 4:58 PM

One option would be to use the existing init_type field, which is currently being sent as page but could be sent as section for this case. Advantage there is that it's just one event to change, and doesn't require any schema change. Disadvantage is that querying "event X in a session whose init was of type Y" is a more complicated query. It's consistent with how we currently log this sort of difference in VE/wikieditor though.

The other option is option is to make a new integration value, discussiontools-newtopic or whatever. Advantage: easier queries. Disadvantage: needs a schema change, becomes a (slightly) more complex query for things which you want to ask about that're common between comment replies and new topics.

JTannerWMF added a project: Product-Analytics.Oct 21 2020, 5:52 PM
JTannerWMF subscribed.

@MNeisler , David needs you to weigh in on an approach

MNeisler added a comment.Oct 22 2020, 4:11 PM

Thanks @DLynch for outlining the options and @JTannerWMF .for the ping

I'm leaning towards the option of using the existing init_type field. While it might be a slightly more complicated query, I think being consistent with how these events are currently stored in the schema will avoid confusion.

Can you confirm if the following is correct to make sure I understand how the two types of events will be logged?

  • Reply Events: All events in sessions where event.action = 'init', event.integration = 'discussiontools', event.init_type = 'page'
  • Discussion Tool Events: All events in sessions where event.action = 'init', event.integration = 'discussiontools', event.init_type = 'section'
• ppelberg moved this task from Ready to Be Worked On to Doing on the Editing-team (Kanban Board) board.Oct 22 2020, 5:29 PM
MNeisler moved this task from Triage to Tracking on the Product-Analytics board.Oct 23 2020, 10:28 PM
DLynch added a comment.Nov 4 2020, 6:26 PM

If by "Discussion Tool Events" you mean "New discussion section events", then yes.

MNeisler added a comment.Nov 4 2020, 9:03 PM
In T265099#6603923, @DLynch wrote:

If by "Discussion Tool Events" you mean "New discussion section events", then yes.

Yes, sorry I meant "New discussion section events". Thanks for confirming. Based on that, I think we should go ahead and use the existing init_type field for differentiating between the two event types.

• ppelberg moved this task from Doing to Ready to Be Worked On on the Editing-team (Kanban Board) board.Nov 18 2020, 3:54 PM
• ppelberg updated the task description. (Show Details)Dec 2 2020, 8:02 PM
• ppelberg added a parent task: T269471: Make New Discussion Tool available as Beta Feature at partner wikis.Dec 23 2020, 2:17 AM
gerritbot added a comment.Jan 12 2021, 6:37 AM

Change 655548 had a related patch set uploaded (by DLynch; owner: DLynch):
[mediawiki/extensions/DiscussionTools@master] Give new-section a specific init_type to distinguish it

https://gerrit.wikimedia.org/r/655548

gerritbot added a project: Patch-For-Review.Jan 12 2021, 6:37 AM
DLynch moved this task from Ready to Be Worked On to Code Review on the Editing-team (Kanban Board) board.Jan 12 2021, 6:37 AM
gerritbot added a comment.Jan 12 2021, 10:02 PM

Change 655548 merged by jenkins-bot:
[mediawiki/extensions/DiscussionTools@master] Give new-section a specific init_type to distinguish it

https://gerrit.wikimedia.org/r/655548

Maintenance_bot removed a project: Patch-For-Review.Jan 12 2021, 10:10 PM
ReleaseTaggerBot added a project: MW-1.36-notes (1.36.0-wmf.27; 2021-01-19).Jan 12 2021, 11:00 PM
matmarex moved this task from Code Review to Ready for Sign Off on the Editing-team (Kanban Board) board.Jan 12 2021, 11:38 PM
• ppelberg reassigned this task from DLynch to MNeisler.Jan 13 2021, 5:49 AM
• ppelberg edited projects, added Editing-team (Tracking); removed Editing-team (Kanban Board).

@MNeisler, I'm assigning this over to you for data QA, assuming that this is necessary for this task.

If you do not think data QA is necessary, please assign the task back over to me to close out.

• ppelberg moved this task from Backlog to Analytics on the Editing-team (Tracking) board.Jan 13 2021, 5:49 AM
MNeisler edited projects, added Product-Analytics (Kanban); removed Product-Analytics.Jan 20 2021, 8:53 PM
LGoto triaged this task as Medium priority.Jan 26 2021, 6:06 PM
MNeisler moved this task from Next 2 weeks to Doing on the Product-Analytics (Kanban) board.Jan 27 2021, 12:36 AM
MNeisler reassigned this task from MNeisler to • ppelberg.Jan 27 2021, 11:59 PM
MNeisler moved this task from Doing to [Deprecated] Done (previously: Needs sign-off) on the Product-Analytics (Kanban) board.

I've reviewed the eventlogging data and confirmed we can differentiate between new discussion tool and reply tool events in EditAttemptStep.

Below are some data regarding new discussion tool events logged since the deployment of the change to the init_type field on Jan 12th. These are events that were recorded as event.action = 'init', event.integration = 'discussiontools', event.init_type = 'section'.

  • The first new discussion tool event was recorded on 21 January 2021.
  • Since then, a total of 24 new discussion tool init events have been recorded in EditAttemptStep (7 on cswiki, 1 on cswikinews, and 16 on enwiki)
  • A total of 6 of these new discussion events have been saved (met saveSuccess).

Please see QA notebook for further details regarding the queries and data.

@ppelberg - Reassigning to you for sign-off. Let me know if you have any questions.

Tacsipacsi subscribed.Jan 28 2021, 2:36 PM

I’ve created two new discussions with the tool on Wikidata so far (first, second). Is it normal that they don’t appear in the statistics?

DLynch added a comment.Jan 28 2021, 4:12 PM

@Tacsipacsi Our logging is "sampled", meaning we only log a fraction of the actions people take. It's generally sampled at 1/16 of sessions, so it's very plausible that you've just not been logged there. As you can see, the drawback to this arrangement is that in low-traffic situations we might not get a good representative sample, so it's certainly possible that we might want to increase our sample rate on less busy wikis.

Tacsipacsi added a comment.Jan 28 2021, 10:58 PM

@DLynch Okay, thanks for the explanation!

• ppelberg added a comment.Feb 3 2021, 5:31 PM
In T265099#6782490, @MNeisler wrote:

I've reviewed the eventlogging data and confirmed we can differentiate between new discussion tool and reply tool events in EditAttemptStep.
Please see QA notebook for further details regarding the queries and data.

@ppelberg - Reassigning to you for sign-off. Let me know if you have any questions.

Thank you, @MNeisler – no questions from me at this time related to this QA. Although, see below for a question in response to the point @DLynch raised in T265099#6784140.

In T265099#6784140, @DLynch wrote:

...As you can see, the drawback to this arrangement is that in low-traffic situations we might not get a good representative sample, so it's certainly possible that we might want to increase our sample rate on less busy wikis.

  • @MNeisler: do you think it is necessary to do what David is describing above? I ask this assuming the people start new topics on talk pages significantly less frequently than they do comment/reply in existing ones.
• ppelberg added a comment.Feb 3 2021, 8:19 PM
In T265099#6801031, @ppelberg wrote:
In T265099#6782490, @MNeisler wrote:

I've reviewed the eventlogging data and confirmed we can differentiate between new discussion tool and reply tool events in EditAttemptStep.
Please see QA notebook for further details regarding the queries and data.

@ppelberg - Reassigning to you for sign-off. Let me know if you have any questions.

Thank you, @MNeisler – no questions from me at this time related to this QA. Although, see below for a question in response to the point @DLynch raised in T265099#6784140.

In T265099#6784140, @DLynch wrote:

...As you can see, the drawback to this arrangement is that in low-traffic situations we might not get a good representative sample, so it's certainly possible that we might want to increase our sample rate on less busy wikis.

  • @MNeisler: do you think it is necessary to do what David is describing above? I ask this assuming the people start new topics on talk pages significantly less frequently than they do comment/reply in existing ones.

@MNeisler and I just talked about this. Megan agrees it would be worthwhile to do what @DLynch is describing: to increase the sampling rate from 1 out of 16 events to 100% of events.

Thinking: we implemented 1/5th sampling in T250086 for the Reply Tool and we assume the New Discussion Tool will be used less frequently than the Reply Tool, thus the assumed need for a higher sampling rate.

Notes

  • We should revisit this sampling rate before the New Discussion Tool is made available by default at larger Wikipedias.

Next steps

  • @ppelberg to file a ticket for increasing sampling rate in the New Discussion Tool to 100%
DLynch added a comment.Feb 3 2021, 9:29 PM

Just as a note, we don't currently have a convenient setting for oversampling the logging for just one part of DiscussionTool. It's all-features or nothing, at the moment.

• ppelberg mentioned this in T273946: Adjust Discussion Tools' sampling rates.Feb 5 2021, 3:00 AM
In T265099#6802011, @DLynch wrote:

Just as a note, we don't currently have a convenient setting for oversampling the logging for just one part of DiscussionTool. It's all-features or nothing, at the moment.

Understood, David. I've filed a new ticket (linked below) where we can work out the specifics about how the oversampling should be adjusted.

In T265099#6801728, @ppelberg wrote:
  • @ppelberg to file a ticket for increasing sampling rate in the New Discussion Tool to 100%

See: T273946.

• ppelberg edited projects, added Editing-team (Kanban Board); removed Editing-team (Tracking).Feb 8 2021, 4:36 PM
• ppelberg mentioned this in T274471: Determine whether Discussion Tools event sampling rates need to be adjusted.Feb 11 2021, 2:00 AM
• ppelberg closed this task as Resolved.Feb 13 2021, 3:10 AM
MNeisler mentioned this in T272544: Differentiate between edits to existing sections and the creation of new sections.Feb 24 2021, 8:15 PM
• ppelberg mentioned this in T284153: [SPIKE] Determine feasibility of setting unique sampling rates for the Reply and New Discussion Tools.Jun 2 2021, 2:50 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL