[SPIKE] Determine whether Reply Tool traffic warrants lowering sampling rate through wider deployment.
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	ppelberg
	Jun 2 2021, 3:03 PM

Description

This task is about determining whether the number of events being emitted by the Reply Tool are high enough to warrant us lowering the sampling rate –currently set at 100% [i] – as it's scaled to be an opt-out feature at all Wikipedias (T262331).

We will couple what we learn in this task with what we learn in T284153 to determine how, and if, the DiscussionTools' sampling rate(s) need to be adjusted (T274471).

Question(s) this spike should answer

Do we anticipate the growth in Reply Tool traffic (read: the number of events it's emitting) to be high enough to warrant us lowering the sampling rate (currently set at 100% [i]) as it's scaled to be an opt-out feature at all Wikipedias (T262331)?

Done

Answers to all ===Open questions are documented in the task description
Followup tickets are created [if necessary]

i. Reference needed

Related Objects
Search...

Status	Assigned	Task
Open	None	T276497 Scale DiscussionTools to all projects
Open	None	T251207 [Epic] Scale DiscussionTools to all Wikipedias
Resolved	None	T262331 [Release ticket] Make Reply Tool available as opt-out preference at all Wikipedias
Resolved	None	T269062 Make the the Reply Tool available as an opt-out preference at all Phase 0, 1, 2 and 3 Wikipedias
Resolved	ppelberg	T274471 Determine whether Discussion Tools event sampling rates need to be adjusted
Resolved	ppelberg	T284154 [SPIKE] Determine whether Reply Tool traffic warrants lowering sampling rate through wider deployment.

Event Timeline

ppelberg created this task.Jun 2 2021, 3:03 PM

ppelberg moved this task from Backlog to Analytics on the Editing-team (Tracking) board.Jun 2 2021, 9:39 PM

@ppelberg

Based on a review of daily traffic for the reply tool, I do not anticipate the increase in reply tool traffic to be high enough to warrant lowering the sampling rate. I recommend we can keep the current 100% reply tool sampling rate for wider deployment.

Reasoning: I reviewed the daily average number of reply tool init and page edit init events coming from wikis where the reply tool is currently deployed as opt-out. Even with oversampling, reply tool edits represent only about 4 to 6% of all init events logged on EditAttemptStep. I estimated a proportional increase in daily reply tool events based on the size (read: number of editors) of larger wikis and continued oversampling through wider deployment should not cause any significant increases in traffic logged on editattemptstep.

MNeisler reassigned this task from MNeisler to ppelberg.Jun 8 2021, 8:15 PM

MNeisler triaged this task as Medium priority.

MNeisler added a project: Product-Analytics (Kanban).

MNeisler moved this task from Next 2 weeks to Done on the Product-Analytics (Kanban) board.

MNeisler subscribed.

ppelberg edited projects, added Editing-team (Kanban Board); removed Product-Analytics (Kanban), Editing-team (Tracking).Jun 8 2021, 8:58 PM

ppelberg added a project: Product-Analytics (Kanban).

ppelberg moved this task from Inbox to Ready for Sign Off on the Editing-team (Kanban Board) board.

In T284154#7143878, @MNeisler wrote:

@ppelberg

Based on a review of daily traffic for the reply tool, I do not anticipate the increase in reply tool traffic to be high enough to warrant lowering the sampling rate. I recommend we can keep the current 100% reply tool sampling rate for wider deployment.

Understood and sounds good to me.

As such, I'm going to:

Resolve this task
De-prioritize T284153 considering we are going to continue sampling Reply and New Discussion Tool events at 100% until new information surfaces that causes us to reevaluate this approach.

ppelberg mentioned this in T284153: [SPIKE] Determine feasibility of setting unique sampling rates for the Reply and New Discussion Tools.Jun 10 2021, 11:56 PM

ppelberg mentioned this in T274471: Determine whether Discussion Tools event sampling rates need to be adjusted.Jun 24 2021, 7:02 PM

[SPIKE] Determine whether Reply Tool traffic warrants lowering sampling rate through wider deployment.Closed, ResolvedPublicActions