Page MenuHomePhabricator

Define initial coverage % for the fawiki and cawiki test safety survey
Closed, ResolvedPublic

Description

User Story

T296486: Draft configuration of test survey before the deploy mentions:

AudienceCoverage
Logged-in editors with at least 5 edits~100 people/day initial coverage

Before we configure the survey, we need to translate this requirement into a % number (0 to 1).

Technical information

The initial coverage % can be calculated based on the number of logged-in editors (with at least 5 edits if possible) per day, so that we get around ~100 bucketed sessions per day.

Eg:

  • Xwiki has 10000 editor unique sessions per month
  • We can extrapolate Xwiki has 10000/30=333 sessions per day
  • If 333 = 1, 100 = x -> 100 / 333 = 0.3 coverage (30%)

Acceptance Criteria

  • The initial coverage rate for cawiki is: 0.2
  • The initial coverage rate for fawiki is: 0.1

Event Timeline

By the 2020 data of monthly editors (inclusive of editors with less than 5 edits) using this math:

  • fawiki: 6,347/30 = 212 daily editors (~0.5 or %50) (@KCVelaga actually looked at weekly averages for the top wikis from 2018 to 2020 and estimates fa.wiki as closer to 1764 weekly editors or 252 daily)
  • cawiki: 1,457/30 = 49 daily editors (~2.0 or 200% ...)

Let's all discuss options when we meet this week if there are issues with doing higher percent coverage?

100 people/day initial coverage was a figure that we eyeballed in a conversation, but the important point is to affect a small % of editors in the initial deploy. If everything goes well, a few days later we can up the coverages once there is confidence everything is working as expected.

To explain the perspective from the engineering side, we need to be very cautious with the initial deploy. If there are any serious issues we don't want all editors to have a broken experience for days.

We've had a discussion with Tanja, Sandister, Krishna, Amin and myself, and we have agreed on the following values:

  • cawiki: 20%, at most 1/5
  • fawiki: 10%, at most 1/10

For the initial deploy, so that in case of bugs the affected editor % is not so big.

Jhernandez claimed this task.
Jhernandez updated the task description. (Show Details)