Page MenuHomePhabricator

QA for EventLogging patch
Closed, ResolvedPublic

Description

This task is about making sure the patch [1] that's intended to address the issue discovered in T232237#5501940 is working as we expect it to.

Done

  • 1. Contributors are being assigned to test buckets correctly
  • 2. We are logging the data we will need to answer our research questions: T221195
  • 3. We are logging the data we will need to answer our follow up questions: T232175#5545364

  1. T232237#5548323

Event Timeline

ppelberg reassigned this task from ppelberg to Mayakp.wiki.Oct 9 2019, 4:23 PM
ppelberg removed a subscriber: Mayakp.wiki.
ppelberg added a comment.EditedOct 9 2019, 7:40 PM

@Mayakp.wiki, this patch (see below) should be ready for QA as soon as tomorrow.

If there are additional details that would be helpful for you to have, please let me know.

cc @MNeisler


Change 539933 merged by jenkins-bot:
[mediawiki/extensions/MobileFrontend@master] Load editor EventLogging code earlier
https://gerrit.wikimedia.org/r/539933

Mayakp.wiki triaged this task as Medium priority.Oct 10 2019, 10:53 PM
Mayakp.wiki moved this task from Triage to Next Up on the Product-Analytics board.

Checked the distribution using Neil/Megan's initial code, for data starting from 10/10 (date when fix was applied). The buckets seem to be balancing now. The data is being collected 50-50 in both buckets.

bucketattemptsattempts%usersusers %
default-source176,26349.96%133,09250.17%
default-visual176,54050.04%132,21249.83%
Mayakp.wiki moved this task from Next Up to Doing on the Product-Analytics board.Oct 15 2019, 12:14 AM
ppelberg added a comment.EditedOct 15 2019, 11:47 PM

Checked the distribution using Neil/Megan's initial code, for data starting from 10/10 (date when fix was applied). The buckets seem to be balancing now. The data is being collected 50-50 in both buckets.

bucketattemptsattempts%usersusers %
default-source176,26349.96%133,09250.17%
default-visual176,54050.04%132,21249.83%

Excellent – thank you, @Mayakp.wiki. A couple resulting questions...

  • 2. Do we now have confidence to say We are logging the data we will need to answer our research questions: T221195?
  • 3. Do we now have confidence to say We are logging the data we will need to answer our follow up questions: T232175#5545364?

...the questions above are borrowed from the task description.

Megan has recently shared the list of fields that will be used to answer the research questions (T221195 and T232175#5545364 ) with me. I will be checking the data in those fields this week to answer Q2 and Q3.

Megan has recently shared the list of fields that will be used to answer the research questions (T221195 and T232175#5545364 ) with me. I will be checking the data in those fields this week to answer Q2 and Q3.

Wonderful – thank you, Maya.

ppelberg added subscribers: DLynch, matmarex.

In light of T235949, we'll need to come up with a new way to bucket contributors in the A/B test (T236337). We'll then also need to test to make sure this new bucketing method is working properly (T236338).

@DLynch, a question for you: as part of T236338, do you think it will be necessary to re-test questions "2." and "3." below?

...@MNeisler and I assume a change to how contributors are bucketed would not affect the instrumentation needed to answer these questions, tho we thought you would know best.

  • 2. Do we now have confidence to say We are logging the data we will need to answer our research questions: T221195?
  • 3. Do we now have confidence to say We are logging the data we will need to answer our follow up questions: T232175#5545364?

cc @matmarex

I can answer as well – I don't think it will be necessary to re-check those. We should only re-check that the buckets are balanced 50-50.

I can answer as well – I don't think it will be necessary to re-check those. We should only re-check that the buckets are balanced 50-50.

Thank you, Bartosz.

@Mayakp.wiki, in light of the above, we can continue testing questions "2." and "3." below; they are not blocked by T236337 and T236338 being completed.

  • 2. Do we now have confidence to say We are logging the data we will need to answer our research questions: T221195?
  • 3. Do we now have confidence to say We are logging the data we will need to answer our follow up questions: T232175#5545364?

cc @MNeisler

Thanks @ppelberg and @matmarex for confirmation regarding Q2 and Q3 !
i'll retest Q1 later once the dependent tickets are complete.

Thanks @ppelberg and @matmarex for confirmation regarding Q2 and Q3 !
i'll retest Q1 later once the dependent tickets are complete.

This sounds great – thank you, Maya.

QA for Q2 and Q3 in progress. Completed checks for few of the metrics.
There are a few observations which I will discuss with Megan next week. Update on checks are documented here.

Mayakp.wiki closed this task as Resolved.EditedWed, Dec 4, 9:02 PM

Completed QA for Q2 and Q3. Everything looks good except for when I checked for Wiki field, I found 1 init event recorded on 10-01-2019 for commonswiki which is not part of the experiment as mentioned in the link. This is definitely Not at all an issue since it is just one event but whatever caused this event to be bucketed could potentially cause many other events from this wiki to be bucketed in the experiment maybe in the future and wanted to bring it to the team's attention - @ppelberg @DLynch @MNeisler
Summary of the checks performed posted in this QA Document.
Will post notebook soon.