Do we have enough people in SearchSatisfaction to perform an A/B/C/D test and not screw up the dashboards?
Closed, ResolvedPublic1 Story Points


What it sounds like; we need to work out if, in a week, we could collect enough unique users to perform an analysis on a test with 3 test buckets. It may have to wait until we can replicate the logic.

Ironholds created this task.Oct 8 2015, 8:24 PM
Ironholds added a subscriber: Ironholds.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 8 2015, 8:24 PM
mpopov added a subscriber: mpopov.Oct 12 2015, 5:45 PM

What are we measuring?

If you're talking about the test that I think you're talking about, our population of interest is a small subset of the overall population (very few queries actually fit the criteria for even inclusion in some of the tests we do). I recall a low sampling rate for SearchSatisfaction schema, and given that few even fit the test's criteria, it seems we won't have enough people. Which, on the upside, means we shouldn't see problems on the dashboards since the % of people affected will be tiny.

Also, I'm making a task for me to review and implement Bayesian categorical data analysis methods. I should note that the Bayesian approach is still slightly dependent on sample size but less in a "we lack the power to detect effects with too little data, meanwhile too much data yields statistical significance despite tiny observed difference" way and more in a "not enough data makes the results too influenced by the choice of prior distribution(s), so we need to be really careful about our choice of prior(s)" way.

mpopov added a comment.EditedOct 12 2015, 9:41 PM


## Get Data
# library(RMySQL)
# con <- dbConnect(drv = MySQL(), host = "analytics-store.eqiad.wmnet", 
#                  dbname = "log", default.file = "/etc/mysql/conf.d/research-client.cnf")
# satisfaction_users <- wmf::mysql_read("SELECT date, COUNT(*) AS users_per_day
#                                       FROM (
#                                       SELECT
#                                       DATE_FORMAT(timestamp, '%Y-%m-%d') AS date,
#                                       CONCAT(clientIp, userAgent) AS user_id,
#                                       COUNT(*) AS events
#                                       FROM TestSearchSatisfaction2_13223897
#                                       GROUP BY date, user_id) AS events_per_user
#                                       GROUP BY date;", "log", con)
# dbDisconnect(con)
# readr::write_csv(satisfaction_users, "~/SearchSatisfactionDaily.csv")

satisfaction_users <- readr::read_csv("~/Documents/Data/SearchSatisfactionDaily.csv")

ggplot(data = satisfaction_users) +
  geom_line(aes(x = date, y = users_per_day), size = 1.1) +
  ggtitle("Users captured with SearchSatisfaction schema per day") +
  scale_y_continuous(name = "Users", breaks = scales::pretty_breaks(n = 10)) +
Ironholds set Security to None.Oct 13 2015, 8:09 PM
Ironholds edited a custom field.
Ironholds moved this task from Backlog to Done on the Discovery-Analysis (Current work) board.

Yes, we do.

Ironholds assigned this task to mpopov.Oct 13 2015, 8:24 PM
Deskana closed this task as "Resolved".Nov 20 2015, 5:24 AM
Deskana added a subscriber: Deskana.
Restricted Application added a subscriber: StudiesWorld. · View Herald TranscriptNov 20 2015, 5:24 AM