Page MenuHomePhabricator

Dashboard: Search results page - dwell time metric
Closed, ResolvedPublic6 Story Points

Description

We should track the dwell time of users that hit the search results page, now that the sister project snippets has launched. We also should investigate if we have visitor information as well.

Let's break this out in a similar manner that the sister project engagement metrics are being displayed: T164854.

Event Timeline

debt created this task.Jul 12 2017, 7:56 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 12 2017, 7:56 PM
debt updated the task description. (Show Details)Jul 12 2017, 8:00 PM
Aklapper renamed this task from Dahsboard: Search results page - dwell time metric to Dashboard: Search results page - dwell time metric.Jul 12 2017, 9:46 PM

Code for estimating and visualizing dwell-time on full-text search results pages arrived at from autocomplete search: https://github.com/wikimedia-research/Discovery-Search-Adhoc-SRPDwellTime

mpopov claimed this task.Aug 28 2017, 5:01 PM
mpopov set the point value for this task to 6.
mpopov moved this task from Backlog to In progress on the Discovery-Analysis (Current work) board.

Change 374396 had a related patch set uploaded (by Bearloga; owner: Bearloga):
[wikimedia/discovery/golden@master] metrics::search::srp_survtime: Track search results page dwell time

https://gerrit.wikimedia.org/r/374396

Change 374396 merged by Chelsyx:
[wikimedia/discovery/golden@master] metrics::search::srp_survtime: Track search results page dwell time

https://gerrit.wikimedia.org/r/374396

Backfilling data from 2017-04-01 through 2017-08-28. Adding that data to the dashboard should be relatively straightforward.

Change 374876 had a related patch set uploaded (by Bearloga; owner: Bearloga):
[wikimedia/discovery/golden@master] metrics::search::srp_survtime: Split by language

https://gerrit.wikimedia.org/r/374876

Change 374876 merged by Chelsyx:
[wikimedia/discovery/golden@master] metrics::search::srp_survtime: Split by language

https://gerrit.wikimedia.org/r/374876

Change 374920 had a related patch set uploaded (by Bearloga; owner: Bearloga):
[wikimedia/discovery/rainbow@develop] [WIP] SRP visit times

https://gerrit.wikimedia.org/r/374920

Change 374920 merged by Chelsyx:
[wikimedia/discovery/rainbow@develop] SRP visit times

https://gerrit.wikimedia.org/r/374920

Change 375057 had a related patch set uploaded (by Bearloga; owner: Bearloga):
[wikimedia/discovery/rainbow@develop] SRP visit times additional fixes

https://gerrit.wikimedia.org/r/375057

Change 375057 merged by Chelsyx:
[wikimedia/discovery/rainbow@develop] SRP visit times additional fixes

https://gerrit.wikimedia.org/r/375057

mpopov removed a project: Patch-For-Review.EditedAug 31 2017, 7:57 PM

Up on beta: http://discovery-beta.wmflabs.org/metrics/#spr_surv

Notes:

  • Half of the searchers stay on the full-text SRP for 25 or more seconds — this is consistent day-to-day even after sister project search deployment.
  • 3/4 of searchers have left the full-text SRP by the 35s mark. Since the deployment of the sister project search, we appear to have more days when this statistic is at 45s, indicating that users are staying on the page a little longer — possibly due to reading the cross-wiki snippets we're presenting them with.
  • 90% of searchers are done with the full-text SRP by the 1m15s mark, although on some days it's 1m45s. We've observed more days like that before the deployment of sister project search, although there was one day post-deployment in particular (2017-06-29) on which it took 2m45s for 90% of the searchers to leave the page.
debt added a comment.Aug 31 2017, 8:18 PM

We chatted on a few minor UI updates for the dwell page during our afternoon meeting, but otherwise, it looks pretty cool!

Thanks for the notes in T170468#3571284, @mpopov!

I don't remember any other changes we wanted to make and since it's deployed to production (https://discovery.wmflabs.org/metrics/#spr_surv), I'm moving this ticket to Done.

debt closed this task as Resolved.Sep 15 2017, 8:25 PM
debt reopened this task as Open.Oct 11 2017, 8:36 PM
debt moved this task from Done to Backlog on the Discovery-Analysis (Current work) board.

Re-opening this ticket. The numbers all seem to be the same for FR/CA and Other Languages, no matter what settings are selected, which seems really odd.

mpopov added a comment.Nov 8 2017, 8:02 PM

Re-opening this ticket. The numbers all seem to be the same for FR/CA and Other Languages, no matter what settings are selected, which seems really odd.

Perhaps this can help illustrate what is going on:

The estimated survival curves are actually pretty different below the 50% mark, but that's not visible through the summary statistics we record.

if (!grepl("^stat1", Sys.info()["nodename"])) {
  message("Creating an auto-closing SSH tunnel in the background...")
  # See https://gist.github.com/scy/6781836 for more info.
  system("ssh -f -o ExitOnForwardFailure=yes stat1006.eqiad.wmnet -L 3307:analytics-store.eqiad.wmnet:3306 sleep 10")
  library(RMySQL)
  con <- dbConnect(MySQL(), host = "127.0.0.1", group = "client", dbname = "log", port = 3307)
} else {
  con <- wmf::mysql_connect("log")
}

library(glue)
library(magrittr)

yyyymmdd <- format(Sys.Date() - 1, "%Y%m%d")
revision_number <- "16909631"

query <- glue("SELECT
  timestamp AS ts, wiki,
  event_uniqueId AS event_id,
  event_searchSessionId AS session_id,
  event_pageViewId AS page_id,
  event_action AS event,
  event_checkin AS checkin
FROM TestSearchSatisfaction2_{revision_number}
WHERE
  LEFT(timestamp, 8) = '{yyyymmdd}'
  AND wiki RLIKE 'wiki$'
  AND NOT wiki RLIKE '^(arbcom)|(be_x_old)'
  AND NOT wiki IN('commonswiki', 'mediawikiwiki', 'metawiki', 'checkuserwiki', 'donatewiki', 'collabwiki', 'foundationwiki', 'incubatorwiki', 'legalteamwiki', 'officewiki', 'outreachwiki', 'sourceswiki', 'specieswiki', 'stewardwiki', 'wikidatawiki', 'wikimania2017wiki', 'movementroleswiki', 'internalwiki', 'otrs_wikiwiki', 'projectcomwiki', 'ombudsmenwiki', 'votewiki', 'chapcomwiki', 'nostalgiawiki', 'otrs_wikiwiki')
  AND event_source = 'autocomplete'
  AND event_subTest IS NULL
  AND event_articleId IS NULL
  AND event_action IN('visitPage', 'checkin');")

# Fetch data from MySQL database:
results <- wmf::mysql_read(query, "log", con)
wmf::mysql_close(con)

results %<>%
  dplyr::mutate(ts = lubridate::ymd_hms(ts)) %>%
  dplyr::arrange(session_id, event_id, ts) %>%
  dplyr::distinct(session_id, event_id, .keep_all = TRUE) %>%
  dplyr::arrange(wiki, session_id, page_id, desc(event), ts) %>%
  dplyr::select(wiki, ts, session_id, page_id, event, checkin) %>%
  dplyr::group_by(session_id, page_id) %>%
  dplyr::filter(event == "visitPage" | (event == "checkin" & checkin == max(checkin, na.rm = TRUE))) %>%
  dplyr::ungroup() %>%
  dplyr::mutate(language = dplyr::case_when(
    wiki == "enwiki" ~ "English",
    wiki %in% c("cawiki", "frwiki") ~ "French and Catalan",
    TRUE ~ "Other languages"
  )) %>%
  data.table::data.table(key = c("wiki", "language", "session_id", "page_id"))

checkins <- c(0, 10, 20, 30, 40, 50, 60, 90, 120, 150, 180, 210, 240, 300, 360, 420)
# ^ this will be used for figuring out the interval bounds for each check-in
# Treat each individual search session as its own thing, rather than belonging
#   to a set of other search sessions by the same user.
page_visits <- as.data.frame(results[, {
  if (any(.SD$event == "checkin")) {
    last_checkin <- max(.SD$checkin, na.rm = TRUE)
    idx <- which(checkins > last_checkin)
    if (length(idx) == 0) idx <- 16 # length(checkins) = 16
    next_checkin <- checkins[min(idx)]
    status <- ifelse(last_checkin == 420, 0, 3)
    data.table::data.table(
      `last check-in` = as.integer(last_checkin),
      `next check-in` = as.integer(next_checkin),
      status = as.integer(status)
    )
  }
}, by = c("wiki", "language", "session_id", "page_id")])

surv <- survival::Surv(
  time = page_visits$`last check-in`,
  time2 = page_visits$`next check-in`,
  event = page_visits$status,
  type = "interval"
)
fit <- survival::survfit(surv ~ page_visits$language)
p <- survminer::ggsurvplot(fit, data = page_visits, risk.table = FALSE)
p <- p + ggplot2::labs(
  x = "Time (seconds)",
  title = glue("Time spent on Wikipedia Search Result Pages (SRPs) on {Sys.Date() - 1}, by language"),
  subtitle = "Dashed lines indicate which survival probabilities we track"
) + ggplot2::scale_y_continuous(labels = scales::percent_format()) +
  ggplot2::scale_color_brewer(palette = "Set1") +
  ggplot2::geom_hline(yintercept = c(0.10, 0.25, 0.50, 0.75, 0.90, 0.95, 0.99), linetype = "dashed")
print(p)

@debt: Does the comment above (T170468#3745776) satisfy your concern?

debt closed this task as Resolved.Nov 15 2017, 8:03 PM
debt moved this task from Backlog to Done on the Discovery-Analysis (Current work) board.

Yup, all good, thanks!