A/B Test (Egg): Check that all is well with the addition of descriptive text to sister project links
Closed, ResolvedPublic1 Story Points

Description

This ticket will be for the Discovery-Analysis team to verify that the test is progressing as we want it to be, once it has been pushed into production.

Please refer to the epic for full details: https://phabricator.wikimedia.org/T131238

debt created this task.Apr 12 2016, 10:02 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 12 2016, 10:02 PM
debt renamed this task from Check that all is well with the addition of descriptive text to sister project links to A/B Test (Egg): Check that all is well with the addition of descriptive text to sister project links.Apr 26 2016, 4:52 PM
debt triaged this task as Normal priority.
debt edited projects, added Discovery-Analysis; removed Discovery-Portal-Backlog.
debt assigned this task to mpopov.May 3 2016, 3:58 PM
Deskana added a subscriber: Deskana.May 3 2016, 8:06 PM

A/B test is live according to T131238#2260900, so this one is ready to go.

mpopov set the point value for this task to 1.
mpopov added a comment.May 5 2016, 5:51 PM
USE log;
SELECT
  cohort, accept_lang_has_english,
  COUNT(1) AS sessions,
  SUM(events) AS total_events
FROM (
  SELECT
    event_session_id AS session_id, event_cohort AS cohort,
    CASE WHEN INSTR(event_accept_language, 'en') THEN 'yes'
         ELSE 'no' END AS accept_lang_has_english,
    COUNT(1) AS events
  FROM WikipediaPortal_14377354
  WHERE LEFT(timestamp, 8) = '20160504'
    AND event_cohort IN('descriptive-text-a', 'descriptive-text-b')
  GROUP BY session_id, cohort, accept_lang_has_english
) AS sessions
GROUP BY cohort, accept_lang_has_english;
cohortaccept_lang_has_englishsessionstotal_events
descriptive-text-ayes13422670
descriptive-text-byes13262658

But if we look at number of sessions with recorded 'other projects' clicks…

SELECT
  cohort, accept_lang_has_english,
  COUNT(1) AS sessions,
  SUM(events) AS total_events
FROM (
  SELECT
    event_session_id AS session_id, event_cohort AS cohort,
    CASE WHEN INSTR(event_accept_language, 'en') THEN 'yes'
         ELSE 'no' END AS accept_lang_has_english,
    COUNT(1) AS events
  FROM WikipediaPortal_14377354
  WHERE LEFT(timestamp, 8) = '20160504'
    AND event_cohort IN('descriptive-text-a', 'descriptive-text-b')
    AND event_section_used = 'other projects'
  GROUP BY session_id, cohort, accept_lang_has_english
) AS sessions
GROUP BY cohort, accept_lang_has_english;
cohortaccept_lang_has_englishsessionstotal_events
descriptive-text-ayes44
descriptive-text-b--00

I'm guessing that since the user has to scroll down quite a bit to even see the change (if they're even enrolled in the A/B test and have been put into the test bucket), we're probably not going to see a lot of clicks from that section (if any!). Hopefully there will be enough to see some impact, and, like, it's still too early to say, but maybe this is something we should have done after collapsing languages.

debt added a comment.May 6 2016, 12:30 PM

Thanks, @mpopov - we knew that this region isn't high in clickthroughs. Let's see what the rest of the test week brings us. :)

debt closed this task as Resolved.May 10 2016, 4:57 PM

Closing this ticket - check was good.