Wikipedia/Wikimedia apps availability test: analyze results
Closed, ResolvedPublic

Description

As explained in T137495, we want to run a test on the wikipedia.org portal page by adding links to the Wikipedia and Wikimedia apps to see what kind of interest it generates.

For the Discovery team, our measurement of success is that the links work and takes the user to the appropriate sites. However, we want to look at event logging to find out if users are clicking through and then we can compare that information to what the Reading/Mobile team collects on downloads of the apps.

Krinkle removed a subscriber: Krinkle.Sep 29 2016, 7:02 PM

Waiting for change to be deployed and for event logging to collect at least a week of data.

None so far, unless if @Jdrewniak or @chelsyx think maybe I'm not filtering the destination field correctly

SELECT
  DATE(timestamp) AS date,
  COUNT(*) AS `clicks on other projects`,
  SUM(IF(INSTR(event_destination, 'itunes.apple.com') > 0, 1, 0)) AS `iOS app link clicks`,
  SUM(IF(INSTR(event_destination, 'play.google.com') > 0, 1, 0)) AS `Android app link clicks`
FROM WikipediaPortal_15890769
WHERE
  LEFT(timestamp, 8) > '20161030'
  AND event_cohort = 'baseline'
  AND event_event_type = 'clickthrough'
  AND event_section_used = 'other projects'
GROUP BY date;
dateclicks on other projectsiOS app link clicksAndroid app link clicks
2016-10-316800
2016-11-017000

I tried clicking the app store icon and watching the network requests my browser sends; it didn't seem to send any to the event logs, which would explain why we don't have any events.

@Jdrewniak Is this something you can investigate and possibly fix in your upcoming deployment?

I think that was literally a matter of no one clicking on these links 😢

I tested this today and it looks like my clicks were logged. @mpopov's query gave me these results:

+------------+--------------------------+---------------------+-------------------------+
| date       | clicks on other projects | iOS app link clicks | Android app link clicks |
+------------+--------------------------+---------------------+-------------------------+
| 2016-10-31 |                       68 |                   0 |                       0 |
| 2016-11-01 |                       74 |                   1 |                       0 |
| 2016-11-02 |                       34 |                   4 |                       3 |
+------------+--------------------------+---------------------+-------------------------+
3 rows in set (1.02 sec)

@Deskana we only log 1 in 200 people on the portal, so your session probably didn't fall into the event-logging group (testing this isn't easy, you have to generate a session id that puts you in the event-logging group... I should probably make that easier)

@Deskana we only log 1 in 200 people on the portal, so your session probably didn't fall into the event-logging group (testing this isn't easy, you have to generate a session id that puts you in the event-logging group... I should probably make that easier)

Ahhh, okay. I think it's fine to leave it the way that it is. :-)

Looking at the portal page, it seems like neither of the app links have the "referrer" or campaign parameter as specified in T137495. These are the parameters that would allow us to precisely track the source of app installs through our respective app stores. Is there a way that we can update these links?

debt added a comment.Nov 2 2016, 5:48 PM

Is there a way that we can update these links?

Absolutely - please let us know what you would like them to be.

JMinor added a comment.Nov 2 2016, 6:00 PM

I think both links from the design ticket are still valid, but to cofirm, here's what we'd like to use for iOS:

https://itunes.apple.com/app/apple-store/id324715238?pt=208305&ct=portal&mt=8

Dbrant added a comment.Nov 2 2016, 6:02 PM

Thanks, @debt. The exact links are specified in the description of T137495. The link for Android should be:

https://play.google.com/store/apps/details?id=org.wikipedia&referrer=campaign_id%3Dportal

debt added a comment.Nov 2 2016, 6:39 PM

Thanks @JMinor and @Dbrant - we'll get those links updated tomorrow!

Change 319654 had a related patch set uploaded (by Jdrewniak):
Bumping portals to master

https://gerrit.wikimedia.org/r/319654

Change 319654 merged by jenkins-bot:
Bumping portals to master

https://gerrit.wikimedia.org/r/319654

Mentioned in SAL (#wikimedia-operations) [2016-11-03T23:10:10Z] <thcipriani@tin> Synchronized portals/prod/wikipedia.org/assets: SWAT: [[gerrit:319654|Bumping portals to master (T146807)]] (duration: 00m 51s)

Mentioned in SAL (#wikimedia-operations) [2016-11-03T23:11:03Z] <thcipriani@tin> Synchronized portals: SWAT: [[gerrit:319654|Bumping portals to master (T146807)]] (duration: 00m 52s)

debt added a comment.Nov 4 2016, 12:20 AM

The app download links have now been updated for campaign tracking (during evening SF swat).

mpopov added a comment.Nov 7 2016, 6:51 PM

Updated counts from event logging:

dateclicks on other projectsiOS app link clicksAndroid app link clicks
2016-11-027666
2016-11-037825
2016-11-049446
2016-11-059314
2016-11-069037
2016-11-077616
debt awarded a token.Nov 8 2016, 8:14 PM

@debt: as requested :)

SELECT
  DATE(timestamp) AS date,
  COUNT(*) AS `clicks on other projects`,
  SUM(IF(INSTR(event_destination, 'itunes.apple.com') > 0, 1, 0)) AS `iOS app link clicks`,
  SUM(IF(INSTR(event_destination, 'play.google.com') > 0, 1, 0)) AS `Android app link clicks`,
  SUM(IF(INSTR(event_destination, '//en.wikipedia.org/wiki/List_of_Wikipedia_mobile_applications') > 0, 1, 0)) AS `Clicks on app list` 
FROM WikipediaPortal_15890769
WHERE
  LEFT(timestamp, 8) > '20161030'
  AND event_cohort = 'baseline'
  AND event_event_type = 'clickthrough'
  AND event_section_used = 'other projects'
GROUP BY date;
dateclicks on other projectsiOS app link clicksAndroid app link clicksClicks on app list
2016-10-3168000
2016-11-0174100
2016-11-0276663
2016-11-0378252
2016-11-0494463
2016-11-0593141
2016-11-0690372
2016-11-0796281
2016-11-0893350
2016-11-0980274
2016-11-1055302
2016-11-1184321
2016-11-1261312
2016-11-1380033
2016-11-1457053
debt added a comment.EditedNov 15 2016, 1:44 AM

Thanks, @mpopov!

Just as a reminder to everyone else, these numbers are from a bucketing algorithm and not true numbers of clicks. :)

Also I checked out pageviews to the list of Wikipedia apps and here's a cool little graph:

Source

SELECT
  TO_DATE(ts) AS date,
  access_method,
  SUM(IF(referer = 'https://www.wikipedia.org/' OR referer = 'http://www.wikipedia.org/', 1, 0)) AS pvs_from_portal,
  SUM(IF(referer = 'https://www.wikipedia.org/' OR referer = 'http://www.wikipedia.org/', 0, 1)) AS pvs_from_elsewhere
FROM webrequest
WHERE
  webrequest_source = 'text'
  AND year = 2016 AND ((month = 10 AND day > 20) OR (month = 11 AND day > 0))
  AND is_pageview
  AND pageview_info['project'] = 'en.wikipedia'
  AND pageview_info['page_title'] = 'List_of_Wikipedia_mobile_applications'
GROUP BY TO_DATE(ts), access_method;

library(tidyverse)

pageviews <- read_csv("~/Desktop/T146807.csv", col_types = "Dcii") %>%
  arrange(date, access_method) %>%
  rename(Date = date,
         `Access Method` = access_method,
         `Wikipedia.org Portal` = pvs_from_portal,
         `Elsewhere or direct` = pvs_from_elsewhere) %>%
  gather(From, Pageviews, -c(Date, `Access Method`))

ggplot(filter(pageviews, `Access Method` != "mobile app"),
              aes(x = Date, y = Pageviews, color = From)) +
  geom_line() +
  geom_vline(xintercept = as.numeric(as.Date("2016-11-01")),
             linetype = "dashed", color = "gray50") +
  geom_point() +
  facet_wrap(~ `Access Method`, nrow = 3) +
  theme_minimal(base_family = "Open Sans") +
  labs(title = "\"List of Wikipedia mobile applications\" pageviews",
       caption = "Dashed line indicates deployment of Wikipedia.org Portal patch to add app links") +
  theme(legend.position = "bottom")
debt added a comment.Nov 17 2016, 8:43 PM

Interesting! :)

debt added a comment.Nov 29 2016, 4:39 PM

Hey @mpopov - can we get another check of these by Friday, Dec 2? That will mark our one month test date, thanks!

mpopov added a comment.Dec 5 2016, 4:19 PM
SELECT
  DATE(timestamp) AS date,
  COUNT(*) AS `clicks on other (sister) projects`,
  SUM(IF(INSTR(event_destination, 'itunes.apple.com') > 0, 1, 0)) AS `iOS app link clicks`,
  SUM(IF(INSTR(event_destination, 'play.google.com') > 0, 1, 0)) AS `Android app link clicks`,
  SUM(IF(event_destination = 'https://en.wikipedia.org/wiki/List_of_Wikipedia_mobile_applications', 1, 0)) AS `Available apps list clicks`
FROM WikipediaPortal_15890769
WHERE
  LEFT(timestamp, 8) > '20161030'
  AND event_cohort = 'baseline'
  AND event_event_type = 'clickthrough'
  AND event_section_used = 'other projects'
GROUP BY date;
dateclicks on other (sister) projectsiOS app link clicksAndroid app link clicksAvailable apps list clicks
2016-10-3168000
2016-11-0174100
2016-11-0276663
2016-11-0378252
2016-11-0494462
2016-11-0593141
2016-11-0690372
2016-11-0796281
2016-11-0893350
2016-11-0980270
2016-11-1055302
2016-11-1184321
2016-11-1261311
2016-11-1380033
2016-11-1486154
2016-11-1545131
2016-11-1681243
2016-11-1778452
2016-11-18730182
2016-11-1982152
2016-11-2069340
2016-11-2170360
2016-11-2257342
2016-11-2382041
2016-11-2456234
2016-11-2574251
2016-11-2654130
2016-11-2776342
2016-11-2853262
2016-11-2954331
2016-11-30120127
2016-12-0165353
2016-12-0281340
2016-12-0354701
2016-12-0473240
2016-12-0554753
Total clicks recorded with EventLogging
clicks on other (sister) projects2659
iOS app link clicks87
Android app link clicks156
Available apps list clicks59
debt added a comment.Dec 7 2016, 5:58 PM

Thanks for the stats, @mpopov ! Is it possible to tell which clickthroughs came from a mobile device or a desktop? I don't need to know which browser or device, just simply if it was a click from a mobile device or a desktop when the user clicked through to one of the 3 new links that were added.

mpopov claimed this task.

@debt: Sure! I'll get those stats pronto.

Update:

clickeddeviceclicksproportion of clicks
Android appdesktop220.13%
Android appmobile1410.87%
clickeddeviceclicksproportion of clicks
iOS appdesktop80.09%
iOS appmobile810.91%
clickeddeviceclicksproportion of clicks
List of appsdesktop110.18%
List of appsmobile510.82%
debt added a comment.Dec 8 2016, 12:41 AM

Thanks! :)

@mpopov can you share the query with us? I'm mostly curious and interested in replicating how you distinguish between desktop and mobile. Also can you confirm the sampling rate? My understanding was 1/200.

mpopov added a comment.EditedDec 15 2016, 12:22 AM

@JKatzWMF Hi Jon! Sure here's the query I used:

SELECT
  DATE(LEFT(timestamp, 8)) AS `date`,
  CASE WHEN (
         INSTR(userAgent, 'Android') > 0
         OR INSTR(userAgent, 'Mobile') > 0
         OR INSTR(userAgent, 'iOS') > 0
         OR INSTR(userAgent, 'Phone') > 0
       ) THEN 'Mobile'
       ELSE 'Desktop' END AS device,
  CASE WHEN INSTR(event_destination, 'itunes.apple.com') > 0 THEN 'iOS App link'
       WHEN INSTR(event_destination, 'play.google.com') > 0 THEN 'Android app link'
       ELSE 'List of apps' END AS clicked,
  COUNT(*) AS clicks
FROM WikipediaPortal_15890769
WHERE
  LEFT(timestamp, 8) >= ${START_DATE} AND LEFT(timestamp, 8) <= ${END_DATE}
  -- ^ or whatever date-subsetting clause you want
  AND event_cohort = 'baseline'
  AND event_event_type = 'clickthrough'
  AND event_section_used = 'other projects'
  AND (
    INSTR(event_destination, 'itunes.apple.com') > 0
    OR INSTR(event_destination, 'play.google.com') > 0
    OR event_destination = 'https://en.wikipedia.org/wiki/List_of_Wikipedia_mobile_applications'
  )
GROUP BY `date`, device, clicked;

The sampling rate is 1/200, correct (https://github.com/wikimedia/portals/blob/50d8a79a2b6b6c226b14e4fe956f085798883a45/dev/wikipedia.org/assets/js/wm-test.js#L7 & https://github.com/wikimedia/portals/blob/50d8a79a2b6b6c226b14e4fe956f085798883a45/dev/wikipedia.org/assets/js/wm-test.js#L77)

Moving to discuss - we'll discuss again in another month with more data.

@debt: November & December totals, as requested:

datedeviceclickedclicks
Nov 2016DesktopAndroid app link17
Nov 2016DesktopiOS App link6
Nov 2016DesktopList of apps10
Nov 2016MobileAndroid app link121
Nov 2016MobileiOS App link59
Nov 2016MobileList of apps42
Dec 2016DesktopAndroid app link15
Dec 2016DesktopiOS App link6
Dec 2016DesktopList of apps10
Dec 2016MobileAndroid app link113
Dec 2016MobileiOS App link63
Dec 2016MobileList of apps53
debt moved this task from To Discuss to Done on the Discovery-Portal-Sprint board.Jan 6 2017, 12:29 AM

Thanks for the updated numbers! I've requested that we start tracking them on the Portal dashboard: T154634.

debt closed this task as Resolved.Jan 6 2017, 12:38 AM