As a product owner, I want lazy loaded images running in production on a mobile web Wikipedia so that I can observe how the enhancement that saves bandwidth impacts usage in a coarse grained fashion.
| Subject | Repo | Branch | Lines +/- | |
|---|---|---|---|---|
| Enable lazy loaded images on Ukranian and Farsi Wikipedias | operations/mediawiki-config | master | +9 -0 |
Querying the most recent table
select count(*) from NavigationTiming_15485142 where wiki = 'bewiki' and timestamp > 20160403000000
yields 353 entries for the past month, so this is definitely a small sample.
@ori increasing the sampling rate for one wiki seems like a bad thing to do as it would have impacts on the global first paint / fully load times which are supposed to reflect all our traffic. How would you recommend going about this?
@Jdlrobson, good thinking to raise this!
@ori et al:
The task title currently suggests bnwiki, not bewiki, although both of those are on the lower end of the spectrum in terms of number of events. The top traffic sites like enwiki dwarf smaller ones like bnwiki, so my guess is amplification of the number of events would largely be washed out. But it's a good point that artificial amplification in a given wiki (likely) introduces skew synthetically on the macro number (i.e., by virtue of changing what gets counted), as opposed to naturally (i.e., as a result of an actual content loading change).
I think we should consider fawiki or ukwiki either in addition to bnwiki or as an alternative. These get relatively larger numbers of events, and although the natural change of different loading strategies is bound to impact the global trend (again, probably more nominally given the top sites account for most events), at least it's a natural consequence.
For a frame of reference, here's data looking at hits for one week. NULL means desktop, stable means mobile web stable, beta means mobile web beta.
select wiki, event_mobileMode, count(*) ct, avg(event_loadEventEnd) load_time from
NavigationTiming_15485142 where timestamp > '20160427' and timestamp < '20160504'
and wiki in ('enwiki', 'fawiki', 'bnwiki', 'ukwiki') group by wiki, event_mobileMode order by load_time asc;
wiki event_mobileMode ct load_time
enwiki NULL 725548 2940.3989
ukwiki NULL 4336 3558.6836
fawiki beta 7 4186.0000
enwiki stable 564376 4834.4134
enwiki beta 201 5382.7438
bnwiki beta 4 6006.3333
fawiki NULL 6824 6591.3143
bnwiki NULL 165 7399.9538
fawiki stable 9779 9505.1877
ukwiki stable 2437 10225.1451
bnwiki stable 295 11228.5442Here's some additional data on bnwiki, fawiki, and ukwiki from Hive looking at one day of pageviews.
Destop v Mobile
select project, access_method, sum(view_count)
from projectview_hourly
where year = 2016 and month = 4 and day = 28
and project in ('bn.wikipedia', 'fa.wikipedia', 'uk.wikipedia')
and agent_type = 'user'
group by project, access_method;
project access_method _c2
bn.wikipedia desktop 57539
bn.wikipedia mobile app 2402
bn.wikipedia mobile web 113066
fa.wikipedia desktop 1049256
fa.wikipedia mobile app 32395
fa.wikipedia mobile web 1594857
uk.wikipedia desktop 1046392
uk.wikipedia mobile app 9386
uk.wikipedia mobile web 662231
Top 3 country level access per wiki for mobile web traffic specifically. When there's a high concentration of users for a given wiki in a particular region, that helps with insights about relative pageview changes and relative speed changes.
select project, country_code, sum(view_count)
from projectview_hourly
where year = 2016 and month = 4 and day = 28
and project in ('bn.wikipedia', 'fa.wikipedia', 'uk.wikipedia')
and agent_type = 'user'
and access_method = 'mobile web'
group by project, country_code;
bn.wikipedia BD 55823
bn.wikipedia US 20892
bn.wikipedia IE 16449
fa.wikipedia IR 1418817
fa.wikipedia US 56514
fa.wikipedia DE 15765
uk.wikipedia UA 620712
uk.wikipedia NL 9387
uk.wikipedia US 6930
Top 3 destination wikis based on top source country above, based on a quick scan. This aids in distinguishing if changes are material based on destination wiki. Granted, there are likely to be some socioeconomic conditions aligned with target langwiki for any given source country, but it's the relative change that's most interesting when comparing what happened in two destination wikis for the same source country of access.
select country_code, project, sum(view_count)
from projectview_hourly
where year = 2016 and month = 4 and day = 28
and country_code in ('BD', 'IR', 'UA')
and agent_type = 'user'
and access_method = 'mobile web'
group by country_code, project;
BD en.wikipedia 411855
BD bn.wikipedia 55823
BD commons.wikimedia 8083
IR fa.wikipedia 1418817
IR en.wikipedia 268671
IR fa.wikisource 6988
UA ru.wikipedia 1079823
UA uk.wikipedia 620712
UA en.wikipedia 104695Speaking to specific analysis for this work, we can of course always update our queries to include/exclude particular wikis from the resultant output to isolate effects.
For our future selves, here's a query by country for bnwiki, fawiki, and ukwiki with primary country sourcing vs US sourcing. Future queries would need to screen out impacts of non-caching via the available fields in NavigationSchema. The following are just averages; more sophisticated analysis would involve study of the median/quantiles/distribution curves (cf. T125414: Investigate how connection speed varies by country)
One week
select wiki, event_originCountry, count(*) ct, avg(event_loadEventEnd) load_time from
NavigationTiming_15485142 where timestamp > '20160427' and timestamp < '20160504'
and event_mobileMode = 'stable' and wiki in ('fawiki', 'bnwiki', 'ukwiki')
and event_originCountry in ('BD', 'US', 'IE', 'IR', 'DE', 'UA', 'NL') group by wiki, event_originCountry order by wiki, event_originCountry, load_time;
wiki event_originCountry ct load_time
bnwiki BD 167 10419.8519
bnwiki US 21 3807.6000
fawiki IR 8768 9680.8648
fawiki US 317 6846.1467
ukwiki UA 2252 10578.8736
ukwiki US 31 10791.6667
Approximately April 12 - May 5 since latest functioning schema rev
select wiki, event_originCountry, count(*) ct, avg(event_loadEventEnd) load_time from
NavigationTiming_15485142 where timestamp < '20160505'
and event_mobileMode = 'stable' and wiki in ('fawiki', 'bnwiki', 'ukwiki')
and event_originCountry in ('BD', 'US', 'IE', 'IR', 'DE', 'UA', 'NL') group by wiki, event_originCountry order by wiki, event_originCountry, load_time
bnwiki BD 427 10603.6950
bnwiki US 58 5786.5556
fawiki IR 24735 9487.8627
fawiki US 935 7815.1058
ukwiki UA 8163 10609.2396
ukwiki US 84 10913.0615I think it would be useful to deploy both lazy loaded references AND images to https://tl.wikipedia.org/
This wiki is about the same size in readership as Bengali wikipedia, in a 2G prevalent area and will give us interesting data about how combining the data changes things. @dr0ptp4kt
do you agree?
I think the benefit of rolling out to fa and uk wikis is the larger audience, thus more confidence that our changes work and larger data sets for navigation timing metrics but I don't envision much benefit aside from those points.
@dr0ptp4kt @Jhernandez it is my suggestion we update this task to launch lazy loading images on all medium wikis with definitions coming from:
I would also like to see us roll out both lazy loading images and references for all small wikis (with exception of Bengali (bnwiki) so as to avoid interfering with the fact we've already enabled lazy loaded images there.
I'm keen for us to get feedback on scale and to get a clearer idea what impact both together have.
To avoid a crazy config we'd enable on 'wikipedia' and then list all the larger wikis as a blacklist.
Any objections?
Another query for our future selves. Although we'll want to analyze image bytes per in-scope pageview (my initial look a month or so ago suggested this was possible in Hive), for the purpose of analyzing loading time, the following tries to confine the analysis so that we can look at the quantiles for several measured fields on like data sets. In this case, this was looking at bnwiki, where lazy loading images on mobile web has been enabled for a while.
The short of it is that, after attempting to factor out lag (roughLoadTimeInitialLagExcluded), the new image loading strategy appeared to be slightly faster at the median when looking at two distinct two week periods where there were clearly no overlapping implementations in the events and the time ranges were roughly for about the same part of the month calendar.
Pre: 20160419-20160502
Post: 20160517-20160530
This seemed to be the case on HTTP/2 and non-HTTP/2, which had different characteristics in the speed distribution curve (more analysis needed).
It should be noted the data on this smaller wiki was fairly sparse - there were only several hundred rows that qualified for analysis. The medium sized wikis will have a greater volume of pageviews and should aid in data analysis. It should be noted that holidays may have some macro pull on the figures, but we'll see.
select left(timestamp,8) as day, webHost, event_originCountry, event_lazyLoadImages, event_firstImage, event_isHttp2, event_mediaWikiVersion, event_requestStart, event_responseStart, event_responseEnd, event_firstPaint, event_domInteractive, event_domComplete, event_loadEventStart, event_loadEventEnd, event_responseEnd-event_responseStart as roughNetworkTimeInitialLagExcluded, event_loadEventEnd-event_responseStart as roughLoadTimeInitialLagExcluded from NavigationTiming_15485142 where timestamp > '20160413' and timestamp < '20160602' and event_action = 'view' and event_isAnon = true and event_mobileMode = 'stable' and event_namespaceId = 0 and event_redirectCount is null and event_responseStart is not null and event_loadEventEnd is not null and wiki = 'bnwiki';
@dr0ptp4kt you have made some changes but not documented the why which makes it confusing for me to follow the narrative here. Why are we no longer shipping to tl.m.wikipedia.org ?
Change 294247 had a related patch set uploaded (by Jdlrobson):
Enable lazy loaded images on Ukranian and Farsi Wikipedias
Thanks for asking, @Jdlrobson. Sprint 74 ran out of points. But this task is at the top of the queue (#2, although I'll drag it to #1). tl.m.wikipedia.org depends on the reference patch, so I decoupled it from this task, which doesn't have the dependency.
Note the idea was to turn on lazy loaded references and images together so I have made an update to this card (email to be sent out).
Change 294247 merged by jenkins-bot:
Enable lazy loaded images on Ukranian and Farsi Wikipedias