x is to be defined by @Tbayer ?

Hasn't been estimated yet.

In T204609#4591518, @Jdlrobson wrote:

x is to be defined by @Tbayer ?

x had been set to 20% (i.e. 10:10:80) a couple of weeks ago in T200792, but @ovasileva and I think 100% couldn't hurt for this initial smaller wiki. I'll update T200792 for consistency.

Sounds good. Ready for estimation!

Note to SWATer: When doing this, could you also do T203589 ?

Change 461420 had a related patch set uploaded (by Pmiazga; owner: Pmiazga):
[operations/mediawiki-config@master] Enable Page issus A/B test for Latvian wiki

https://gerrit.wikimedia.org/r/461420

Swatting at 4pm PST

Change 461420 merged by jenkins-bot:
[operations/mediawiki-config@master] Enable Page issuses A/B test for Latvian wiki

https://gerrit.wikimedia.org/r/461420

Mentioned in SAL (#wikimedia-operations) [2018-09-19T23:14:57Z] <catrope@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Enable Page issues A/B test on lvwiki (T204609) (duration: 00m 58s)

OK, the pageissues table just materialized in Hadoop with the data from the first hour - 19 events, 14 of which seem to be your test views. Let's wait a bit for the caches...

We have over half a day's worth of data in the table now, including from daytime hours in Latvia, and
the caches should have caught up. But the event rate remains surprisingly low (see Grafana and query below) - about 1-2 events per minute, whereas lv.m.wikipedia.org receives 70-80k views/day currently (https://stats.wikimedia.org/v2/#/lv.wikipedia.org/reading/total-page-views/normal|bar|1-Month|access~mobile-web ) or around 40-60 views/minute. Maybe content quality is very high on this wiki... (Will do further checks.)

SELECT year, month, day, hour, COUNT(*) AS events FROM event.pageissues WHERE year >0 GROUP BY year, month, day, hour ORDER BY year, month, day, hour LIMIT 10000;

year	month	day	hour	events
2018	9	19	23	19
2018	9	20	0	8
2018	9	20	1	8
2018	9	20	2	11
2018	9	20	3	8
2018	9	20	4	37
2018	9	20	5	50
2018	9	20	6	91
2018	9	20	7	67
2018	9	20	8	66
2018	9	20	9	70
2018	9	20	10	68
2018	9	20	11	67

OK, let's look at the article https://lv.m.wikipedia.org/wiki/Filozofija , which currently is tagged as lacking references and receives between 20-80 pageviews per day.

However, it hasn't generated any PageIssues events so far:

SELECT COUNT(*) FROM event.pageissues 
WHERE year >0 
AND event.pageTitle = 'Filozofija';

_c0
0
1 row selected (58.149 seconds)

On the other hand, here is a list of the 100 pages that have generated the most events so far.

Interestingly, the top entry (on Pepin the Short) doesn't contain any visible page issues - rather, it seems there is an ambox class coming from this hidden infobox template: https://lv.wikipedia.org/wiki/Veidne:Infokaste%2B

Data via

SELECT '|',
CONCAT('[[https://lv.m.wikipedia.org/wiki/',event.pageTitle,'|',event.pageTitle,']]') AS page, '|',
COUNT(*) AS events
FROM event.pageissues 
WHERE year >0 
GROUP BY event.pageTitle
ORDER BY events DESC LIMIT 100;

Not sure if this is relevant or not, but ~half the time I load the page I do see the page issue appear, with the treatment that is currently on production. Whereas other times it doesn't present itself at all:

whereas with other pages on the list, e.g. Ropažu sporta centrs, it alternates between the new and current version as expected:

In T204609#4602883, @alexhollender wrote:

Not sure if this is relevant or not, but ~half the time I load the page I do see the page issue appear, with the treatment that is currently on production. Whereas other times it doesn't present itself at all:

whereas with other pages on the list, e.g. Ropažu sporta centrs, it alternates between the new and current version as expected:

Correct. It's an A/B test so 50% of the time you should be bucketed into the old treatment and 50% into the new treatment.

@Jdlrobson compare the first set of screenshots with the second here: T204609#4602883 — for the first set, 50% of the time I'm getting the old treatment, 50% of the time I'm not getting any page issue showing up at all...

Interestingly, the top entry (on Pepin the Short) doesn't contain any visible page issues - rather, it seems there is an ambox class coming from this hidden infobox template: https://lv.wikipedia.org/wiki/Veidne:Infokaste%2B

Yes, this is interesting.. in the old treatment, we create a link unconditionally as the entry point to issues. In the new treatment, because the ambox is hidden (but in the page) it's impossible to click it so there is no way to view it. However we'll log events in both cases as the ambox is present.

We could reveal it in the new treatment by using an !important rule, but this would go against the wishes of editors so I'm not sure a good idea.

.issues-group-B .ambox { display: block !important; }

Not clear what to do in this situation...

In T204609#4601837, @Tbayer wrote:

OK, let's look at the article https://lv.m.wikipedia.org/wiki/Filozofija , which currently is tagged as lacking references and receives between 20-80 pageviews per day.

However, it hasn't generated any PageIssues events so far:

SELECT COUNT(*) FROM event.pageissues 
WHERE year >0 
AND event.pageTitle = 'Filozofija';

_c0
0
1 row selected (58.149 seconds)

Do we know what user agents those views are? Just want to rule out this being a problem with grade C browsers... (e.g. browsers we don't run JS)

In T204609#4603143, @Jdlrobson wrote:

Interestingly, the top entry (on Pepin the Short) doesn't contain any visible page issues - rather, it seems there is an ambox class coming from this hidden infobox template: https://lv.wikipedia.org/wiki/Veidne:Infokaste%2B

Yes, this is interesting.. in the old treatment, we create a link unconditionally as the entry point to issues. In the new treatment, because the ambox is hidden (but in the page) it's impossible to click it so there is no way to view it. However we'll log events in both cases as the ambox is present.

We could reveal it in the new treatment by using an !important rule, but this would go against the wishes of editors so I'm not sure a good idea.

.issues-group-B .ambox { display: block !important; }

Not clear what to do in this situation...

The template itself is hidden. I read this as editors don't want to show it. It's also hidden on desktop: https://lv.wikipedia.org/w/index.php?title=Pipins_%C4%AAsais&mobileaction=toggle_view_desktop

In T204609#4603202, @Jdlrobson wrote:

In T204609#4601837, @Tbayer wrote:

OK, let's look at the article https://lv.m.wikipedia.org/wiki/Filozofija , which currently is tagged as lacking references and receives between 20-80 pageviews per day.

However, it hasn't generated any PageIssues events so far:

SELECT COUNT(*) FROM event.pageissues 
WHERE year >0 
AND event.pageTitle = 'Filozofija';

_c0
0
1 row selected (58.149 seconds)

Do we know what user agents those views are? Just want to rule out this being a problem with grade C browsers... (e.g. browsers we don't run JS)

Here's a list. BTW, some events have since come in for that page,[1] but there were at least 24 pageviews (mobile web, non-bot) that should have generated events and didn't:

Data via:

SELECT user_agent_map['browser_family'] AS browser, SUM(view_count) AS views
FROM wmf.pageview_hourly
WHERE year = 2018 AND month = 9 AND day = 20 AND hour <= 15
AND page_title = 'Filozofija'
AND project = 'lv.wikipedia'
AND access_method = 'mobile web'
AND agent_type = 'user'
GROUP BY user_agent_map['browser_family']
ORDER BY views DESC LIMIT 100;

SELECT year, month, day, hour, COUNT(*) FROM event.pageissues 
WHERE year > 0 
AND event.pageTitle = 'Filozofija'
GROUP BY year, month, day, hour
ORDER BY year, month, day, hour LIMIT 10000;

year	month	day	hour	_c4
2018	9	20	16	1
2018	9	20	17	2
2018	9	20	18	2
3 rows selected (60.163 seconds)

Here is the distribution of actions so far. This does not look impossible a priori (although it would mean quite a low issue clickthrough ratio of <=2% in both control and test). So the missing events are not caused by an entire category of actions missing.

Data via

SELECT event.action AS action, COUNT(*) AS events
FROM event.pageissues 
WHERE year >0 
GROUP BY event.action
ORDER BY events DESC LIMIT 10000;

Interesting... can we rule out the following?

• doNotTrack header was present
• events triggered errors due to uri length and were not processed
• events havent made it to hive yet

I'll think about other reasons in meantime...

And the distribution of values of the sectionNumbers and issuesSeverity fields looks plausible too - at least there are a lot of different kinds of combinations represented.

Data via

SELECT COUNT(*) AS events, event.sectionNumbers AS sectionNumbers
FROM event.pageissues 
WHERE year >0 
GROUP BY event.sectionNumbers
ORDER BY events DESC LIMIT 100;

SELECT COUNT(*) AS events, event.issuesSeverity AS issuesSeverity
FROM event.pageissues 
WHERE year >0 
GROUP BY event.issuesSeverity
ORDER BY events DESC LIMIT 100;

No red flags in issuesVersion, isAnon, and namespaceId either.

SELECT event.issuesVersion AS issuesVersion, COUNT(*) AS events
FROM event.pageissues 
WHERE year >0 
GROUP BY event.issuesVersion;

issuesversion	events
new2018	791
old	591
2 rows selected (20.221 seconds)


SELECT event.isAnon AS isAnon, COUNT(*) AS events
FROM event.pageissues 
WHERE year >0 
GROUP BY event.isAnon;

isanon	events
false	27
true	1355
2 rows selected (20.227 seconds)


SELECT event.namespaceId AS namespaceId, COUNT(*) AS events
FROM event.pageissues 
WHERE year >0 
GROUP BY event.namespaceId;

namespaceid	events
0	1382
1 row selected (34.209 seconds)

Could you provide some example pages which have DEFAULT priority? It would be useful to verify they are behaving as expected as they were difficult to test...

In T204609#4604676, @Jdlrobson wrote:

Could you provide some example pages which have DEFAULT priority? It would be useful to verify they are behaving as expected as they were difficult to test...

Here is a list for all events where the first element of issuesSeverity was 'DEFAULT' (also split out by action):

Data via

SELECT '|',
CONCAT('[[https://lv.m.wikipedia.org/wiki/',event.pageTitle,'|',event.pageTitle,']]') AS page, '|',
event.action AS action, '|',COUNT(*) AS events, '|'
FROM event.pageissues 
WHERE year >0 
AND event.issuesSeverity[0] = 'DEFAULT'
GROUP BY event.pagetitle, event.action 
ORDER BY page, action LIMIT 10000;

In T204609#4604645, @Jdlrobson wrote:

Interesting... can we rule out the following?

• doNotTrack header was present

Yes, that can be ruled out. Compare the PageIssues event rate from T204609#4601701 (or [2] below) with e.g. the print button event rate of the Print schema (lvwiki, Minerva, sampled at 10%).[1]

• events triggered errors due to uri length and were not processed

Can be ruled out. That was a very rare occurrence even in T196904 (where the event query string contained a page title / URL twice, and we only have one page title field here). Besides, it wouldn't explain the inconsistent logging for the same page in the https://lv.m.wikipedia.org/wiki/Filozofija example.

• events havent made it to hive yet

Super unlikely. (Other schemas, e.g. Print [1], don't seem to be seeing such a delay. And repeating the query from T204609#4601701 >13h later doesn't show any retroactive increases in the events logged.[2])

I'll think about other reasons in meantime...

SELECT year, month, day, hour, 10 * COUNT(*) AS print_buttons_shown 
FROM event.print 
WHERE year = 2018 AND month = 9 AND day = 20 
AND wiki ='lvwiki'
AND event.skin = 'minerva'
AND event.action = 'shownPrintButton'
GROUP BY year, month, day, hour 
ORDER BY year, month, day, hour LIMIT 10000;

year	month	day	hour	print_buttons_shown
2018	9	20	0	240
2018	9	20	1	20
2018	9	20	2	110
2018	9	20	3	240
2018	9	20	4	690
2018	9	20	5	1770
2018	9	20	6	2790
2018	9	20	7	2180
2018	9	20	8	2270
2018	9	20	9	2120
2018	9	20	10	2970
2018	9	20	11	2040
2018	9	20	12	1720
2018	9	20	13	1470
2018	9	20	14	1610
2018	9	20	15	2510
2018	9	20	16	2860
2018	9	20	17	3010
2018	9	20	18	3050
2018	9	20	19	2040
2018	9	20	20	1380
2018	9	20	21	760
2018	9	20	22	130
2018	9	20	23	110

SELECT year, month, day, hour, COUNT(*) AS events FROM event.pageissues WHERE year >0 GROUP BY year, month, day, hour ORDER BY year, month, day, hour LIMIT 10000

year	month	day	hour	events
2018	9	19	23	19
2018	9	20	0	8
2018	9	20	1	8
2018	9	20	2	11
2018	9	20	3	8
2018	9	20	4	37
2018	9	20	5	50
2018	9	20	6	91
2018	9	20	7	67
2018	9	20	8	66
2018	9	20	9	70
2018	9	20	10	68
2018	9	20	11	67
2018	9	20	12	55
2018	9	20	13	43
2018	9	20	14	72
2018	9	20	15	72
2018	9	20	16	77
2018	9	20	17	119
2018	9	20	18	119
2018	9	20	19	114
2018	9	20	20	64
2018	9	20	21	37
2018	9	20	22	18
2018	9	20	23	15
2018	9	21	0	7
2018	9	21	1	4
27 rows selected (46.251 seconds)

I'll think about other reasons in meantime...

@ovasileva asked if caching might be the cause. I think it's unlikely, but it is theoretically possible that the devices cache more aggressively than I realised. The rate of events does look to be increasing today (according to grafana there are more today than yesterday), so let's keep an eye on this in case we need to consider this.

Here is a look at the ratio of pageloaded events from the PageIssues to all applicable views on lvwiki (more precisely, mobile web (-domain) pageviews to mainspace pages, excluding spider views).

For an (imperfect) comparison, per T201123#4607488 around 10% of articles on lvwiki have Ambox issues.

I would say that a ratio around 1-2% like observed for much of yesterday (Sep 20) is implausibly low, even if one assumes that pages with issues are less popular. It does seem to have increased today though, so perhaps we need to check our assumptions about caching.

The next step might be to limit this to a known set of pages with issues (like Filozofija above), for a more direct comparison.

Data via

SELECT CONCAT(views_list.year,'-',
  LPAD(views_list.month,2,'0'),'-',
  LPAD(views_list.day,2,'0'),' ',
  LPAD(views_list.hour,2,'0'),'h') AS datehour, 
pageloaded_count_list.pageloaded_events AS pageloaded_events, 
views_list.views AS all_pageviews,
ROUND(pageloaded_count_list.pageloaded_events/views_list.views,4) AS ratio 
FROM (
  SELECT year, month, day, hour, COUNT(*) AS pageloaded_events 
  FROM event.pageissues 
  WHERE year = 2018 AND month = 9 AND day >= 20 
  AND event.action = 'pageLoaded'
  GROUP BY year, month, day, hour) AS pageloaded_count_list
JOIN (
  SELECT year, month, day, hour, SUM(view_count) AS views
  FROM wmf.pageview_hourly
  WHERE year = 2018 AND month = 9 AND day >= 20
  AND project = 'lv.wikipedia'
  AND access_method = 'mobile web'
  AND agent_type = 'user'
  AND namespace_id = 0
  GROUP BY year, month, day, hour) AS views_list
ON 
  pageloaded_count_list.hour = views_list.hour AND 
  pageloaded_count_list.day = views_list.day AND 
  pageloaded_count_list.month = views_list.month AND 
  pageloaded_count_list.year = views_list.year 
ORDER BY datehour LIMIT 10000;

For the record: @Jdlrobson has found the likely reason for the initially low event rate ("Minerva A/B tests are not subject to HTML caching time. Config added inside SkinMinerva is subject to the rules of HTML caching and can take several days ..."). The fix is being worked on at T205355: A/B config flag should be subject to ResourceLoader caching rules not HTML caching rules

Indeed, the rates since the weekend looks more plausible (if still below the 10%):

Data via

SELECT CONCAT(views_list.year,'-',
  LPAD(views_list.month,2,'0'),'-',
  LPAD(views_list.day,2,'0'),' ') AS date, 
pagetoken_count_list.pagetoken_counts  AS pagetokens, 
views_list.views AS all_pageviews,
ROUND(pagetoken_count_list.pagetoken_counts/views_list.views,4) AS ratio 
FROM (
  SELECT year, month, day, COUNT(DISTINCT event.pagetoken) AS pagetoken_counts 
  FROM event.pageissues 
  WHERE year = 2018 AND month = 9 AND day >= 20 
  GROUP BY year, month, day) AS pagetoken_count_list
JOIN (
  SELECT year, month, day, SUM(view_count) AS views
  FROM wmf.pageview_hourly
  WHERE year = 2018 AND month = 9 AND day >= 20
  AND project = 'lv.wikipedia'
  AND access_method = 'mobile web'
  AND agent_type = 'user'
  AND namespace_id = 0
  GROUP BY year, month, day) AS views_list
ON 
  pagetoken_count_list.day = views_list.day AND 
  pagetoken_count_list.month = views_list.month AND 
  pagetoken_count_list.year = views_list.year 
ORDER BY date LIMIT 10000;

Re-checking the ratio of pageloaded events per pageview after the fix for T205355 has been deployed:
This looks much more plausible now than earlier (T204609#4607546), with rates around the estimated 10%.

In T204609#4604633, @Tbayer wrote:

Here is the distribution of actions so far. This does not look impossible a priori (although it would mean quite a low issue clickthrough ratio of <=2% in both control and test). [...]

After some other checks that looked fine (will post the detailed results here), I happened to look at the frequency of action types again.[1]

It turns out that the issue clickthrough rate is now even lower than in the above initial check - even below the rate of (generic) edit button clicks, which is quite surprising. Concretely, e.g. 0.40% in test and 0.17% in control for top (page-level) issues.[2] (Perhaps one reason why the ratio was higher than normal in T204609#4604633 was that that initial check only covered a relatively small number of events where our own testing with deliberate issue-clicking was still impacting the clickthrough ratio.)

This rate is similarly low across browsers[3] (i.e. it's not a bug where one major browser fails to send issueClicked events). And on the other hand we hand-tested earlier in QA that issueClicked events were being sent correctly for several browser/OS combinations. So it seems that this is real and that people may in fact tapping on the issues notice even less often than on the generic edit button, even though we might have expected otherwise based on the received wisdom that consumption actions (reading) are generally orders of magnitude more frequent than contribution actions (editing).

I.e. the above observation should not hold up the launch of the full A/B test planned for today (T200792), although it may affect some of the sample sizes needed (in the earlier sample size estimate at T200792#4489268 I had assumed that editClicked would be less frequent that issueClicked).

SELECT event.action AS action, COUNT(*) AS events
FROM event.pageissues 
WHERE year = 2018 AND month = 9 AND day >= 27
GROUP BY event.action
ORDER BY events DESC LIMIT 10000;

action	events
pageLoaded	25710
editClicked	109
issueClicked	75
modalClose	18
modalEditClicked	4
modalInternalClicked	2
6 rows selected (696.26 seconds)

SELECT event.issuesVersion AS version, 
SUM(IF(event.action = 'issueClicked', 1, 0)) / SUM(IF(event.action = 'pageLoaded', 1, 0)) AS issuesclickratio,
SUM(IF(event.action = 'pageLoaded', 1, 0)) AS pageloaded_events
FROM event.pageissues 
WHERE year = 2018 AND month = 9 AND day >= 27
AND event.sectionnumbers[0] = 0
GROUP BY event.issuesVersion;

version	issuesclickratio	pageloaded_events
new2018	0.003957528957528957	10360
old	0.001713796058269066	10503
2 rows selected (24.093 seconds)

SELECT useragent.browser_family AS browser, 
SUM(IF(event.action = 'issueClicked', 1, 0)) / SUM(IF(event.action = 'pageLoaded', 1, 0)) AS issuesclickratio,
SUM(IF(event.action = 'pageLoaded', 1, 0)) AS pageloaded_events
FROM event.pageissues 
WHERE year = 2018 AND month = 9 AND day >= 27
GROUP BY useragent.browser_family
ORDER BY pageloaded_events DESC LIMIT 50;

browser	issuesclickratio	pageloaded_events
Chrome Mobile	0.0025455796353629173	14535
Mobile Safari	0.002388263391334016	5862
Samsung Internet	0.00358719646799117	3624
Chrome	0.008893280632411068	1012
Android	0.0	116
Chrome Mobile iOS	0.010416666666666666	96
BingPreview	0.0	91
Firefox Mobile	0.0	90
Chrome Mobile WebView	0.0	67
Facebook	0.0	59
Opera Mobile	0.0	51
[...]
24 rows selected (44.168 seconds)

@Tbayer is going to run one more data check and then this will then be resolved.