At first glance, nothing majorly unusual in the graphs but we should check again in a week to check whether cached pages could be masking a problem.

Agreed.

Did clicks to hamburger via JS go up significantly in the past week according to MobileUIWebClickTracking.

The number of hamburger clicks has gone down significantly since Sunday, Jan 31st. Yesterday (Tuesday, 2nd February) shows a little over 30% of the clicks than Sunday. That being said, this appears to follow a general trend in the data and isn't any more alarming than the other relative decreases.

Is this an artefact of how how we're collecting this data? I'm not sure what I meant by this question.

Are you guys looking at the page-ui-daily graph at http://mobile-reportcard.wmflabs.org/? Or somewhere else?

This data is a mess and I don't think we can rely on it.

It seems the fix to T108723 would have dropped clicks to hamburger from 50% to 10%. It got merged December 28th so in theory went out on the 14th January and would mess with these results (but then we had lots of deployment issues recently).

Looking at main-menu-daily graph on http://mobile-reportcard.wmflabs.org/

• Clicks to the main menu are sampled by the defaulting Schema sample rate of 0.5
• 17th-24th there seems to have been an outage to logging to this table.
• 28th-30th January clicks to the home link dropped from 50k to 22k - halved without explanation.

Looking at page-ui-daily graph on http://mobile-reportcard.wmflabs.org/

• On January 10th 124,183 clicks to hamburger
• On January 18th 35,732 clicks to hamburger. Seems to consistent(ish) with the sampling drop.
• On Feb 3rd 36,110 clicks to hamburger so it hasn't decreased and it hasn't increased.

My analysis using this unreliable data is that moving to top has had no negative impact on clicks to the hamburger.
Given the downwards trend on main-menu-daily it's possible we actually broke stuff in the process.
My feeling is that something else led to the drop in the clicks to the hamburger as reverting the code to the top doesn't seem to have done much.

@bmansurov to look at Special:MobileMenu PVs.

This is the command I'm running in hue:

SELECT sum(view_count) as view_count
FROM wmf.pageview_hourly 
WHERE
  CONCAT(year, "-", LPAD(month, 2, "0"), "-", LPAD(day, 2, "0")) BETWEEN "2016-01-20" AND "2016-02-03"
  AND agent_type = "user"
  AND access_method = "mobile web"
  AND page_title = "Special:MobileMenu"
GROUP BY CONCAT(year, "-", LPAD(month, 2, "0"), "-", LPAD(day, 2, "0"));

Here is some data:

The pageview API returned the following data (which is surprisingly different from the above). I think we should consider the pageview API results more seriously than the above SQL result:

Quick look at the data tells us that the pageviews of the Special:MobileMenu haven't changed much. We hoped they would go down with the mobile menu patch. I have a theory that since the mobile menu is being loaded via ResourceLoader, there is still a noticeable delay which is preventing users from seeing the menu drawer. I think we should consider reviving https://gerrit.wikimedia.org/r/#/c/260746/ now.

Thanks for running this data @bmansurov.
On the plus side we've not made anything worse.

With async JavaScript the fact is we're never going to be able to load the menu JavaScript code earlier. We might want to hide the menu icon for JavaScript users until the code has finished loading but this could be a user experience degradation for users with JavaScript as it might lead to them not clicking on the menu at all and at worse slowing them down from accessing something in it.

My opinion is by pushing the HTML size down we'll get JavaScript loading quicker and see less traffic to the Special:MobileMenu page (we should keep watching that)

I suggest we send a mail out (mainly for the benefit of Jon Katz who was asking for this change) and get some input from @Nirzar around what we want to do here.

Any other thoughts?

In T124675#1999931, @bmansurov wrote:

The pageview API returned the following data (which is surprisingly different from the above).

That may be because the linked API result is restricted to the English Wikipedia, whereas the above query covered all projects.

I think we should consider the pageview API results more seriously than the above SQL result:

Why?

In T124675#2000611, @Tbayer wrote:

In T124675#1999931, @bmansurov wrote:

The pageview API returned the following data (which is surprisingly different from the above).

That may be because the linked API result is restricted to the English Wikipedia, whereas the above query covered all projects.

Interesting. I thought only enwiki had a page named 'Special:MobileMenu'.

I think we should consider the pageview API results more seriously than the above SQL result:

Why?

The API is more tested. The query may have a flaw in it maybe?

In other news, an email about this task has been sent to reading-wmf.

In T124675#2000654, @bmansurov wrote:

In T124675#2000611, @Tbayer wrote:

In T124675#1999931, @bmansurov wrote:

The pageview API returned the following data (which is surprisingly different from the above).

That may be because the linked API result is restricted to the English Wikipedia, whereas the above query covered all projects.

Interesting. I thought only enwiki had a page named 'Special:MobileMenu'.

No, such special pages usually exist under the same (English, original) name on all projects, although they may redirect to translated names (e.g. https://de.m.wikipedia.org/wiki/Special:MobileMenu --> https://de.m.wikipedia.org/wiki/Spezial:Mobiles_Men%C3%BC ).

I think we should consider the pageview API results more seriously than the above SQL result:

Why?

The API is more tested. The query may have a flaw in it maybe?

OK, good point, but on the other hand Hive is the data source here, and your query above (in the last edited version) seems solid to me. I checked by restricting it to enwiki (and rewriting the GROUP BY just a little bit, Hive/HQL is a bit peculiar about enforcing the use of partition fields there). Fortunately the resulting numbers are indeed exactly the same as in the API result you posted above - the universe is in balance again ;)

SELECT year, month, day, CONCAT(year, "-", LPAD(month, 2, "0"), "-", LPAD(day, 2, "0")) AS date, sum(view_count) as view_count
FROM wmf.pageview_hourly 
WHERE
  CONCAT(year, "-", LPAD(month, 2, "0"), "-", LPAD(day, 2, "0")) BETWEEN "2016-01-20" AND "2016-02-03"
  AND agent_type = "user"
  AND access_method = "mobile web"
  AND project = 'en.wikipedia'
  AND page_title = "Special:MobileMenu"
GROUP BY year, month, day ORDER BY year, month, day LIMIT 10000;

Result (dummy columns removed):

Thanks, @Tbayer. (btw, you may want to remove the double post above)

In T124675#2000756, @bmansurov wrote:

Thanks, @Tbayer. (btw, you may want to remove the double post above)

Done, not sure what happened there (looks a bit like T955, although I don't recall hitting submit twice; I've recently seem the same issue here).

With async JavaScript the fact is we're never going to be able to load the menu JavaScript code earlier. We might want to hide the menu icon for JavaScript users until the code has finished loading but this could be a user experience degradation for users with JavaScript as it might lead to them not clicking on the menu at all and at worse slowing them down from accessing something in it.

I think that showing a user a button that they expect will open a menu but, in fact, might cause their UA to navigate away from the page that they're on is a worse user experience than hiding the button until it will function as expected /cc @Nirzar.

I think that showing a user a button that they expect will open a menu but, in fact, might cause their UA to navigate away from the page that they're on is a worse user experience than hiding the button until it will function as expected.

We could also inline JavaScript. We shouldn't force users to load the whole page just to navigate away from it.

We could also inline JavaScript. We shouldn't force users to load the whole page just to navigate away from it.

I agree in general. However, in this case, the user would be navigating to the Special:MobileMenu page.

So that we're not comparing apples to oranges... our change went out on the 15th Oct 2015 to English Wikipedia to move the menu to the bottom.

@Tbayer 's analysis was based on data collected by event logging and showed a decrease in clicks to the hamburger menu. If this was true I'd expect to see an increase in visits to the Special:MobileMenu page as well....

So... I ran some [[
https://wikimedia.org/api/rest_v1/metrics/pageviews/per-article/en.wikipedia/mobile-web/user/Special:MobileMenu/daily/20151012/20151013

If I look at before and after the async JavaScript changes (5th August 2015) there is a much more obvious spike as the changes went into effect.

One way to look at this is maybe an increase in click throughs to the menu is a good thing as it could mean slower connections are now interacting with our site.

It's also worth noting that due to cache, we'd need to wait 30 days before being able to draw a complete conclusion here about whether it's helped (so far we only have 6 days).

"think that showing a user a button that they expect will open a menu but, in fact, might cause their UA to navigate away from the page that they're on is a worse user experience than hiding the button until it will function as expected"

Not sure I can fully agree. How do we know? If a user wants to click random maybe this allows them to do so quicker than if we blocked the action until JS has loaded. Surely getting them to their destination is more important and I don't think we can safely say that (on a 2G connection that menu may become completely inaccessible to them if we hide the link).

My personal feeling is we should chew on this problem some more, focus on improving performance of the site so time to first interactive gets so low this is not a problem. My preference would be to revisit this known problem later in the quarter.

Not sure I can fully agree. How do we know? If a user wants to click random maybe this allows them to do so quicker than if we blocked the action until JS has loaded.

To repeat your question: "How do we know?". You're assuming that the cost of navigating to the Special:MobileMenu page is trivial compared to loading a JS asset. I'm not sure agree.

I do agree that we should revisit this at the end of Sprint I.

In T124675#2003148, @Jdlrobson wrote:

So that we're not comparing apples to oranges... our change went out on the 15th Oct 2015 to English Wikipedia to move the menu to the bottom.

@Tbayer 's analysis was based on data collected by event logging and showed a decrease in clicks to the hamburger menu. If this was true I'd expect to see an increase in visits to the Special:MobileMenu page as well....

Just to clarify the history and give credit where it's due, my linked analysis was focused on finding out whether that decrease in October was concentrated on particular wikis or user agents (answer: no and no). The fact that a decrease had happened had already been determined at that point, e.g. by yourself based on the "main-menu-daily" graph at http://mobile-reportcard.wmflabs.org/ .

Looking at that graph again now, I'm noticing that around Nov 10 (shortly after we had examined it in that ticket), it bounced back to almost the levels before the mid-October drop. Do we know the reason for that?

Change 276106 had a related patch set uploaded (by Giuseppe Lavagetto):
jobqueue_redis: remove temporarily rdb1003 for reimaging

https://gerrit.wikimedia.org/r/276106

Change 276106 merged by Giuseppe Lavagetto:
jobqueue_redis: remove temporarily rdb1003 for reimaging

https://gerrit.wikimedia.org/r/276106

Removing sprint tag for time being.