@Peter in the previous task you asked

Just curious, did you get the UA in the example when you tested @Jdlrobson ?

When I checked before it was possible see that it was mini through the user agent: https://wikitech.wikimedia.org/wiki/ProxyBrowsers#User_Agent_2
or maybe it's still different depending on os.

The official docs: http://www.ucweb.com/download/UCBrowser_User_Agent_en.pdf

Yeh I saw that but for whatever reason on my Android Nexus 5 with UC Browser Mini Smooth 10.7.6 with speed mode on it's reporting itself as simply:
"Mozilla/5.0 (X11; U; Linux i686; zh-CN; rv: 1.2.3.4) Gecko/"

With speed mode off it's much more helpful:
Mozilla/5.0 (Linux; U; Android 6.0.1; en-US; Nexus_5 Build/MMB29S) AppleWebKit/528.5+ (KHTML, like Gecko) Version/3.1.2 Mobile Safari/525.20.1 UCBrowser/10.7.6.805 Mobile

That pdf reports "In Speed Mode, UC uses JUC UA to obtain basic version pages. It saves traffic and it’s faster. In non-speed mode, UC uses Mozilla 5.0 UA to obtain advanced pages, pages display richer."

Change 314195 had a related patch set uploaded (by Jdlrobson):
UC Mini should be blacklisted

https://gerrit.wikimedia.org/r/314195

If only there was a way to feature detect data savings mode from Opera Mini and UC Mini...

It looks like a request header "X_UCBROWSER_DEVICE_UA" is being sent.
Is there anyway we can detect this in ResourceLoader?

Ah I see, the user agent when you check inside the browser is as you explained @Jdlrobson . Server side you can see if it's extreme mode.

I installed UC Mini Smooth on my phone and got different UAs. See https://requestb.in/1htn8v71?inspect.

Yep that matches what I got in https://wikitech.wikimedia.org/wiki/ProxyBrowsers#User_Agent_2 so server side we can see if UC is using extreme mode, checking in Javascript fails (and that is good because we should avoid that path).

@ori yup this is consistent with what I'm seeing - different UAs on the client when in extreme mode.

How can we handle this in ResourceLoader? The startup module works entirely with feature detection and client user agents. Introducing server side checks would cause problems with how we cache, unless we can vary it for different devices...

In T147369#2691138, @Jdlrobson wrote:

It looks like a request header "X_UCBROWSER_DEVICE_UA" is being sent.
Is there anyway we can detect this in ResourceLoader?

Does the UA make any RL requests, e.g. for the startup module?

While it looks generic at first glance, the extreme mode UA "Mozilla/5.0 (X11; U; Linux i686; zh-CN; rv: 1.2.3.4) Gecko/" is actually very consistent and recognizable.

It's missing the last part of the UA normally found for Firefox, with the date and browser name, like: "Gecko/20101026 Firefox/3.6.12"

Therefore I don't think there's any risk of it matching a linux user with their OS set to Chinese using an old Firefox.

In fact I can't find any evidence that 1.2.3.4 was ever a real Gecko version. It looks made up, and probably a telltale sign of UCMini's proxy/crawler/cache.

Now of course if we go down the rabbit hole of UA matching we're always at risk of them changing it. Feature detection would be best. If someone can consistently generate hits with their server-side agent, we might be able to figure out a more foolproof way to detect it than UA parsing?

Reviews welcomed if we want to go down the UA rabbit hole:
https://gerrit.wikimedia.org/r/#/c/314195/

Note I tried to see what JS the browser supported using
https://html5test.com
but that didn't work
http://featurejs.com suggests that localStorage and canvas are not supported by UCMini..
I'm on a bus now but if remember correctly we feature detect localStorage by just checking the window. It's definitely worth attempting to upgrade this test a la modernizer to see if that solves the problem:
https://github.com/viljamis/feature.js/blob/master/feature.js#L122
This seems the safest bet to me.

That localStorage test is a bit naive. QUOTA_EXCEEDED_ERR is thrown when localStorage is full, for example. This would have to be well tested, we don't want some users to start having a nojs experience because their localStorage is full or disabled. Maybe canvas testing is a safer bet, although there might be similar caveats with people disabling canvas on their browser (not sure if that's a thing).

I'm not sure its naive - it's just not intended to be used in the way we do things with ResourceLoader. I've seen the same feature detection test in many other places. Yes if localStorage is full it will not work but then if that happened localStorage is not useable.

https://developer.mozilla.org/en-US/docs/Web/API/Web_Storage_API/Using_the_Web_Storage_API

Do we care about support for IE8? Canvas is not available there (I forget if support is deprecated or not):
http://caniuse.com/#feat=canvas

@phuedx @bmansurov @jhobs @Jhernandez @pmiazga @Gilles @Krinkle @ori @Peter would it help for us to get together and hash out the details sometimes next week? We're keen to get this fixed within next 2 weeks and I feel like we could reach a decision in 30 mins. If so who would want to be present in such a discussion?

I think we could spend some time investigating how other sites do. Would be interesting to check how sites in India do it, for them this should be a common thing.

re: @Jdlrobson
Sounds reasonable to me. Why not just throw out a meeting, make everyone optional, and give everyone permission to reschedule it? Should hopefully be able to find a time that works for people who are eager to attend.

@Jdlrobson - Any updates, did you end up scheduling a meeting?

We just met to discuss this. Here are the next steps.

• @Krinkle to look at logs to see patterns for UC Mini browser user agent
• We'll then merge the above patch which uses regexs with some amendments to fix this on the short term
• I will open a task to ensure that we investigate in more depth whether UC Mini has an issue in the startup module as on the long term we may want to enable JS for this browser.
• We also feel it's going to be useful having a contact in UC going forward. @Gilles has tried contacting them before in English, without reply, but I'm going to contact them in Chinese (since they are Chinese based) to see if we have more luck.

perhaps we can move out of the sprint and swap for https://phabricator.wikimedia.org/T147314 or https://phabricator.wikimedia.org/T140485 ?

I personally would wait until after refactoring before working on any more Hovercards tasks.

Searching for a trailing Gecko/ user agent in the raw requests logs yielded no results (searched the last 7 days worth of requests).

I ran two searches (on stat1002 using kafkacat, topic webrequest_text, field user_agent).

1. Ends with "Gecko/" (grep 'Gecko/"') – 0 matches.
2. Contains "Gecko/" followed by non-number (grep 'Gecko/[^0-9]' - Firefox uses Gecko/YYYYMMDD) – 12 matches. Unrelated to UC (two samples below).

2016-10-28, GET it.wikipedia.org, user_agent:
"Mozilla/5.0 (Windows NT 6.1; rv:12.0) Gecko/ 20120405 Firefox/14.0.1"

2016-10-29, GET en.wikipedia.org, user_agent:
"Mozilla/5.0 (Windows NT 6.1; WOW64; rv:22.0) Gecko/ /22.0"

This is good news, as it means there are no other browsers we've seen that use this kind of pattern. The bad news is that UC Mini Speedmode itself didn't show up in these searches, either. Most likely because they use a different user-agent in their HTTP requests, compared to the navigator.userAgent interface in JavaScript. The latter of which is not mentioned in their official documentation.

I tried UC Browser for iOS, but while it does have a "Speedmode" opt-in feature, it didn't alter the user-agent (neither server-side or client-side). There isn't a "UC Mini" for iOS it seems, but I was able to install UC Mini from the Play Store on an Android emulator. It has Speed Mode enabled by default and features the following two user-agents (test.script):

HTTP User-Agent:
"UCWEB/2.0 (MIDP-2.0; U; Adr 4.2.2; en-US; ..) .. UCBrowser/10.7.9.856 U2/1.0.0 Mobile"

JS navigator.userAgent:
"Mozilla/5.0 (X11; U; Linux i686; zh-CN; rv:1.2.3.4) Gecko/"

Given all this, the regex currently used at https://gerrit.wikimedia.org/r/#/c/314195/ seems fine to me. There's nothing else out there we saw in a weeks' worth of requests that matches this regex (not even a weird bot or script somewhere).

See also:

• http://stackoverflow.com/a/40433933/319266

This patch has +1s from myself and Baha. @phuedx or @jhobs please can one of you commit to testing and +2ing?

We should probably update https://www.mediawiki.org/wiki/Compatibility/Software_for_using_MediaWiki with information about UC Mini and its two modes once this is fixed. UC Mini would be grade A in normal mode and grade B in extreme mode, right?

Updated: https://www.mediawiki.org/w/index.php?title=Compatibility%2FSoftware_for_using_MediaWiki&type=revision&diff=2279783&oldid=1678222

Change 314195 merged by jenkins-bot:
UC Mini should be blacklisted

https://gerrit.wikimedia.org/r/314195

We serve the right version but the search is still broken or at least the user experience? I think T136803 should be-opened or if we wanna create a new issue. When I try with UCBrowser v10 in SpeedMode the search box is a link, clicking on the link I get to the search box at the end of the page and there we have a super small search icon (that is clickable!) that would not pass any usability tests :)

In T147369#2778890, @Gilles wrote:

We should probably update https://www.mediawiki.org/wiki/Compatibility/Software_for_using_MediaWiki with information about UC Mini and its two modes once this is fixed. UC Mini would be grade A in normal mode and grade B in extreme mode, right?

That page is outdated and the historical version, please update https://www.mediawiki.org/wiki/Compatibility#Browsers instead.

@Peter is it happening on an Android phone?

@Peter I'm seeing this too. Interestingly search works on the beta cluster, wikimedia.commons.org and mediawiki.org which makes me wonder if they might be doing some Wikipedia specific hacks that are out of our control?

mediawiki.org:

wikipedia.org:

@bmansurov sorry yes Android it is.

@Jdlrobson aha I see, hmm. Was you able to reach out to them?

I think we still get the Grade A experience for wikipedia.org for UC mini speed mode? No search button, no line under the first h2. ping @Krinkle

However faking the UA using "User Agent switcher" in Chrome and set the agent to "Mozilla/5.0 (X11; U; Linux i686; zh-CN; rv: 1.2.3.4) Gecko/" it works for me locally (I get the grade C).

Ok finally got rid off the cache (some pages was the old version some the new). I get the same look and feel as you @Jdlrobson and isn't it the grade A? No search button, no line after the first headline?

@Peter it is my theory that UC Mini is mangling the HTML for wikipedia domains given that other domains (e.g. mediawiki) on the same browser show more grade C like experiences https://phabricator.wikimedia.org/T147369#2805433 . Given it is a popular site this seems likely.

It's thus my belief that we can't do anything here accept hope UC one day reply to us!

I suggested to @Peter in our team meeting to check the actual contents of the article when a stale one is encountered. It might be that they're caching wikipedia aggressively on their proxies for many days and we're just not seeing the update yet. On very active articles the difference in article content should be a telltale sign.

@Gilles it was the local cache in UC mini that made the strange behavior I got, some pages cached forever. When I finally got rid off it I get the same behavior as @Jdlrobson .

I see your point @Jdlrobson I thought it was so strange that they removes that line after that first h-tag but I guess they render the page wider on the server side (the line is added if the width is smallish). That match.

When I try https://en.m.wikipedia.beta.wmflabs.org I got the button (not perfectly styled). Could be that they compress/slize the data harder on wikipedia.org. We should try to see if we can make that button render differently, would be nice to not have to test on production :)

I've checked how youtube looks using UC Mini in Speed mode. I've used Youtube since it's really popular in India see https://meta.wikimedia.org/wiki/New_Readers/Findings/India and then probably works good in UC Mini.

They serve a different version on the server user agent with super simple CSS (and inline styling). I think in the long run, we should aim for having a much simpler CSS for Grade C, skip styling and just give the content to the user. That will make Opera Mini & UC Mini much simpler in the future.