Drop support for ES3 javascript browsers in MediaWiki
Closed, ResolvedPublic

Description

Following removal of Grade A support for Internet Explorer 8 (T118303), we've been focussing on better defining the requirements for our JavaScript runtime. Specifically to define it based on features and specifications, rather than specific browser names and versions.

Once we've done the first step of that (T102318) we can also research what the difference is between our current browser support and which ones don't fully support ES5. When we've identified those browsers and dropped support for them we can enforce ES5 requirement via a simple feature test for ES5 strict mode (e.g. es5-skip).

I expect the timeline for this to span the next 2-3 months depending on workload and more important quarterly priorities.


ES5 support (feature test sampling)

Test results (breakdown by browser) at T128115#3066522

Related Objects

Krinkle created this task.Feb 25 2016, 7:02 PM
Restricted Application added subscribers: StudiesWorld, Aklapper. · View Herald TranscriptFeb 25 2016, 7:02 PM
Ricordisamoa added a subscriber: Ricordisamoa.
ori added a subscriber: ori.Feb 29 2016, 7:46 PM

Note that AssetCheck (and any other tools depending on PhantomJS 1) will break.

Krinkle moved this task from Inbox to Backlog on the Performance-Team board.Feb 29 2016, 7:47 PM
Krinkle triaged this task as "Normal" priority.Feb 29 2016, 7:54 PM
Krinkle moved this task from Inbox to Backlog on the MediaWiki-ResourceLoader board.

Change 296853 had a related patch set uploaded (by Krinkle):
rlfeature: Collect data about ES5 support

https://gerrit.wikimedia.org/r/296853

Change 296853 merged by jenkins-bot:
rlfeature: Collect data about ES5 support

https://gerrit.wikimedia.org/r/296853

Change 297102 had a related patch set uploaded (by Krinkle):
rlfeature: Collect data about ES5 support

https://gerrit.wikimedia.org/r/297102

Change 297102 merged by jenkins-bot:
rlfeature: Collect data about ES5 support

https://gerrit.wikimedia.org/r/297102

Mentioned in SAL [2016-07-01T22:22:24Z] <krinkle@tin> Synchronized php-1.28.0-wmf.8/extensions/WikimediaEvents/modules/: T128115 (duration: 00m 30s)

Mentioned in SAL [2016-07-01T22:23:04Z] <krinkle@tin> Synchronized php-1.28.0-wmf.8/extensions/WikimediaEvents/extension.json: T128115 (duration: 00m 37s)

Krinkle edited the task description. (Show Details)Jul 1 2016, 10:39 PM
Krinkle edited the task description. (Show Details)

The production test is running at about 98.6% success rate (i.e., if we were to require strict support, ~1.4% of current JS-executing users would lose JS). Most of them are IE9 and Mobile Safari 5 (iOS 4). This is acceptable, but not wonderful.

An alternative might be to require IE9's flavour of ES5 (everything except strict), which should mean that we can use most ES5-requiring libraries/etc. and remove cruft, and would allow us to do T96901: ResourceLoader JavaScript parser should allow ES5 syntax features now.

Thoughts?

[..] Most of them are IE9 and Mobile Safari 5 (iOS 4). [..] An alternative might be to require IE9's flavour of ES5 (everything except strict).

Mobile Safari 5 is most likely worse than IE 9.

According to https://kangax.github.io/compat-table/es5/:

  • IE 9 (85% of ES5 features). Missing: strict mode.
  • Safari 5 (59% of ES5 features). Missing: strict mode, free property names, Object.* methods, Date.* methods, Function#bind, modern parseInt, unicode variable names.

(Note that's for desktop Safari, not Mobile Safari. The compat table has entries for iOS 7+, but not iOS 4.)

[..] Most of them are IE9 and Mobile Safari 5 (iOS 4). [..] An alternative might be to require IE9's flavour of ES5 (everything except strict).

Mobile Safari 5 is most likely worse than IE 9.

Yeah, so my idea would be to drop JS for Mobile Safari 5 but keep it for IE9.

Krinkle added a comment.EditedJul 7 2016, 10:00 PM

Most of them are IE9 and Mobile Safari 5 (iOS 4).

I said that on IRC based on eyeballing a tail of the stream for a few minutes. Since the hitrate per minute is quite low, that was rather biased. Here's a better query.

This is from a sample of the last 1,000,000 statsv messages (~5 hour window). About 35,000 of those were for es5strict metrics. 357 messages reported es5strict.fail (~1%):

results.txt
2       Internet Explorer 7     (MSIE 7.0; Trident/5.0)  # IE 9 in IE 7 compat mode
141     Internet Explorer 9     (MSIE 9.0; Trident/5.0)
11      IE Mobile 9             (MSIE 9.0; Windows Phone; Trident/5.0; IEMobile/9.0)

1       iOS Safari 3.2          (OS 3_2; Version/4.0 Mobile/ Safari/)
2       iOS Safari 4.1          (OS 4_1; Version/4.0 Mobile/ Safari/)
2       iOS Safari 4.2          (OS 4_2; Version/5.0 Mobile/ Safari/)
5       iOS Safari 4.3          (OS 4_3; Version/5.0 Mobile/ Safari/)
1       iOS Safari 5.0          (OS 5_0; Version/5.1 Mobile/ Safari/)

2       Android Browser 2.1     (Android 2.1; Version/4.0 Mobile Safari/)
4       Android Browser 2.2     (Android 2.2; Version/4.0 Mobile Safari/)
121     Android Browser 2.3     (Android 2.3; Version/4.0 Mobile Safari/)
2       Android Browser 3.1     (Android 3.1; Version/4.0 Mobile Safari/)
1       Android Browser 3.2     (Android 3.2; Version/4.0 Mobile Safari/)
27      Android Browser 4.0     (Android 4.0; Version/4.0 Mobile Safari/)

1       Safari 4.0              (Windows; Version/4.0 Safari/)
2       Safari 4.0              (Mac OS X; Version/4.0 Safari/)
3       Safari 4.1              (Mac OS X; Version/4.1 Safari/)
4       Safari 5.0              (Mac OS X; Version/5.0 Safari/)

3       Firefox 3.5             (Gecko/2009 Firefox/3.5)
10      Firefox 3.6             (Gecko/ Firefox/3.6)

1       Dolfin 2.0              (SAMSUNG-GT; Bada/1.0; Dolfin/2.0 Mobile)
1       Google Earth 6.1        (AppleWebKit/; Google Earth/6.1; Safari/)
2       Google Earth 7.1        (AppleWebKit/; Google Earth/7.1; Safari/)

1       Chrome 5                (Chrome/5.)
1       Chrome 10               (Chrome/10.)
4       Chrome 11               (Chrome/11.)
2       Chrome 12               (Chrome/12.)

Top 3:

  • 152 Internet Explorer 9 (MSIE 9)
  • 127 Android Browser 2 (Android 2; Version/4; Mobile Safari)
  • 27 Android Browser 4 (Android 4; Version/4; Mobile Safari)

The most common engine version is Mobile Safari 4 with 166 hits in total (combine: iOS 3-4.1, Android, 2-4, Safari 4).

ori added a comment.Jul 7 2016, 10:36 PM

It'd be good to know how much logged-in traffic we get from these browsers. My sense is that we tend to overestimate the value that our JavaScript code provides for readers.

Krinkle moved this task from Backlog to Blocked on the Performance-Team board.Dec 6 2016, 12:40 AM
Krinkle added a comment.EditedMar 2 2017, 1:51 AM

Another run down.

From Grafana

https://grafana.wikimedia.org/dashboard/db/resourceloader-feature-test?from=1467763200000&to=1488326400000

Related changes

This metric is collected using our current JavaScript pipeline, as such it only runs after the Grade A feature test passes. (see startup.js)

Breakdown

From all statsv messages currently in Kafka's buffer (last 14 days; Feb 16 - March 1). 20967 hits for es5strict.fail.

Top 15 ua.family + ua.major:

9888 IE 9
3572 Android 2
2425 Android 4
 884 Mobile Safari 5
 874 Firefox 3
 736 IE Mobile 9
 390 IE 7
 232 Android 3
 226 BlackBerry WebKit 6
 199 Safari 5
 182 Opera Mobile 11
 180 Mobile Safari 4
 178 Chrome 12
 165 Safari 4
 138 Amazon Silk 1

Top 15 ua.family ua.major + os.family os.major:

6777 IE 9 (Windows 7)
3572 Android 2 (Android 2)
3108 IE 9 (Windows Vista)
2425 Android 4 (Android 4)
 859 Mobile Safari 5 (iOS 4)
 736 IE Mobile 9 (Windows Phone 7)
 316 IE 7 (Windows 7)
 232 Android 3 (Android 3)
 229 Firefox 3 (Windows XP)
 226 BlackBerry WebKit 6 (BlackBerry OS 6)
 190 Firefox 3 (Windows 7)
 188 Safari 5 (Mac OS X 10)
 176 Firefox 3 (Mac OS X 10)
 158 Safari 4 (Mac OS X 10)
 128 Amazon Silk 1 (Android)

Looking only at browsers we actively support as Grade A (See Browser support matrix):

February 2017 (total of es5strict.fail was 0.59% of the grand total):

  • 0.27% - Internet Explorer 9
  • 0.10% - Android 2
  • 0.06% - Android 4
  • 0.02% - Mobile Safari 5 (iOS 4)
  • 0.006% - Safari 5
  • ...
  • Total es5strict-fail: 0.59%

Compared to last year:

July 2016
Total es5strict-fail: 1.0%

  • 0.42% - Internet Explorer 9
  • 0.35% - Android 2
  • 0.07% - Android 4
  • 0.03% - Mobile Safari 5
  • 0.01% - Safari 5

1 in 168 seems like a reasonable cut-off threshold. Let's do it.

It'd be good to know how much logged-in traffic we get from these browsers. My sense is that we tend to overestimate the value that our JavaScript code provides for readers.

The user agent is collected as part of the event capsule for the Edit schema. Queried from Kafka using kafkacat (eventloggin_Edit). Has 3708277 entries in the current buffer (last 2 weeks). Note that this represents all 7 types of Edit beacon actions (see https://meta.wikimedia.org/wiki/Schema:Edit). Aside from the obvious successful saving of an edit, it also includes things like initialisation of the editor and viewing action=edit- which is presumably why some search engines are included.

Top 50 ua.family + ua.major

1057841 Chrome 56
 378230 Firefox 51
 254590 Other
 234679 IE 11
 211989 Mobile Safari 10
 136902 Chrome Mobile 56
  81915 Edge 14
  73770 Chrome 49
  72271 Chrome Mobile 55
  62251 Chrome 55
  57645 Safari 10
  56337 Chrome 34
  42738 Samsung Internet 4
  40414 Opera 43
  33255 Mobile Safari 9
  32209 Googlebot 2
  30744 Yandex Browser 17
  29246 Android 4
  23780 YandexBot 3
  22584 com/bot
  20570 Chrome 54
  20527 Samsung Internet 3
  20378 Firefox 45
  18834 IE 10
  18809 3iCrawler 2
  18465 Firefox 50
  17252 Chrome Mobile iOS 56
  16970 Safari 9
  16569 Firefox 47
  15209 Chrome Mobile 30
  15091 IE 9
  14913 IE 6
  14320 bingbot 2
  13001 archive.org_bot
  12687 IE 8
  12642 Chrome 50
  12210 Chrome Mobile 50
  12164 Chrome Mobile 54
  11731 Chrome Mobile 43
   9904 Firefox 48
   9628 Mobile Safari 8
   8856 Chrome 53
   8342 IE 7
   7490 Chrome Mobile 52
   7477 Edge 13
   7451 Chrome Mobile 40
   7445 Chrome Mobile 46
   6981 Mobile Safari 7
   6978 Chrome 48
   6961 Chrome Mobile 28

Of particular interest:

total 3708277
--

1057841 Chrome 56
 378230 Firefox 51
 254590 Other
...
  29246 Android 4
...
  18834 IE 10
  15091 IE 9
  14913 IE 6
  12687 IE 8
   8342 IE 7
...

# top 50 ends here
# below is not in the top 50

...
   3579 Safari 5
...
   1810 Mobile Safari 5
...
    817 Android 2
...
BrowserPage views (Feb 2017 - from T128115#3066522)Edit beacons (percentage of total)
Internet Explorer 9(47% of 0.59%)     0.27%0.41%
Android 2(17% of 0.59%)     0.10%0.02%
Android 4(11% of 0.59%)     0.06%0.78%
Mobile Safari 5(4% of 0.59%)       0.02%0.05%
Safari 5(1% of 0.59%)       0.006%0.09%
total that doesn't support ES50.59%1.35%

Change 340893 had a related patch set uploaded (by Jforrester):
[mediawiki/core] [WIP] startup: Drop support for ES3 browsers

https://gerrit.wikimedia.org/r/340893

Jhernandez added a subscriber: Jhernandez.
Paladox added a subscriber: Paladox.Mar 3 2017, 7:00 PM
Krinkle added a comment.EditedMar 7 2017, 9:55 PM

T100879: Blacklist Safari<5.1 from receiving JavaScript in the startup module.

I'm not 100% sure that requiring ES5 strict will effectively drop Safari 5.0 and lower, and thusly require Safari 5.1.

In fd5c3b5817 (deprecate JSON polyfill), the feature test documents that it dropped Safari 4 and now requires Safari 5.0. We documented this because the data gathered at T141344#2784065 showed failures coming from Safari 4.x. However, at the same time, http://caniuse.com/#feat=json clearly marks Safari 4 as supporting JSON.

This time, again, the data gathered here at T128115#3066522 shows failures from Safari 5.0. But, http://caniuse.com/#feat=use-strict shows Safari 5.0 and 5.1 as partially supporting strict mode. Enough for the feature test to pass, anyway.

This is reason take another look at the data. This time, looking at both the passes and the failures.

Looking through the same es5strict-fail dataset again (Kafka topic "statsv"; Feb 16 - March 1; 20,967 hits for "es5strict.fail").

  • 7779 matches for Version/[45].* Safari
    • 6225 matches contain Android (excluded below)
    • 348 matches for Version/4.* Safari
      • 227 matches for Version/4.0.* Safari
      • 120 matches for Version/4.1.* Safari
    • 1206 matches for Version/5.* Safari
      • 1171 matches for Version/5.0.* Safari
      • 35 matches for Version/5.1.* Safari

I also generated a new data set for es5strict-pass (Kafka topic "statsv"; Feb 23 - March 6; 4,211,379 hits for "es5strict.pass" containing "Safari")

  • 219,802 matches for Version/[45].* Safari
    • 7805 matches not containing Android (excluded below)
      • 253 matches for Version/4.* Safari
        • 253 matches for Version/4.0.* Safari
        • 0 matches for Version/4.1.* Safari
      • 7552 matches for Version/5.* Safari
        • 935 matches for Version/5.0.* Safari
        • 6617 matches for Version/5.1.* Safari
      • Remaining non-Android "Safari"-matches are mostly Chrome (which contains "Safari" in its UA-string) and newer versions of Safari

The same data interpreted by ua-parser:

es5strict.fail.Safari
# cat es5strict.fail.Safari45 |  ./bin/uaparse | grep -vE 'Android|Chrome' | sort | uniq -c | sort -rn
    859 Mobile Safari 5 (iOS 4)
    188 Safari 5 (Mac OS X 10)
    158 Safari 4 (Mac OS X 10)
    103 Mobile Safari 4 (iOS 3)
     77 Mobile Safari 4 (iOS 4)
     25 Mobile Safari 5 (iOS 5)
      ...
es5strict.pass.Safari
# cat es5strict.pass.Safari45 |  ./bin/uaparse | grep -vE 'Android|Chrome' | sort | uniq -c | sort -rn
   3683 Safari 5 (Mac OS X 10)
   3325 Mobile Safari 5 (iOS 5)
    338 Kindle 3 (Kindle 3)
    113 Mobile Safari 4 (iOS 3)
    106 Safari 5 (Windows 7)
     86 Safari 5 (Windows 8)
     62 WebKit Nightly 535 (Linux)
     41 Mobile Safari 4 (iOS 5)
     37 WebKit Nightly 534 (Linux)
     37 Mobile Safari 5 (iOS 10)
     28 Safari 5 (Linux)
     24 Mobile Safari 5 (iOS 4)
     21 Safari 4 (Mac OS X 10)
     19 Safari 5 (Windows XP)
     19 Safari 5 (Windows Vista)
     18 Mobile Safari 5 (iOS 9)
     16 UC Browser 3 (iOS 5)
     15 Safari 4 (Linux)
     15 Midori 0 (Mac OS X)
     12 Mobile Safari 4 (iOS 4)
      ...
browser + ospassfail
Mobile Safari 4 (iOS 3)113103
Mobile Safari 4 (iOS 4)1277
Mobile Safari 4 (iOS 5)410
Mobile Safari 5 (iOS 4)24859
Mobile Safari 5 (iOS 5)368325
Mobile Safari 5 (iOS 9)180
Mobile Safari 5 (iOS 10)370
Safari 4 (Mac OS X 10)21158
Safari 5 (Mac OS X 10)3683188

This data suggests to me that ES5 may have been backported to a minor version of Safari 4 and 5, which in turn was release through a minor iOS update. But a closer look at the minor versions doesn't help anything. It's a mess.

  • Safari 5.1.9 (Mac OS X) consistently passes es5strict. (0 failures)
  • Mobile Safari 5.1 consistently passes es5strict. (0 failures)
  • Mobile Safari 5.0 (iOS 5.0.1), Mobile Safari 5.0 (iOS 5.1), Mobile Safari 5.0 (iOS 5.1.1) sometimes passes, sometimes fails. Same exact UA strings found in both datasets.
  • Mobile Safari 5.0.2 (iOS 4.3.2) sometimes passes, sometimes fails. Same exact UA strings found in both datasets.
  • Mobile Safari 5.x (iOS 4.3.4) consistently fails es5strict.
  • Mobile Safari 5.x (iOS 4.3.1) consistently fails es5strict.

WebKit build versions don't help either:

es5strict.pass.Safari45.AppleWebKit
   3 AppleWebKit/528
   9 AppleWebKit/530
 187 AppleWebKit/531
  12 AppleWebKit/532
  37 AppleWebKit/533
7372 AppleWebKit/534
  68 AppleWebKit/535
  33 AppleWebKit/537
   6 AppleWebKit/600
  15 AppleWebKit/601
  36 AppleWebKit/602
es5strict.fail.Safari45.AppleWebKit
   6 AppleWebKit/530
 141 AppleWebKit/531
  71 AppleWebKit/532
1232 AppleWebKit/533
 104 AppleWebKit/534
TheDJ added a subscriber: TheDJ.Mar 10 2017, 9:14 AM

@Krinkle Remember that 'in app' web views were on a different version of webkit as Safari mobile for a long time, that created quite a mess. The web views also got unique back ports that might have messed with those version numbers.

Krinkle edited the task description. (Show Details)Mar 17 2017, 9:18 PM

Change 340893 merged by jenkins-bot:
[mediawiki/core@master] startup: Drop JavaScript support for ES3-only browsers

https://gerrit.wikimedia.org/r/340893

Jdforrester-WMF closed this task as "Resolved".Tue, Apr 4, 9:15 PM
Jdforrester-WMF removed a project: Patch-For-Review.
Jdforrester-WMF claimed this task.

Change 346466 had a related patch set uploaded (by Krinkle):
[mediawiki/extensions/WikimediaEvents@master] rlfeature: Remove es5strict support tracking

https://gerrit.wikimedia.org/r/346466

Change 346466 merged by jenkins-bot:
[mediawiki/extensions/WikimediaEvents@master] rlfeature: Remove es5strict support tracking

https://gerrit.wikimedia.org/r/346466