Page MenuHomePhabricator
Create Task
Maniphest T104942

TLS and *.wap/*.mobile multi-level subdomains of wikipedia.org
Closed, ResolvedPublic
Actions

Description

In addition to $lang.m.wikipedia.org and $lang.zero.wikipedia.org (which are fine!), we also have $lang.wap.wikipedia.org and $lang.mobile.wikipedia.org. These latter two seem to be apache-level rewrites (rather than redirects) to $lang.m.wikipedia.org at https://github.com/wikimedia/operations-puppet/blob/production/modules/mediawiki/files/apache/sites/redirects/redirects.dat#L484 . From my desktop browser they appear equivalent to .m., but perhaps there are still hacks in place for WAP on special user-agents?

These subdomains are not compatible with our TLS certs, so it's a problem that blocks HSTS-preload for wikipedia.org indirectly. My understanding is they're intended to be deprecated going forward, in which case perhaps we can just hurry up plans for their demise?

Details

SubjectRepoBranchLines +/-
Remove wap/mobile subdomains from redirectsoperations/puppetproduction+0 -17
Remove mobile.wp.ooperations/dnsmaster+0 -1
Remove wap and mobile subdomainsoperations/dnsmaster+1 -8
Customize query in gerrit

Related Objects
Search...

Event Timeline

BBlack created this task.Jul 7 2015, 5:34 AM
BBlack raised the priority of this task from to Needs Triage.
BBlack updated the task description. (Show Details)
BBlack added projects: acl*sre-team, Traffic, HTTPS, Mobile.
BBlack subscribed.
Restricted Application added subscribers: Matanya, Aklapper. · View Herald TranscriptJul 7 2015, 5:34 AM
BBlack added a parent task: T102824: Clean up DNS/redirects for TLS.Jul 7 2015, 5:35 AM
BBlack added a parent task: T104244: Preload HSTS.
Chmarkine subscribed.Jul 7 2015, 8:03 AM

How many requests to these domains there are in the log? *.wap was deprecated in early 2009, and *.mobile was deprecated in late 2011. Google has 39,300 results for site:*.mobile.wikipedia.org, which is fewer than the 96,700 results for site:www.*.wikipedia.org. I think it is fine to delete them from DNS.

BBlack added a comment.EditedJul 7 2015, 11:16 PM

We're already sending rel=canonical to the desktop sites from both of them as well, like we do for .m.. The traffic logs look similar to other stuff we've killed recently - not statistically significant, but it is there. My inclination is to dump them. From the per-domain-count list, vs 17425538 for en.m.wikipedia.org and 98696288 for en.wikipedia.org:

$ egrep '\.(wap|mobile)\.' per-domain-count
  17587 "en.mobile.wikipedia.org"
   6199 "zh.wap.wikipedia.org"
   3845 "zh.mobile.wikipedia.org"
   2790 "ru.mobile.wikipedia.org"
   2679 "en.wap.wikipedia.org"
   1874 "fr.wap.wikipedia.org"
   1380 "de.mobile.wikipedia.org"
   1308 "fr.mobile.wikipedia.org"
    974 "ru.wap.wikipedia.org"
    937 "de.wap.wikipedia.org"
    759 "it.wap.wikipedia.org"
    661 "pt.mobile.wikipedia.org"
    517 "ja.mobile.wikipedia.org"
    458 "it.mobile.wikipedia.org"
    444 "vi.wap.wikipedia.org"
    417 "es.wap.wikipedia.org"
    341 "fa.mobile.wikipedia.org"
    300 "nl.mobile.wikipedia.org"
    300 "ar.mobile.wikipedia.org"
    280 "hu.mobile.wikipedia.org"
    278 "es.mobile.wikipedia.org"
    274 "cs.mobile.wikipedia.org"
    263 "no.mobile.wikipedia.org"
    262 "fi.wap.wikipedia.org"
    234 "id.wap.wikipedia.org"
    225 "th.mobile.wikipedia.org"
    195 "cs.wap.wikipedia.org"
    174 "sr.mobile.wikipedia.org"
    155 "oc.mobile.wikipedia.org"
    142 "ms.mobile.wikipedia.org"
    137 "ru.m.mobile.wikipedia.org"
    136 "id.mobile.wikipedia.org"
    131 "sv.mobile.wikipedia.org"
    130 "pl.wap.wikipedia.org"
    129 "sr.wap.wikipedia.org"
    126 "el.wap.wikipedia.org"
    119 "vi.mobile.wikipedia.org"
    108 "he.mobile.wikipedia.org"
    104 "tr.mobile.wikipedia.org"
     93 "pl.mobile.wikipedia.org"
     85 "he.wap.wikipedia.org"
     81 "fa.wap.wikipedia.org"
     57 "bg.wap.wikipedia.org"
     57 "ba.wap.wikipedia.org"
     52 "af.wap.wikipedia.org"
     48 "zh-yue.mobile.wikipedia.org"
     47 "ta.mobile.wikipedia.org"
     43 "mg.mobile.wikipedia.org"
     42 "ba.mobile.wikipedia.org"
     38 "bg.mobile.wikipedia.org"
     36 "ka.mobile.wikipedia.org"
     32 "el.mobile.wikipedia.org"
     31 "te.mobile.wikipedia.org"
     30 "eu.wap.wikipedia.org"
     30 "eu.mobile.wikipedia.org"
     26 "fi.mobile.wikipedia.org"
     24 "nv.mobile.wikipedia.org"
     23 "se.mobile.wikipedia.org"
     19 "no.wap.wikipedia.org"
     18 "zh-yue.wap.wikipedia.org"
     18 "ka.wap.wikipedia.org"
     17 "se.wap.wikipedia.org"
     15 "ia.mobile.wikipedia.org"
     13 "ksh.mobile.wikipedia.org"
     11 "ug.mobile.wikipedia.org"
     10 "ia.wap.wikipedia.org"
      9 "roa-tara.mobile.wikipedia.org"
      9 "am.wap.wikipedia.org"
      7 "am.mobile.wikipedia.org"
      5 "is.mobile.wikipedia.org"
      5 "arc.mobile.wikipedia.org"
      3 "mi.mobile.wikipedia.org"
      3 "kl.mobile.wikipedia.org"
      2 "www.fr.mobile.wikipedia.org"
      2 "pms.mobile.wikipedia.org"
      2 "arc.wap.wikipedia.org"
      1 "zh.wap.wikipedia.org:443"
      1 "za.wap.wikipedia.org"
      1 "www.zh.wap.wikipedia.org"
      1 "www.zh.mobile.wikipedia.org"
      1 "www.pnt.mobile.wikipedia.org"
      1 "www.nds-nl.mobile.wikipedia.org"
      1 "www.jp.wap.wikipedia.org"
      1 "www.epo.wap.wikipedia.org"
      1 "www.en.mobile.wikipedia.org"
      1 "www.dk.mobile.wikipedia.org"
      1 "uploa0000000000000000000000000000000000000000l,application/vnd.wap.xhtml+xml,application/xhtml+xml, image/vnd.wap.wbmp, image00000000000000000000e/jpeg, image/png, i00000000000000000000*/*;q=0.1"
      1 "ug.wap.wikipedia.org"
      1 "static.mobile.espn.go.com"
      1 "ko.mobile.wikipedia.org"
      1 "kl.wap.wikipedia.org"
      1 "ig.wap.wikipedia.org"
      1 "bat-smg.mobile.wikipedia.org"
      1 "bar.wap.wikipedia.org"
      1 "ab.mobile.wikipedia.com"
      1 "aa.wap.wikipedia.org"
gerritbot subscribed.Jul 9 2015, 10:11 PM

Change 223972 had a related patch set uploaded (by BBlack):
Remove wap and mobile subdomains

https://gerrit.wikimedia.org/r/223972

gerritbot added a project: Patch-For-Review.Jul 9 2015, 10:11 PM
BBlack added subscribers: dr0ptp4kt, Yurik, • DFoy.Jul 10 2015, 2:11 AM
BBlack triaged this task as Medium priority.Jul 13 2015, 6:43 PM
BBlack moved this task from Backlog to Upcoming on the Traffic board.Jul 13 2015, 7:03 PM
BBlack mentioned this in T104681: HTTPS Plans (tracking / high-level info).Jul 14 2015, 12:12 AM
dr0ptp4kt added a comment.Jul 15 2015, 7:59 PM

I added Reading web engineers to the patch and will add them to this ticket as well in case they have any concerns.

dr0ptp4kt set Security to None.Jul 15 2015, 8:00 PM
dr0ptp4kt added subscribers: • Jhernandez, • jhobs, • bmansurov and 3 others.
dr0ptp4kt added a subscriber: • Tnegrin.Jul 15 2015, 8:05 PM
BBlack added a comment.Jul 15 2015, 8:17 PM

@dr0ptp4kt thanks :)

• Tnegrin added a comment.Jul 15 2015, 10:27 PM

@Jhernandez, @dr0ptp4kt -- what's the level of effort for Reading here?

BBlack added a subscriber: faidon.Jul 15 2015, 10:30 PM

It shouldn't be a matter of any real effort. As far as I know, both of these subdomains were already deprecated years ago and aren't advertised or linked anywhere. Traffic stats are fairly low, probably from organic discovery of them, but can't be sure. @Chmarkine gave links and google stats on that earlier in the ticket, and I posted up some numbers from @faidon's stats on our terminators as well.

Mostly, we're just trying to get confirmation of dis-use before we pull them, so that we don't find out the Hard Way that there was still some obscure need for these somewhere.

• Tnegrin added a comment.Jul 15 2015, 10:59 PM

cool -- I'll defer to the expertise of Adam and Joaquin on this.

dr0ptp4kt added a comment.Jul 16 2015, 12:29 PM

A glance at the User-Agent values suggests the domains are commonly being visited by devices supporting HTML instead of WML. I think people are just coming from links from other sites. We deprecated WML sometime back. I'll email mobile-l and wikitech-l to see if anyone knows of a reason to keep the domains.

dr0ptp4kt added a comment.Jul 16 2015, 12:41 PM

Here are the mails to wikitech-l and mobile-l.

https://lists.wikimedia.org/pipermail/wikitech-l/2015-July/082408.html

https://lists.wikimedia.org/pipermail/mobile-l/2015-July/009453.html

Ricordisamoa subscribed.Jul 16 2015, 12:46 PM
dr0ptp4kt added a comment.Jul 16 2015, 1:28 PM

As mentioned on threads: looks like the user pageviews for wap.wikipedia.org and mobile.wikipedia.org subdomains are approximately 0.02% of the size of pageviews for m.wikipedia.org subdomains based on a recent one day check.

hive> select count(*) from
wmf.webrequest where
year = 2015 and month = 7 and day = 14
and access_method = 'mobile web'
and (uri_host like '%.wap.wikipedia.org' OR uri_host like '%.mobile.wikipedia.org')
and is_pageview = true and agent_type = 'user';

35,543

hive> select count(*) from
wmf.webrequest where
year = 2015 and month = 7 and day = 14
and access_method = 'mobile web'
and uri_host like '%.m.wikipedia.org'
and is_pageview = true and agent_type = 'user';

202,024,891

dr0ptp4kt added a project: Reading-Admin.Jul 16 2015, 1:53 PM
dr0ptp4kt moved this task from Backlog to Doing on the Reading-Admin board.
• Tnegrin added a comment.Jul 16 2015, 3:06 PM

Hi Adam -- can you please look at the UAs for the queries that come to
these domains? The question was asked on mobile-l and I think it would be
interesting (and hopefully straightforward) -- I think pulling the UA and
limiting to the top 10 would be enough.

thanks,

-Toby

Chmarkine added a comment.Jul 16 2015, 4:02 PM

Could you look at the referrers as well? Do most of the requests come from search engines?

dr0ptp4kt added a comment.Jul 16 2015, 4:40 PM

@Tnegrin, @Chmarkine:

I'll pull together some User-Agent and Referer stuff.

dr0ptp4kt added a comment.Jul 16 2015, 10:01 PM

@Tnegrin, @Chmarkine: and now, the data.

Queries spanning multiple days seem to sometimes flake out when not restricted to specific hours. So I ran queries against specific hours 1, 7, 13, and 19 for for 1-14 July 2015 for pageviews and pageviews by browser family.

For the Referer stuff I ran the query against those same hours, but for just two days - one in June and one in July (went back to June just to make sure nothing drastically changed due to the HTTPS cutover). Most external Referer values do indeed look to be coming from search engines.

Pageviews

Pageviews appeared within a reasonably consistent range throughout a 14 day period. Pageviews on the mobiledot/wapdot subdomains as compared to mdot subdomains for Wikipedia hovered around the 0.01-0.02% range, pretty low.

mobile/wap subdomains

hive> select day, count(*) from
wmf.webrequest where
year = 2015 and month = 7 and day < 15
and (hour = 1 or hour = 7 or hour = 13 or hour = 19)
and access_method = 'mobile web'
and (uri_host like '%.wap.wikipedia.org' OR uri_host like '%.mobile.wikipedia.org')
and is_pageview = true and agent_type = 'user'
group by day;

1 3393
2 3642
3 4320
4 4432
5 4797
6 4515
7 5660
8 6106
9 5724
10 4680
11 4959
12 5905
13 5186
14 5665

mdot subodmains

hive> select day, count(*) from
wmf.webrequest where
year = 2015 and month = 7 and day < 15
and (hour = 1 or hour = 7 or hour = 13 or hour = 19)
and access_method = 'mobile web'
and uri_host like '%.m.wikipedia.org'
and is_pageview = true and agent_type = 'user'
group by day;

1 32786020
2 32349055
3 32265286
4 35559106
5 37742879
6 36129682
7 34010858
8 33406901
9 32659640
10 32285858
11 35103953
12 39178703
13 34505514
14 34308916

User-Agent - Browser Family

UA trends were relatively consistent. Sogou and UC were relatively high in the mobiledot/wapdot case as compared to the mdot Wikipedia case, probably owing to the apparent geo/language, as evidenced by external Referer values.

Just a sidenote: not sure bingbot should be considered of user agent type.

What follows are the top 10 browsers families for mobiledot/wapdot Wikipedia access, followed by the top 25 browser families for mdot Wikipedia access for a point of reference and general fun.

mobile/wap subdomains

hive> select day, user_agent_map['browser_family'], count(*) from
wmf.webrequest where
year = 2015 and month = 7 and day < 15
and (hour = 1 or hour = 7 or hour = 13 or hour = 19)
and access_method = 'mobile web'
and (uri_host like '%.wap.wikipedia.org' OR uri_host like '%.mobile.wikipedia.org')
and is_pageview = true and agent_type = 'user'
group by day, user_agent_map['browser_family'];

day family count
1 Sogou Explorer 683
1 UC Browser 606
1 bingbot 492
1 Android 345
1 Chrome 253
1 Mobile Safari 226
1 IE 206
1 QQ Browser Mobile 159
1 Firefox 118
1 Chrome Mobile 95
2 Sogou Explorer 813
2 UC Browser 769
2 Android 421
2 Chrome 395
2 Mobile Safari 219
2 UP.Browser 214
2 Chrome Mobile 210
2 IE 171
2 QQ Browser Mobile 157
2 Firefox 93
3 Sogou Explorer 1405
3 UC Browser 744
3 Android 521
3 Chrome 376
3 Mobile Safari 233
3 IE 231
3 QQ Browser Mobile 221
3 Firefox 142
3 Chrome Mobile 131
3 bingbot 101
4 UC Browser 954
4 Android 754
4 Sogou Explorer 669
4 Chrome 445
4 Mobile Safari 266
4 bingbot 257
4 IE 255
4 Firefox 203
4 Chrome Mobile 168
4 QQ Browser Mobile 161
5 UC Browser 1072
5 Sogou Explorer 944
5 Android 652
5 Chrome 500
5 bingbot 353
5 Mobile Safari 250
5 IE 246
5 QQ Browser Mobile 217
5 Chrome Mobile 199
5 Firefox 173
6 UC Browser 1152
6 Sogou Explorer 676
6 Android 584
6 Chrome 500
6 Mobile Safari 351
6 IE 267
6 Firefox 230
6 QQ Browser Mobile 230
6 bingbot 142
6 Baidu Browser 131
7 Sogou Explorer 1874
7 UC Browser 918
7 Chrome 610
7 Android 573
7 IE 503
7 Mobile Safari 349
7 QQ Browser Mobile 278
7 Firefox 166
7 Chrome Mobile 150
7 Baidu Browser 71
8 Sogou Explorer 2158
8 UC Browser 1068
8 Android 789
8 Chrome 506
8 IE 399
8 Mobile Safari 254
8 Chrome Mobile 244
8 QQ Browser Mobile 206
8 Firefox 165
8 Baidu Browser 146
9 UC Browser 1095
9 Chrome 852
9 Sogou Explorer 823
9 Android 688
9 bingbot 486
9 IE 457
9 Mobile Safari 293
9 Chrome Mobile 259
9 QQ Browser Mobile 229
9 Firefox 200
10 Sogou Explorer 926
10 UC Browser 823
10 Android 530
10 IE 425
10 Chrome 381
10 Other 334
10 bingbot 300
10 Mobile Safari 248
10 QQ Browser Mobile 193
10 Firefox 170
11 UC Browser 989
11 Android 664
11 Sogou Explorer 524
11 Chrome 504
11 Baidu Browser 489
11 Mobile Safari 387
11 IE 334
11 Other 286
11 QQ Browser Mobile 213
11 Chrome Mobile 192
12 UC Browser 1152
12 Sogou Explorer 1126
12 Android 796
12 Chrome 552
12 IE 418
12 Mobile Safari 414
12 QQ Browser Mobile 312
12 Other 298
12 Chrome Mobile 252
12 Firefox 209
13 UC Browser 1046
13 Sogou Explorer 905
13 Android 613
13 IE 573
13 Chrome 429
13 Other 401
13 Mobile Safari 358
13 Chrome Mobile 224
13 QQ Browser Mobile 191
13 Firefox 153
14 Sogou Explorer 1109
14 Android 1012
14 UC Browser 929
14 Chrome 545
14 Mobile Safari 491
14 Other 414
14 IE 318
14 Chrome Mobile 302
14 QQ Browser Mobile 234

mdot subdomains

hive> select day, user_agent_map['browser_family'], count(*) from
wmf.webrequest where
year = 2015 and month = 7 and day < 15
and (hour = 1 or hour = 7 or hour = 13 or hour = 19)
and access_method = 'mobile web'
and uri_host like '%.m.wikipedia.org'
and is_pageview = true and agent_type = 'user'
group by day, user_agent_map['browser_family'];

day family count
1 Mobile Safari 12132057
1 Chrome Mobile 8801804
1 Android 3749250
1 Other 2528518
1 Opera Mini 1403237
1 Chrome 900893
1 IE Mobile 681078
1 Chrome Mobile iOS 621432
1 BingPreview 449956
1 UC Browser 297750
1 IE 204187
1 Firefox Mobile 178981
1 BlackBerry WebKit 152236
1 Amazon Silk 118894
1 Opera Mobile 102316
1 NetFront NX 64352
1 Firefox 53725
1 Yandex Browser 50416
1 UP.Browser 40531
1 NetFront 37946
1 Nokia Browser 29383
1 bingbot 28545
1 Nokia Services (WAP) Browser 24726
1 Opera 22199
1 Safari 19415
1 BlackBerry 18870
2 Mobile Safari 12004029
2 Chrome Mobile 8784320
2 Android 3744649
2 Other 2524019
2 Opera Mini 1376195
2 Chrome 866146
2 IE Mobile 691822
2 Chrome Mobile iOS 612938
2 UC Browser 294477
2 BingPreview 250466
2 IE 189961
2 Firefox Mobile 162782
2 BlackBerry WebKit 145485
2 Amazon Silk 115699
2 Opera Mobile 105234
2 NetFront NX 62323
2 Firefox 55917
2 Yandex Browser 52083
2 bingbot 43918
2 UP.Browser 40129
2 NetFront 35811
2 Nokia Browser 29375
2 Nokia Services (WAP) Browser 23477
2 Opera 21882
2 Safari 19424
3 Mobile Safari 12138382
3 Chrome Mobile 8737983
3 Android 3661523
3 Other 2496413
3 Opera Mini 1331507
3 Chrome 850364
3 IE Mobile 681066
3 Chrome Mobile iOS 626966
3 BingPreview 289122
3 UC Browser 279870
3 IE 168306
3 Firefox Mobile 162661
3 BlackBerry WebKit 143762
3 Amazon Silk 123083
3 Opera Mobile 102074
3 NetFront NX 60822
3 Firefox 53944
3 Yandex Browser 51476
3 bingbot 48486
3 UP.Browser 40755
3 NetFront 35489
3 Nokia Browser 29147
3 Nokia Services (WAP) Browser 22745
3 Opera 21359
3 BlackBerry 17753
4 Mobile Safari 13730159
4 Chrome Mobile 9532199
4 Android 4022570
4 Other rMW25177828d600
4 Opera Mini 1345187
4 Chrome 964526
4 IE Mobile 741965
4 Chrome Mobile iOS 702966
4 BingPreview 468216
4 UC Browser 286850
4 Firefox Mobile 182468
4 IE 168662
4 BlackBerry WebKit 154417
4 Amazon Silk 141286
4 Opera Mobile 108665
4 NetFront NX 68877
4 Firefox 58791
4 Yandex Browser 54158
4 bingbot 44750
4 UP.Browser 41254
4 NetFront 37933
4 Nokia Browser 29757
4 Opera 23560
4 Nokia Services (WAP) Browser 22688
4 Safari 17528
5 Mobile Safari 14771801
5 Chrome Mobile 10252627
5 Android 4290877
5 Other 2497504
5 Opera Mini 1430023
5 Chrome 1096304
5 IE Mobile 803601
5 Chrome Mobile iOS 783147
5 UC Browser 310218
5 Firefox Mobile 196526
5 IE 183440
5 BlackBerry WebKit 168692
5 Amazon Silk 153276
5 BingPreview 140881
5 Opera Mobile 116913
5 Firefox 85277
5 NetFront NX 70825
5 Yandex Browser 58552
5 bingbot 48201
5 UP.Browser 42779
5 NetFront 40441
5 Nokia Browser 31471
5 Opera 27185
5 Nokia Services (WAP) Browser 23803
5 Safari 21766
6 Mobile Safari 14228044
6 Chrome Mobile 9753676
6 Android 3929643
6 Other 2532060
6 Opera Mini 1375394
6 Chrome 997612
6 IE Mobile 734240
6 Chrome Mobile iOS 731946
6 UC Browser 290711
6 BingPreview 284861
6 IE 178792
6 Firefox Mobile 172885
6 BlackBerry WebKit 157233
6 Amazon Silk 139901
6 Opera Mobile 109337
6 Firefox 81048
6 NetFront NX 64400
6 Yandex Browser 54215
6 UP.Browser 43042
6 bingbot 40085
6 NetFront 36980
6 Nokia Browser 29496
6 Opera 24721
6 Safari 24390
6 Nokia Services (WAP) Browser 24015
7 Mobile Safari 12806729
7 Chrome Mobile 9304625
7 Android 3815258
7 Other 2541857
7 Opera Mini 1383079
7 Chrome 994497
7 IE Mobile 706593
7 Chrome Mobile iOS 655893
7 UC Browser 287160
7 BingPreview 259074
7 IE 176418
7 Firefox Mobile 168050
7 BlackBerry WebKit 152605
7 Amazon Silk 126040
7 Opera Mobile 104946
7 Firefox 81895
7 bingbot 60863
7 NetFront NX 60185
7 Yandex Browser 53208
7 UP.Browser 43063
7 NetFront 37451
7 Nokia Browser 29615
7 Safari 24805
7 Nokia Services (WAP) Browser 24232
7 Opera 23075
8 Mobile Safari 12659503
8 Chrome Mobile 9184525
8 Android 3764545
8 Other 2495933
8 Opera Mini 1375041
8 Chrome 1020657
8 IE Mobile 696917
8 Chrome Mobile iOS 643371
8 UC Browser 283650
8 IE 188007
8 Firefox Mobile 168692
8 BlackBerry WebKit 155079
8 Amazon Silk 122635
8 Firefox 117103
8 Opera Mobile 106414
8 NetFront NX 57835
8 Yandex Browser 53029
8 UP.Browser 43942
8 NetFront 36813
8 bingbot 33094
8 Safari 30213
8 Nokia Browser 29398
8 Opera 27378
8 Nokia Services (WAP) Browser 24380
8 BlackBerry 18825
9 Mobile Safari 12354742
9 Chrome Mobile 9064487
9 Android 3696875
9 Other 2486943
9 Opera Mini 1357777
9 Chrome 914434
9 IE Mobile 695167
9 Chrome Mobile iOS 628220
9 UC Browser 289429
9 IE 184369
9 Firefox Mobile 163949
9 BlackBerry WebKit 150928
9 Amazon Silk 117744
9 Opera Mobile 106225
9 Firefox 70187
9 NetFront NX 60153
9 Yandex Browser 51033
9 UP.Browser 45109
9 NetFront 37170
9 Nokia Browser 29762
9 Nokia Services (WAP) Browser 24050
9 Opera 23910
9 Safari 20247
9 BlackBerry 18461
9 Dolfin 11563
10 Mobile Safari 12316817
10 Chrome Mobile 8919801
10 Android 3617929
10 Other 2488354
10 Opera Mini 1288886
10 Chrome 880277
10 IE Mobile 679295
10 Chrome Mobile iOS 629070
10 UC Browser 286555
10 IE 194987
10 Firefox Mobile 165586
10 BlackBerry WebKit 148878
10 Amazon Silk 120113
10 Opera Mobile 103405
10 Firefox 67870
10 NetFront NX 60565
10 Yandex Browser 51304
10 UP.Browser 43992
10 NetFront 36353
10 Nokia Browser 29745
10 Opera 24687
10 Nokia Services (WAP) Browser 22633
10 Safari 19044
10 BlackBerry 17754
10 Dolfin 10840
11 Mobile Safari 13795165
11 Chrome Mobile 9607690
11 Android 3860025
11 Other 2465461
11 Opera Mini 1349373
11 Chrome 967251
11 IE Mobile 744211
11 Chrome Mobile iOS 718981
11 UC Browser 295544
11 Firefox Mobile 184472
11 IE 181590
11 BlackBerry WebKit 167669
11 Amazon Silk 139257
11 Opera Mobile 109767
11 Firefox 70353
11 NetFront NX 64651
11 Yandex Browser 55602
11 UP.Browser 44536
11 bingbot 42287
11 NetFront 37872
11 Safari 30587
11 Nokia Browser 29739
11 Opera 28711
11 Nokia Services (WAP) Browser 23403
11 BlackBerry 16700
12 Mobile Safari 15686143
12 Chrome Mobile 10817867
12 Android 4276783
12 Other 2454413
12 Opera Mini 1412342
12 Chrome 1170379
12 IE Mobile 845740
12 Chrome Mobile iOS 806681
12 UC Browser 313527
12 Firefox Mobile 205955
12 IE 185790
12 BlackBerry WebKit 173916
12 Amazon Silk 158612
12 Opera Mobile 120208
12 Firefox 84299
12 NetFront NX 71344
12 Yandex Browser 61163
12 UP.Browser 45226
12 NetFront 41501
12 bingbot 39173
12 Nokia Browser 34482
12 Opera 27024
12 Nokia Services (WAP) Browser 23988
12 Safari 23434
12 BlackBerry 18223
13 Mobile Safari 13399404
13 Chrome Mobile 9569999
13 Android 3763515
13 Other 2489720
13 Opera Mini 1358259
13 Chrome 959016
13 IE Mobile 737357
13 Chrome Mobile iOS 692172
13 UC Browser 287651
13 IE 189446
13 Firefox Mobile 174859
13 BlackBerry WebKit 160102
13 Amazon Silk 132538
13 Opera Mobile 109905
13 Firefox 73997
13 NetFront NX 65558
13 Yandex Browser 53710
13 UP.Browser 44708
13 NetFront 36526
13 Nokia Browser 30000
13 Opera 27932
13 Nokia Services (WAP) Browser 23985
13 Safari 20944
13 BlackBerry 17992
13 bingbot 11291
14 Mobile Safari 13232133
14 Chrome Mobile 9536148
14 Android 3800470
14 Other 2494306
14 Opera Mini 1355835
14 Chrome 964733
14 IE Mobile 734601
14 Chrome Mobile iOS 680583
14 UC Browser 284872
14 IE 194704
14 Firefox Mobile 172817
14 BlackBerry WebKit 153291
14 Amazon Silk 123805
14 Opera Mobile 105970
14 Firefox 69767
14 NetFront NX 62671
14 Yandex Browser 52945
14 UP.Browser 47034
14 NetFront 36542
14 Nokia Browser 30271
14 Opera 25178
14 Nokia Services (WAP) Browser 24734
14 Safari 20192
14 BlackBerry 18818
14 bingbot 12905
14 Dolfin 11430

Referer

It should be sufficient to just look at four distinct hours from two days for the mobiledot/wapdot case because inbound links independent of search engines would be pretty evident pretty fast.

As it turned out there were 29 distinct external Referer host value for the first days' hours being reviewed and there were 39 distinct external Referer host values for the second days' hours being reviewed. The top ten external Referer hostnames are listed below along with the counts for "unknown" (e.g., bookmark, start page, or Referer-suppressed scheme:// changes) and "internal" referrals.

hive> select month, day, ref, count(*) from
(select month, day,
case when referer_class = 'external' then parse_url(referer,'HOST') else referer_class end as ref
from
wmf.webrequest where
year = 2015 and (month = 6 or month = 7) and day = 5
and (hour = 1 or hour = 7 or hour = 13 or hour = 19)
and access_method = 'mobile web'
and (uri_host like '%.wap.wikipedia.org' OR uri_host like '%.mobile.wikipedia.org')
and is_pageview = true and agent_type = 'user') r
group by month, day, ref;

month day ref count
6 5 unknown 1411
6 5 internal 974
6 5 m.baidu.com 120
6 5 www.baidu.com 44
6 5 m.sp.sm.cn 39
6 5 www.sogou.com 23
6 5 m5.baidu.com 7
6 5 wap.sogou.com 7
6 5 image.baidu.com 6
6 5 r.search.yahoo.com 6
6 5 www.bioboard.de 6
6 5 www.google.com 4
7 5 internal 2647
7 5 unknown 1068
7 5 m.baidu.com 521
7 5 www.baidu.com 308
7 5 m.sp.sm.cn 62
7 5 www.sogou.com 35
7 5 www.bing.com 28
7 5 wap.baidu.com 25
7 5 m5.baidu.com 14
7 5 www.google.com.tw 9
7 5 m.sogou.com 7
7 5 www.google.com 7

BBlack added a comment.Jul 16 2015, 10:38 PM

Phabricator pro-tip: You can put long lists in quoted code blocks with a lines=NN at the top and they'll scroll. For example, "View Raw" on this comment:

day	family	count
1	Mobile Safari	12132057
1	Chrome Mobile	8801804
1	Android	3749250
1	Other	2528518
1	Opera Mini	1403237
1	Chrome	900893
1	IE Mobile	681078
1	Chrome Mobile iOS	621432
1	BingPreview	449956
1	UC Browser	297750
1	IE	204187
1	Firefox Mobile	178981
1	BlackBerry WebKit	152236
1	Amazon Silk	118894
1	Opera Mobile	102316
1	NetFront NX	64352
1	Firefox	53725
1	Yandex Browser	50416
1	UP.Browser	40531
1	NetFront	37946
1	Nokia Browser	29383
1	bingbot	28545
1	Nokia Services (WAP) Browser	24726
1	Opera	22199
1	Safari	19415
1	BlackBerry	18870
2	Mobile Safari	12004029
2	Chrome Mobile	8784320
2	Android	3744649
2	Other	2524019
2	Opera Mini	1376195
2	Chrome	866146
2	IE Mobile	691822
2	Chrome Mobile iOS	612938
2	UC Browser	294477
2	BingPreview	250466
2	IE	189961
2	Firefox Mobile	162782
2	BlackBerry WebKit	145485
2	Amazon Silk	115699
2	Opera Mobile	105234
2	NetFront NX	62323
2	Firefox	55917
2	Yandex Browser	52083
2	bingbot	43918
2	UP.Browser	40129
2	NetFront	35811
2	Nokia Browser	29375
2	Nokia Services (WAP) Browser	23477
2	Opera	21882
2	Safari	19424
3	Mobile Safari	12138382
3	Chrome Mobile	8737983
3	Android	3661523
3	Other	2496413
3	Opera Mini	1331507
3	Chrome	850364
3	IE Mobile	681066
3	Chrome Mobile iOS	626966
3	BingPreview	289122
3	UC Browser	279870
3	IE	168306
3	Firefox Mobile	162661
3	BlackBerry WebKit	143762
3	Amazon Silk	123083
3	Opera Mobile	102074
3	NetFront NX	60822
3	Firefox	53944
3	Yandex Browser	51476
3	bingbot	48486
3	UP.Browser	40755
3	NetFront	35489
3	Nokia Browser	29147
3	Nokia Services (WAP) Browser	22745
3	Opera	21359
3	BlackBerry	17753
4	Mobile Safari	13730159
4	Chrome Mobile	9532199
4	Android	4022570
4	Other	2517782
4	Opera Mini	1345187
4	Chrome	964526
4	IE Mobile	741965
4	Chrome Mobile iOS	702966
4	BingPreview	468216
4	UC Browser	286850
4	Firefox Mobile	182468
4	IE	168662
4	BlackBerry WebKit	154417
4	Amazon Silk	141286
4	Opera Mobile	108665
4	NetFront NX	68877
4	Firefox	58791
4	Yandex Browser	54158
4	bingbot	44750
4	UP.Browser	41254
4	NetFront	37933
4	Nokia Browser	29757
4	Opera	23560
4	Nokia Services (WAP) Browser	22688
4	Safari	17528
5	Mobile Safari	14771801
5	Chrome Mobile	10252627
5	Android	4290877
5	Other	2497504
5	Opera Mini	1430023
5	Chrome	1096304
5	IE Mobile	803601
5	Chrome Mobile iOS	783147
5	UC Browser	310218
5	Firefox Mobile	196526
5	IE	183440
5	BlackBerry WebKit	168692
5	Amazon Silk	153276
5	BingPreview	140881
5	Opera Mobile	116913
5	Firefox	85277
5	NetFront NX	70825
5	Yandex Browser	58552
5	bingbot	48201
5	UP.Browser	42779
5	NetFront	40441
5	Nokia Browser	31471
5	Opera	27185
5	Nokia Services (WAP) Browser	23803
5	Safari	21766
6	Mobile Safari	14228044
6	Chrome Mobile	9753676
6	Android	3929643
6	Other	2532060
6	Opera Mini	1375394
6	Chrome	997612
6	IE Mobile	734240
6	Chrome Mobile iOS	731946
6	UC Browser	290711
6	BingPreview	284861
6	IE	178792
6	Firefox Mobile	172885
6	BlackBerry WebKit	157233
6	Amazon Silk	139901
6	Opera Mobile	109337
6	Firefox	81048
6	NetFront NX	64400
6	Yandex Browser	54215
6	UP.Browser	43042
6	bingbot	40085
6	NetFront	36980
6	Nokia Browser	29496
6	Opera	24721
6	Safari	24390
6	Nokia Services (WAP) Browser	24015
7	Mobile Safari	12806729
7	Chrome Mobile	9304625
7	Android	3815258
7	Other	2541857
7	Opera Mini	1383079
7	Chrome	994497
7	IE Mobile	706593
7	Chrome Mobile iOS	655893
7	UC Browser	287160
7	BingPreview	259074
7	IE	176418
7	Firefox Mobile	168050
7	BlackBerry WebKit	152605
7	Amazon Silk	126040
7	Opera Mobile	104946
7	Firefox	81895
7	bingbot	60863
7	NetFront NX	60185
7	Yandex Browser	53208
7	UP.Browser	43063
7	NetFront	37451
7	Nokia Browser	29615
7	Safari	24805
7	Nokia Services (WAP) Browser	24232
7	Opera	23075
8	Mobile Safari	12659503
8	Chrome Mobile	9184525
8	Android	3764545
8	Other	2495933
8	Opera Mini	1375041
8	Chrome	1020657
8	IE Mobile	696917
8	Chrome Mobile iOS	643371
8	UC Browser	283650
8	IE	188007
8	Firefox Mobile	168692
8	BlackBerry WebKit	155079
8	Amazon Silk	122635
8	Firefox	117103
8	Opera Mobile	106414
8	NetFront NX	57835
8	Yandex Browser	53029
8	UP.Browser	43942
8	NetFront	36813
8	bingbot	33094
8	Safari	30213
8	Nokia Browser	29398
8	Opera	27378
8	Nokia Services (WAP) Browser	24380
8	BlackBerry	18825
9	Mobile Safari	12354742
9	Chrome Mobile	9064487
9	Android	3696875
9	Other	2486943
9	Opera Mini	1357777
9	Chrome	914434
9	IE Mobile	695167
9	Chrome Mobile iOS	628220
9	UC Browser	289429
9	IE	184369
9	Firefox Mobile	163949
9	BlackBerry WebKit	150928
9	Amazon Silk	117744
9	Opera Mobile	106225
9	Firefox	70187
9	NetFront NX	60153
9	Yandex Browser	51033
9	UP.Browser	45109
9	NetFront	37170
9	Nokia Browser	29762
9	Nokia Services (WAP) Browser	24050
9	Opera	23910
9	Safari	20247
9	BlackBerry	18461
9	Dolfin	11563
10	Mobile Safari	12316817
10	Chrome Mobile	8919801
10	Android	3617929
10	Other	2488354
10	Opera Mini	1288886
10	Chrome	880277
10	IE Mobile	679295
10	Chrome Mobile iOS	629070
10	UC Browser	286555
10	IE	194987
10	Firefox Mobile	165586
10	BlackBerry WebKit	148878
10	Amazon Silk	120113
10	Opera Mobile	103405
10	Firefox	67870
10	NetFront NX	60565
10	Yandex Browser	51304
10	UP.Browser	43992
10	NetFront	36353
10	Nokia Browser	29745
10	Opera	24687
10	Nokia Services (WAP) Browser	22633
10	Safari	19044
10	BlackBerry	17754
10	Dolfin	10840
11	Mobile Safari	13795165
11	Chrome Mobile	9607690
11	Android	3860025
11	Other	2465461
11	Opera Mini	1349373
11	Chrome	967251
11	IE Mobile	744211
11	Chrome Mobile iOS	718981
11	UC Browser	295544
11	Firefox Mobile	184472
11	IE	181590
11	BlackBerry WebKit	167669
11	Amazon Silk	139257
11	Opera Mobile	109767
11	Firefox	70353
11	NetFront NX	64651
11	Yandex Browser	55602
11	UP.Browser	44536
11	bingbot	42287
11	NetFront	37872
11	Safari	30587
11	Nokia Browser	29739
11	Opera	28711
11	Nokia Services (WAP) Browser	23403
11	BlackBerry	16700
12	Mobile Safari	15686143
12	Chrome Mobile	10817867
12	Android	4276783
12	Other	2454413
12	Opera Mini	1412342
12	Chrome	1170379
12	IE Mobile	845740
12	Chrome Mobile iOS	806681
12	UC Browser	313527
12	Firefox Mobile	205955
12	IE	185790
12	BlackBerry WebKit	173916
12	Amazon Silk	158612
12	Opera Mobile	120208
12	Firefox	84299
12	NetFront NX	71344
12	Yandex Browser	61163
12	UP.Browser	45226
12	NetFront	41501
12	bingbot	39173
12	Nokia Browser	34482
12	Opera	27024
12	Nokia Services (WAP) Browser	23988
12	Safari	23434
12	BlackBerry	18223
13	Mobile Safari	13399404
13	Chrome Mobile	9569999
13	Android	3763515
13	Other	2489720
13	Opera Mini	1358259
13	Chrome	959016
13	IE Mobile	737357
13	Chrome Mobile iOS	692172
13	UC Browser	287651
13	IE	189446
13	Firefox Mobile	174859
13	BlackBerry WebKit	160102
13	Amazon Silk	132538
13	Opera Mobile	109905
13	Firefox	73997
13	NetFront NX	65558
13	Yandex Browser	53710
13	UP.Browser	44708
13	NetFront	36526
13	Nokia Browser	30000
13	Opera	27932
13	Nokia Services (WAP) Browser	23985
13	Safari	20944
13	BlackBerry	17992
13	bingbot	11291
14	Mobile Safari	13232133
14	Chrome Mobile	9536148
14	Android	3800470
14	Other	2494306
14	Opera Mini	1355835
14	Chrome	964733
14	IE Mobile	734601
14	Chrome Mobile iOS	680583
14	UC Browser	284872
14	IE	194704
14	Firefox Mobile	172817
14	BlackBerry WebKit	153291
14	Amazon Silk	123805
14	Opera Mobile	105970
14	Firefox	69767
14	NetFront NX	62671
14	Yandex Browser	52945
14	UP.Browser	47034
14	NetFront	36542
14	Nokia Browser	30271
14	Opera	25178
14	Nokia Services (WAP) Browser	24734
14	Safari	20192
14	BlackBerry	18818
14	bingbot	12905
14	Dolfin	11430
• demon subscribed.Jul 17 2015, 12:08 AM

Even more pro-tip: put it in a pastebin and then embed it with {P123}. Like this:

P123 old e-mail about dual booting woes
1Hey <snip>,
2
3You're a Mac guy, and are generally tech savvy in things I am not, and I am
4faced with a mild conundrum. This is super low priority and is something I'm
5only trying to resolve in my spare time anyway, so don't feel any urgent need
6to answer this before you have a chance. I thank you in advance for any
7advice you can give by the way.
8
9Backstory: I have an Intel iMac running Leopard on an SATA drive. About 2
10years ago, I decided to dual-boot Ubuntu and OSX off that same drive. In the
11process of doing so, I converted my GPT -> MBR (I was having some
12difficulties in getting Ubuntu to work off the GPT). I ended up using rEFIt
13instead of Grub since it played nicer with OSX's HFS+ partition. I have since
14gotten rid of the Ubuntu install and given its space back to OSX's HFS+
15partition.
16
17I want to upgrade OSX to Snow Leopard on this box, but every time I try--
18either through the Launcher or booting to the upgrade disc--it won't let me
19because "Mac OS X cannot be installed on Macintosh HD, because this disk
20cannot be used to start up your computer." From Googling around, I quickly
21determined this was because OSX cannot be installed unless GPT is being
22used--despite the fact that OSX will boot from a MBR just fine.
23
24So began my saga of attempting to rid myself of my MBR. I eventually found
25that gdisk will do this for me. After some wrangling there[0], I managed to
26convert my MBR back into a legitimate GPT without wiping my disk \o/
27Attempted to install Snow Leopard again, same failure. I then found that the
28MBR that GPT keeps in a separate sector for back-compat reasons was out
29of sync. According to [1], this could also be causing my problem. I came across
30gptsync, which very easily re-sync'd the two boot formats. Still no luck on
31installing. The current output of my partition info (gptsync & fdisk -d) is [2].
32
33I am at a complete loss at this point, and anything you can point me to would
34help. I'd really really like to not wipe the disk if I can avoid it (I've gotten this far
35with all my data intact). If you have no f'ing clue, that's ok too! Feel free to pass
36me off if you can think of someone who knows.
37
38Thanks!!
39
40-Chad
41
42
43[0] http://www.rodsbooks.com/gdisk/mbr2gpt.html
44[1] http://reviews.cnet.com/8301-13727_7-10330520-263.html
45[2] http://p.defau.lt/?G1ePYx4u2KjlKQW04ny_Xw

Then you can reuse it in as many tasks as you want!

dr0ptp4kt moved this task from Doing to Waiting on the Reading-Admin board.Jul 17 2015, 8:40 PM
• Tnegrin added a comment.Jul 17 2015, 8:49 PM

This has been an enlightening thread in many ways! Thanks everybody.

I'm not seeing anything in the longer queries that causes any alarm and I'd like to move this forward. Any additional concerns?

-Toby

dr0ptp4kt added a comment.Jul 24 2015, 11:18 PM

I pinged on wikitech-l one last time.

BBlack added a comment.Jul 27 2015, 12:48 PM

wikitech-l thread seems in agreement as well, so moving forward on this. Thanks for looking into it everyone :)

gerritbot added a comment.Jul 27 2015, 12:49 PM

Change 223972 merged by BBlack:
Remove wap and mobile subdomains

https://gerrit.wikimedia.org/r/223972

BBlack closed this task as Resolved.Jul 27 2015, 12:51 PM
BBlack claimed this task.
BBlack moved this task from Upcoming to Done on the Traffic board.
BBlack mentioned this in rODNS3bdae4c2f525: Remove wap and mobile subdomains.Jul 27 2015, 12:52 PM
BBlack removed a parent task: T102824: Clean up DNS/redirects for TLS.Jul 27 2015, 1:06 PM
BBlack added a parent task: T102814: Decom old multiple-subdomain wikis in wikipedia.org.
BBlack removed a parent task: T104244: Preload HSTS.Jul 27 2015, 1:13 PM
gerritbot added a comment.Aug 14 2015, 6:44 PM

Change 231610 had a related patch set uploaded (by BBlack):
Remove mobile.wp.o

https://gerrit.wikimedia.org/r/231610

gerritbot added a comment.Aug 14 2015, 6:47 PM

Change 231610 merged by BBlack:
Remove mobile.wp.o

https://gerrit.wikimedia.org/r/231610

BBlack mentioned this in rODNSe9449685e3b7: Remove mobile.wp.o.Aug 14 2015, 6:47 PM
gerritbot added a comment.Aug 14 2015, 6:51 PM

Change 231612 had a related patch set uploaded (by BBlack):
Remove wap/mobile subdomains from redirects

https://gerrit.wikimedia.org/r/231612

gerritbot added a comment.Aug 14 2015, 6:52 PM

Change 231612 merged by BBlack:
Remove wap/mobile subdomains from redirects

https://gerrit.wikimedia.org/r/231612

BBlack mentioned this in rOPUPd8539578e8a6: Remove wap/mobile subdomains from redirects.Aug 14 2015, 6:52 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL