Page MenuHomePhabricator

Decom www.$lang hostnames/redirects
Closed, ResolvedPublic


We generically create www sub-domainnames for all desktop language domainnames @

None of these match our SSL cert wildcards. They're insecure redirects that we should probably delete from DNS and from apache redirects, but I'd like to get some impact input before just blindly killing them on my own...

Event Timeline

BBlack raised the priority of this task from to Needs Triage.
BBlack updated the task description. (Show Details)
BBlack added projects: acl*sre-team, Traffic, HTTPS.
BBlack added subscribers: BBlack, Aklapper.

Yes, related. This is an actual task though (as in an action to take), rather than just a related problem-statement.

Change 218909 had a related patch set uploaded (by BBlack):
Remove www.$lang DNS T102815

Noted on irc, tons of results in:

01:40 < Mjbmr>

I wonder why google ignores the fact that those redirect and have rel=canonical in the content? Can we get google to get rid of these before we kill them?

Those links seem to be all over the place (especially in print, some people think it's better to put www. in front of every domain to make it clear that it's a website …):

Noted on irc, tons of results in:

01:40 < Mjbmr>

I wonder why google ignores the fact that those redirect and have rel=canonical in the content? Can we get google to get rid of these before we kill them?

Maybe we need to ask Google to re-crawl these URLs: Probably we shouldn't merge, since Google wrote [1]:

Don't use the robots.txt file for canonicalization purposes.


Well, I think maybe we can re-visit the detailed advice given in from above and go audit that we're not doing something obscure wrong, but... AFAIK addresses like have for a long time (perhaps forever) been pure 301's to content with correct rel=canonical, so I think we've been playing by the rules and it's getting us nowhere... Also, there are other search engines to consider as well, for some of which robots may be the only answer.

While taking a quick peek: perhaps not directly related? But I did notice our canonical URLs only seem to canonicalize the protocol and domain, not the rest of the URL. e.g. accessing via and produce different rel=canonical values (re-using the path of the request URL).

Getting rid of them completely would be ideal, but if it turns out to not be possible, why can't we just handle those and 301 them to the proper www-less https URL?

It doesn't change things a whole lot from today (which also redirects them, just to www-less first and then HTTPS after). The core issue is that we can't HSTS the hostname without a valid cert, so those links will always be insecure redirects wherever they exist (in browser history, search results, etc).

@Krinkle said elsewhere:

And by default the www.en.wikipedia entries are not included
I couldn't find any index hits from that come up when looking for the title only. So it seems deduplicated properly. They just expose it when looking for that specific site

If we don't think "real" searches are hitting this, it may be possible to simply remove them, then.

I took a quick 2-minute log of such hits on just one prominent frontend cache (totally statistically invalid, but still...), with the URL-Path, Host-header, and Referer noted, and counted 15 distinct hits, almost all with Yahoo Search as the referer. Extrapolating loosely based on that machine's percentage of total reqs, we're looking at something on the order of 1.25 reqs/sec globally, or 0.003125% of all text-cluster requests.

The 15 logged requests (number on the left is effectively a request-id in this context):

root@cp1065:~# time varnishlog -c -n frontend -m 'RxHeader:Host: www\.en\.wiki' |egrep --line-buffered '(Host:|RxURL|Referer:)'
   86 RxURL        c /wiki/Monica_Potter
   86 RxHeader     c Host:
   86 RxHeader     c Referer:
   52 RxURL        c /wiki/Jurassic_World
   52 RxHeader     c Host:
   52 RxHeader     c Referer:
   28 RxURL        c /wiki/Billy_Redden
   28 RxHeader     c Host:
   28 RxHeader     c Referer:
   46 RxURL        c /wiki/Here_Comes_the_Boom
   46 RxHeader     c Host:
   46 RxHeader     c Referer:
   72 RxURL        c /wiki/What_to_Expect_When_You%27re_Expecting_%28film%29
   72 RxHeader     c Host:
   72 RxHeader     c Referer:
   29 RxURL        c /wiki/Dwayne_Johnson
   29 RxHeader     c Host:
   29 RxHeader     c Referer:
   66 RxURL        c /wiki/Special:RecentChangesLinked/Patsnap
   66 RxHeader     c Host:
   51 RxURL        c /wiki/The_Music_Man_%281962_film%29
   51 RxHeader     c Host:
   51 RxHeader     c Referer:
   70 RxURL        c /wiki/Myrna_Loy
   70 RxHeader     c Referer:
   70 RxHeader     c Host:
   17 RxURL        c /wiki/John_Leguizamo
   17 RxHeader     c Host:
   17 RxHeader     c Referer:
   25 RxURL        c /wiki/Jurassic_Park_III
   25 RxHeader     c Host:
   25 RxHeader     c Referer:
  156 RxURL        c /wiki/Jeff_Goldblum
  156 RxHeader     c Host:
  156 RxHeader     c Referer:
  112 RxURL        c /wiki/Richard_Attenborough
  112 RxHeader     c Host:
  112 RxHeader     c Referer:
   51 RxURL        c /wiki/Media_Lab_(disambiguation)
   51 RxHeader     c Host:
   41 RxURL        c /wiki/Jurassic_World
   41 RxHeader     c Host:
   41 RxHeader     c Referer:

real	2m0.246s
user	0m8.852s
sys	0m0.416s

So, working off of @faidon's per-domain-count used in the donate ticket:

faidon@oxygen:~$ egrep '"www\.[a-z]{2}\.' per-domain-count |head -10
  10750 ""
   4398 ""
   1996 ""
   1779 ""
   1175 ""
    871 ""
    694 ""
    664 ""
    544 ""
    321 ""

In the enwiki case, this amounts to 00.002% of the main domain hits. In the zh case, which seems to be the worst, it's more like 00.16%. Either way, it's not significant traffic, and AFAIK we never link or expose these anywhere, although yahoo seems to occasionally refer users (very low rate).

Basically, I don't know what else we can do here except go ahead and shut these off (patch ref'd earlier). They're as dead as we can make them in terms of exposure already.

BBlack claimed this task.