Page MenuHomePhabricator

Create a secure redirect service for large count of non-canonical / junk domains
Closed, ResolvedPublic

Description

Given recent progress on production puppetization of LetsEncrypt.org (LE), and LE itself improving in general in recent months (moved from beta to production status, has proved itself a bit, ratelimits are reasonable, etc), I think we can now really contemplate the idea of doing a secure redirector service to cover large counts of junk domains. We talked this out a bit on IRC, and AFAICS there's now no real technical blockers to making this happen; we'll probably be able to handle hundreds of one-off domainnames for this through LE mechanisms.

One noteable tradeoff is it will have to be an SNI-dependent service for the bulk of the names. That means many of these secure redirects will not work for certain older browsers (notably IE[78]-on-XP, Android 2.x, and some very old feature phones like Symbian and Blackberry). Given the alternative is to dead-park (no browser functionality or at least no true redirect) the bulk of these domains, the SNI limitation is probably acceptable, and we can certainly arrange the certificate sets such that the highest-value ones are on the default SNI server for greater compatibility than the rest.

What it basically boils down to now is:

  • Decide on a reasonable SAN list length limit per cert: 100
  • Prioritize which "junk" domains should be in the primary (works for non-SNI) SAN list
  • Puppetize a service role built around modules/nginx + acme-chief that can redirect a configured large set of domainnames securely.
  • Assign a new public IP for this in eqiad + codfw LVS ranges.
  • Deploy this service in eqiad + codfw (possibly on virtual hosts as the load should be fairly light). Probably manual gdnsd inter-DC failover at least initially until we sort out x-dc LE-cert issues.

Details

SubjectRepoBranchLines +/-
operations/dnsmaster+1 -45
operations/puppetproduction+3 -8
operations/puppetproduction+102 -102
operations/puppetproduction+2 -156
operations/dnsmaster+1 -0
operations/dnsmaster+11 -11
operations/puppetproduction+28 -5
operations/puppetproduction+50 -1
operations/dnsmaster+1 -1
operations/dnsmaster+1 -1
operations/dnsmaster+2 -2
operations/dnsmaster+4 -4
operations/dnsmaster+3 -3
operations/dnsmaster+3 -3
operations/dnsmaster+4 -4
operations/dnsmaster+2 -2
operations/dnsmaster+7 -7
operations/dnsmaster+3 -3
operations/dnsmaster+3 -4
operations/dnsmaster+17 -17
operations/puppetproduction+32 -0
operations/puppetproduction+50 -0
operations/dnsmaster+12 -11
operations/dnsmaster+2 -2
operations/puppetproduction+4 -4
operations/dnsmaster+6 -6
operations/puppetproduction+6 -6
operations/puppetproduction+380 -326
operations/dnsmaster+5 -0
operations/dnsmaster+4 -4
operations/puppetproduction+5 -5
operations/dnsmaster+14 -14
operations/dnsmaster+2 -0
operations/puppetproduction+1 -1
operations/puppetproduction+1 -1
operations/puppetproduction+1 -1
operations/puppetproduction+5 -14
operations/puppetproduction+2 -2
operations/puppetproduction+3 -0
operations/puppetproduction+1 -0
operations/dnsmaster+1 -1
operations/puppetproduction+78 -0
operations/dnsmaster+79 -0
operations/dnsmaster+16 -15
operations/dnsmaster+20 -0
operations/puppetproduction+2 -2
operations/puppetproduction+12 -12
operations/puppetproduction+11 -1
operations/puppetproduction+14 -16
operations/dnsmaster+9 -1
operations/puppetproduction+12 -6
operations/puppetproduction+10 -0
operations/puppetproduction+12 -1
operations/puppetproduction+1 -0
operations/puppetproduction+11 -0
operations/puppetproduction+1 -0
operations/puppetproduction+8 -4
operations/puppetproduction+386 -0
operations/puppetproduction+177 -165
operations/puppetproduction+10 -0
operations/dnsmaster+8 -0
operations/puppetproduction+0 -7
operations/puppetproduction+15 -0
operations/puppetproduction+127 -0
Show related patches Customize query in gerrit

Related Objects

StatusSubtypeAssignedTask
Resolved ema
DeclinedNone
OpenNone
ResolvedBBlack
ResolvedVgutierrez
ResolvedVgutierrez
ResolvedVgutierrez
ResolvedBBlack
ResolvedNone
ResolvedNone
DuplicateNone
ResolvedKrenair
ResolvedBBlack
ResolvedMarcoAurelio
ResolvedKrenair
Resolvedscfc
ResolvedVgutierrez
ResolvedVgutierrez
ResolvedVgutierrez
OpenVgutierrez
ResolvedVgutierrez

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Change 522055 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] lvs: Add ncredir service to high-traffic1

https://gerrit.wikimedia.org/r/522055

Change 521414 merged by Vgutierrez:
[operations/dns@master] Add ncredir-lb records

https://gerrit.wikimedia.org/r/521414

Change 523106 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/dns@master] Split langlist helper in two

https://gerrit.wikimedia.org/r/523106

Change 523114 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/dns@master] Add a ncredir-parking zone

https://gerrit.wikimedia.org/r/523114

Change 523115 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/dns@master] Switch wikipedia.com to the ncredir-parking DNS zonefile

https://gerrit.wikimedia.org/r/523115

Change 523106 merged by Vgutierrez:
[operations/dns@master] Split langlist helper in two

https://gerrit.wikimedia.org/r/523106

Change 523114 merged by Vgutierrez:
[operations/dns@master] Add a ncredir-parking zone

https://gerrit.wikimedia.org/r/523114

Change 522055 merged by Vgutierrez:
[operations/puppet@production] lvs: Add ncredir service to high-traffic1

https://gerrit.wikimedia.org/r/522055

Change 523115 merged by Vgutierrez:
[operations/dns@master] Switch wikipedia.com to the ncredir-parking DNS zonefile

https://gerrit.wikimedia.org/r/523115

Change 523676 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] ncredir: Enable monitoring

https://gerrit.wikimedia.org/r/523676

Change 523676 merged by Vgutierrez:
[operations/puppet@production] ncredir: Enable monitoring

https://gerrit.wikimedia.org/r/523676

Change 523680 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] cumin: Add ncredir aliases

https://gerrit.wikimedia.org/r/523680

Change 523680 merged by Vgutierrez:
[operations/puppet@production] cumin: Add ncredir aliases

https://gerrit.wikimedia.org/r/523680

Change 523682 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] lvs: Fix typo on icinga check command definition for ncredir

https://gerrit.wikimedia.org/r/523682

Change 523682 merged by Vgutierrez:
[operations/puppet@production] lvs: Fix typo on icinga check command definition for ncredir

https://gerrit.wikimedia.org/r/523682

Change 523700 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] lvs: Fix icinga checks for ncredir and ncredir-https

https://gerrit.wikimedia.org/r/523700

Change 523700 merged by Vgutierrez:
[operations/puppet@production] lvs: Fix icinga checks for ncredir and ncredir-https

https://gerrit.wikimedia.org/r/523700

ncredir service has been deployed successfully and it's currently serving live traffic for wikipedia.com:

$ curl -v https://en.wikipedia.com/wiki/Special:Random -o /dev/null 2>&1 |fgrep -i location:
< location: https://en.wikipedia.org/wiki/Special:Random
$ curl -v http://en.wikipedia.com/wiki/Special:Random -o /dev/null 2>&1 |fgrep -i location:
< Location: https://en.wikipedia.org/wiki/Special:Random

Change 523877 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] ncredir: Set notes_url for https_ncredir

https://gerrit.wikimedia.org/r/523877

Change 523878 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] lvs: Enable paging for ncredir checks

https://gerrit.wikimedia.org/r/523878

Change 523877 merged by Vgutierrez:
[operations/puppet@production] ncredir: Set notes_url for https_ncredir

https://gerrit.wikimedia.org/r/523877

Change 523878 merged by Vgutierrez:
[operations/puppet@production] lvs: Enable paging for ncredir checks

https://gerrit.wikimedia.org/r/523878

Change 523888 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] ncredir: Fix notes_url

https://gerrit.wikimedia.org/r/523888

Change 523888 merged by Vgutierrez:
[operations/puppet@production] ncredir: Fix notes_url

https://gerrit.wikimedia.org/r/523888

Change 523902 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/dns@master] Redirect already configured wikipedia non canonical domains to ncredir

https://gerrit.wikimedia.org/r/523902

Change 523924 had a related patch set uploaded (by BBlack; owner: BBlack):
[operations/dns@master] Add domain root addrs for ncredir

https://gerrit.wikimedia.org/r/523924

Change 523924 merged by Vgutierrez:
[operations/dns@master] Add domain root addrs for ncredir

https://gerrit.wikimedia.org/r/523924

Change 523902 merged by Vgutierrez:
[operations/dns@master] Redirect already configured wikipedia non canonical domains to ncredir

https://gerrit.wikimedia.org/r/523902

Change 524092 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] nc_redirects.dat: Reenable rules for non-canonical wikipedia.org domains

https://gerrit.wikimedia.org/r/524092

Change 524093 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/dns@master] Point several wikipedia non-canonical domains to ncredir-parking

https://gerrit.wikimedia.org/r/524093

Change 524092 merged by Vgutierrez:
[operations/puppet@production] nc_redirects.dat: Reenable rules for non-canonical wikipedia.org domains

https://gerrit.wikimedia.org/r/524092

Change 524093 merged by Vgutierrez:
[operations/dns@master] Point several wikipedia non-canonical domains to ncredir-parking

https://gerrit.wikimedia.org/r/524093

Change 295249 abandoned by BBlack:
ncredir hostname and service IP

Reason:
Better things were done in the interim!

https://gerrit.wikimedia.org/r/295249

Change 292785 abandoned by BBlack:
redirects.dat - split non-canonical to separate section

Reason:
Better stuff done recently!

https://gerrit.wikimedia.org/r/292785

Change 528316 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/dns@master] ncredir: Let ncredir take over wikimedia.com and linked DNS zones

https://gerrit.wikimedia.org/r/528316

Change 528320 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/dns@master] Let ncredir take care of wikimediacommons non canonical domains

https://gerrit.wikimedia.org/r/528320

Change 528321 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] nc_redirects.dat: Re-enable wikimediacommons rules

https://gerrit.wikimedia.org/r/528321

Change 528321 merged by Vgutierrez:
[operations/puppet@production] nc_redirects.dat: Re-enable wikimediacommons rules

https://gerrit.wikimedia.org/r/528321

Change 528320 merged by Vgutierrez:
[operations/dns@master] Let ncredir take care of wikimediacommons non canonical domains

https://gerrit.wikimedia.org/r/528320

Change 528345 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] nc_redirects.dat: Re-enable voyagewiki.(org|com) rules

https://gerrit.wikimedia.org/r/528345

Change 528368 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/dns@master] Let ncredir take care of voyagewiki.com and voyagewiki.org

https://gerrit.wikimedia.org/r/528368

Change 528345 merged by Vgutierrez:
[operations/puppet@production] nc_redirects.dat: Re-enable voyagewiki.(org|com) rules

https://gerrit.wikimedia.org/r/528345

Change 528368 merged by Vgutierrez:
[operations/dns@master] Let ncredir take care of voyagewiki.com and voyagewiki.org

https://gerrit.wikimedia.org/r/528368

Change 528394 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/dns@master] ncredir: Handle more domains to the non canonical redirect service

https://gerrit.wikimedia.org/r/528394

Change 528397 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] ncredir: Introduce non-canonical-redirect-5

https://gerrit.wikimedia.org/r/528397

Change 528398 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] nc_redirects.dat: Add rules to support non-canonical-redirect-5

https://gerrit.wikimedia.org/r/528398

Change 528316 merged by Vgutierrez:
[operations/dns@master] ncredir: Let ncredir take over wikimedia.com and linked DNS zones

https://gerrit.wikimedia.org/r/528316

Change 528397 merged by Vgutierrez:
[operations/puppet@production] ncredir: Introduce non-canonical-redirect-5

https://gerrit.wikimedia.org/r/528397

Change 528398 merged by Vgutierrez:
[operations/puppet@production] nc_redirects.dat: Add rules to support non-canonical-redirect-5

https://gerrit.wikimedia.org/r/528398

Change 528394 merged by Vgutierrez:
[operations/dns@master] ncredir: Handle more domains to the non canonical redirect service

https://gerrit.wikimedia.org/r/528394

Change 532870 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/dns@master] Update wikisource non canonical domains

https://gerrit.wikimedia.org/r/532870

Change 532871 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/dns@master] Update wiktionary non canonical domains

https://gerrit.wikimedia.org/r/532871

Change 532872 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/dns@master] Update wikivoyage non canonical domains

https://gerrit.wikimedia.org/r/532872

Change 532873 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/dns@master] Update wikiversity non canonical domains

https://gerrit.wikimedia.org/r/532873

Change 532874 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/dns@master] Update wikiquote non canonical domains

https://gerrit.wikimedia.org/r/532874

Change 532875 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/dns@master] Update wikinews non canonical domains

https://gerrit.wikimedia.org/r/532875

Change 532876 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/dns@master] Update wikibooks non canonical domains

https://gerrit.wikimedia.org/r/532876

Change 532879 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/dns@master] Update wikimediafoundation non canonical domains

https://gerrit.wikimedia.org/r/532879

Change 532880 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/dns@master] Update wikipedia non canonical domains

https://gerrit.wikimedia.org/r/532880

Change 532881 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/dns@master] Redirect wikimania.com to the non canonical redirect service

https://gerrit.wikimedia.org/r/532881

Change 532882 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/dns@master] Redirect mediawiki.com to the non canonical redirect service

https://gerrit.wikimedia.org/r/532882

Change 532870 merged by Vgutierrez:
[operations/dns@master] Update wikisource non canonical domains

https://gerrit.wikimedia.org/r/532870

Change 532871 merged by Vgutierrez:
[operations/dns@master] Update wiktionary non canonical domains

https://gerrit.wikimedia.org/r/532871

Change 532872 merged by Vgutierrez:
[operations/dns@master] Update wikivoyage non canonical domains

https://gerrit.wikimedia.org/r/532872

Change 532949 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/dns@master] Feed more parked domains to the non canonical redirect service

https://gerrit.wikimedia.org/r/532949

Change 532950 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] ncredir: Add non-canonical-redirect-6

https://gerrit.wikimedia.org/r/532950

Change 532951 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] ncredir: Add redirection rules for domains added in non-canonical-cert-6

https://gerrit.wikimedia.org/r/532951

Change 532873 merged by Vgutierrez:
[operations/dns@master] Update wikiversity non canonical domains

https://gerrit.wikimedia.org/r/532873

Change 532874 merged by Vgutierrez:
[operations/dns@master] Update wikiquote non canonical domains

https://gerrit.wikimedia.org/r/532874

Change 532875 merged by Vgutierrez:
[operations/dns@master] Update wikinews non canonical domains

https://gerrit.wikimedia.org/r/532875

Change 532876 merged by Vgutierrez:
[operations/dns@master] Update wikibooks non canonical domains

https://gerrit.wikimedia.org/r/532876

Change 532879 merged by Vgutierrez:
[operations/dns@master] Update wikimediafoundation non canonical domains

https://gerrit.wikimedia.org/r/532879

Change 532880 merged by Vgutierrez:
[operations/dns@master] Update wikipedia non canonical domains

https://gerrit.wikimedia.org/r/532880

Change 532881 merged by Vgutierrez:
[operations/dns@master] Redirect wikimania.com to the non canonical redirect service

https://gerrit.wikimedia.org/r/532881

Change 532882 merged by Vgutierrez:
[operations/dns@master] Redirect mediawiki.com to the non canonical redirect service

https://gerrit.wikimedia.org/r/532882

Change 532950 merged by Vgutierrez:
[operations/puppet@production] ncredir: Add non-canonical-redirect-6

https://gerrit.wikimedia.org/r/532950

Change 532951 merged by Vgutierrez:
[operations/puppet@production] ncredir: Add redirection rules for domains added in non-canonical-cert-6

https://gerrit.wikimedia.org/r/532951

Change 532949 merged by Vgutierrez:
[operations/dns@master] Feed more parked domains to the non canonical redirect service

https://gerrit.wikimedia.org/r/532949

Change 533141 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] redirects.dat: Get rid of non canonical domains rules

https://gerrit.wikimedia.org/r/533141

Change 533142 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] redirects.dat: Enforce HTTPS for canonnical domains

https://gerrit.wikimedia.org/r/533142

Change 533213 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/dns@master] Point wikimania.org to the non canonical redirect service

https://gerrit.wikimedia.org/r/533213

Change 533219 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/dns@master] Point wikimedia.community to the non canonical redirect service

https://gerrit.wikimedia.org/r/533219

Change 533213 merged by BBlack:
[operations/dns@master] Point wikimania.org to the non canonical redirect service

https://gerrit.wikimedia.org/r/533213

Change 533219 merged by BBlack:
[operations/dns@master] Point wikimedia.community to the non canonical redirect service

https://gerrit.wikimedia.org/r/533219

Change 533141 merged by BBlack:
[operations/puppet@production] redirects.dat: Get rid of non canonical domains rules

https://gerrit.wikimedia.org/r/533141

Change 533142 merged by BBlack:
[operations/puppet@production] redirects.dat: Enforce HTTPS for canonical domains

https://gerrit.wikimedia.org/r/533142

Change 550931 had a related patch set uploaded (by BBlack; owner: BBlack):
[operations/puppet@production] VCL: Remove host regex from TLS redirect and STS

https://gerrit.wikimedia.org/r/550931

Change 550931 merged by BBlack:
[operations/puppet@production] VCL: Remove host regex from TLS redirect

https://gerrit.wikimedia.org/r/550931

@Vgutierrez: Hi, all related patches in Gerrit have been merged or abandoned. Is there more to do in this task? Asking as you are set as task assignee. Thanks in advance! (You can change the task status via Add Action...Change Status in the dropdown menu.)

@Vgutierrez: No reply; assuming this is resolved. If not, please reopen.