Correct me if I'm wrong, but I would think all VE traffic would already be uncacheable at the Varnish level anyways, since it happens in the context of a session (although in the future we might fix this with content composition work). As for the rest of this discussion, I don't think I understand the context enough to say anything about its sanity or whether it increases any attack surface in a way that matters.

In T205897#4953769, @faidon wrote:

The medium-term plan is for this data to be entered into Netbox after a server is racked but before it's provisioned or even powered up, and that data to be used by our tooling to configure and execute the provisioning itself (DHCP configuration, switchport, OS install etc.).

So, I don't think we can reasonably expect our on-site techs to look at a box and say "oh this port is enp4s0f0p1" and record it as such :)

In T205897#4950741, @Volans wrote:

In T205897#4950310, @ayounsi wrote:

• How should we name server interfaces? The physical Port 1, Port 2, etc. or the Linux naming (enp5s0f0, enp5s0f1, etc) My vote so far would go with #2. Even if it's harder to parse for a human, it should stay consistent.

My vote is for whatever is clearer/simpler for the people that has to physically interact with them as long as they uniquely identify the parts without ambiguity.

In T215071#4941080, @Tgr wrote:

According to the article Censorship of Wikipedia, one effect of the switch to https was that it is now not possible to censor individual articles.

Conversely, now when China decides to censor articles about Tienanmen Square, their citizens also lose access to basic health information, STEM education background material and all other content that would probably have far more positive impact on their lives than articles about politics... it's a double edged sword. Arguably, strictly from the censorship / access angle, HTTPS was a bad trade-off. (There are a number of other reasons why it was absolutely necessary; but for merging subdomains that's probably not the case.)

In T214529#4936555, @CDanis wrote:

Corrected errors are normal and expected to occur on healthy
hardware. They do not need user's attention until they repeatedly
occurred at a same place.

Apparently, you haven't been on enough maintanance calls, trying to
calm down the customer about the hardware error he sees in his
logs...

Actually, that's why. Reporting all corrected errors make users
worried, call support, and asking to replace healthy hardware...

So it seems possible that nothing is actually 'wrong' here? But I have very little confidence in anything at this point.

Sounds good to me!

Expounding on the lamentations above in a more realistic triage sort of sense:

The linked ESNI ticket is kind of a random user question ticket, and not actually one created for working on it (which still off in the Future, but obviously we'll implement ESNI as soon as we realistically can, as part of planned work).

See also T178011 for last time. Why didn't the icinga EDAC check catch this?

https://wikitech.wikimedia.org/wiki/Cache_servers#Depool_and_downtime is correct, it just needs to be depooled (it will auto-depool on shutdown, but a manual depool is preferable).

In T211254#4902524, @mark wrote:

It may be possible to get more space in various shady ways, but it's not possible by following RIR rules.

In T211254#4902250, @mark wrote:

In T211254#4902223, @BBlack wrote:

It's the same basic rationale as moving WMCS out of 10.68.0.0/16. We could obviously leave them there and just manage our ACLs better with more automation, but it pays some pretty big dividends when address spaces are clearly split on such a big security and functional boundary as Prod-v-WMCS. Humans will always look at IPs as well in various debugging and configuration tasks. Having similar/shared/adjacent numbering for these two realms invites confusion and mistakes.

In a world where there's ample address space (such as 10/8 in our context), yes. In today's world where IPv4 address space is scarce and we can likely not get any more, not so much.

I would personally have preferred that with the renumbering.of WMCS they simply acquired new public IPv4 space of their own

That's simply not realistic, they can't "acquire" IPv4 address space of their own. They're part of this organisation, this ASN, and need to use our PI/PA space where we have it available before we collectively can get more.

It's the same basic rationale as moving WMCS out of 10.68.0.0/16. We could obviously leave them there and just manage our ACLs better with more automation, but it pays some pretty big dividends when address spaces are clearly split on such a big security and functional boundary as Prod-v-WMCS. Humans will always look at IPs as well in various debugging and configuration tasks. Having similar/shared/adjacent numbering for these two realms invites confusion and mistakes.

See also this email thread where Michael Chan (broadcom driver dev) asks for firmware level output, sees the same numbers we have on cp1088, and tells them to upgrade: https://www.spinics.net/lists/netdev/msg519478.html

In T203194#4880054, @Vgutierrez wrote:

on the Dell community forum there is a post with several people reporting the same issue. The last one suggesting that Dell could be replacing hardware to fix the issue.

I don't have any suggestions, no. Develop a straw-patch which at least serves in code terms to document the intent (e.g. the explicit header and URI values matched/transformed, etc) and we'll poke at it!

It's a confusing set of things going on here, and it's going to need fixups on both the network/data/data.yaml side and the VCL side. Just to recap the historical situation for clarity:

Right. I'm not up to speed on where all related changes are, but from VCL's point of view its definition of wikimedia_nets was meant to include labs, whereas its nearly identical wikimedia_trust is meant to exclude labs.

Yeah. It's hard to "prove" whether we have this bug fixed other than running a supposed fix on the bnxt_en cp10 fleet for a while as a statistical test, but probably the sooner we start on that the better.

Actually, it is already in the 4.9.y LTS/stable branch, here: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?h=linux-4.9.y&id=b2be15bb02b961146177d49204de22df3dddd415

I suspect our bug is fixed by:

Update for the record: with recent changes to authdns CI and deployment scripts, this scenario should no longer be possible and workarounds shouldn't be necessary! (see also related distant past incident T103915)

This is fixed now, no workarounds should be needed.

Resolving this, as recent work has fixed a lot of it (other than discovery issues specifically), and at this point all the text above is woefully outdated and pointing in Wrong directions. We can file some smaller-scope tasks about remaining cleanups in this space.

We've done all this and gone way past it at this point. We might tag some future improvements here, but I think it's safe to call the basic task Resolved. Thanks @Volans !

localssl.erb would probably be more appropriate and is the site file, but it's a generic TLS reverse proxy setup with a few parameters, and there's not yet any "this is for Traffic termination" flag (classparam?) puppetized, but you could make one.

I don't know off-hand if we can live without them all for manual debugging and such, or if nginx is the best place to remove them. There might be other ways to suppress them in Varnish without affecting analytics, but that needs some digging and testing. We can't universally strip them in tlsproxy's nginx.conf like the patch above, though, because that config is also re-used for various applayer TLS termination behind varnish as well, and thus would hide any of those headers that are being sent from the applayer up to varnish itself.

In T167972#4837684, @mobrovac wrote:

In T167972#4837630, @BBlack wrote:

I thought this was in a different ticket somewhere at one point, but in any case I just noticed it during someone's meta-updates, and the description seems slightly off. What we really want here is full alignment of the public and private API URIs: If the public is hitting https://en.wikipedia.org/api/rest_v1/foo, that should be what RB accepts as well (in Host: + URI terms). The description seems to leave out the /api/ part of it. Or in other words, make it unecessary for Varnish to do the transformation it does here: https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/production/modules/varnish/templates/text-backend.inc.vcl.erb#38 .

Oh you mean have RB behave exactly like the app servers, whereby RB should be able to accept by default requests to http://restbase.discovery.wmnet:7231/api/rest_v1/{path} with the Host: {domain} header set and automatically transform that itself to http://restbase.discovery.wmnet:7231/{domain}/v1/{path}? That is doable, but what would be the advantage given that Varnish strips the Host header?

I thought this was in a different ticket somewhere at one point, but in any case I just noticed it during someone's meta-updates, and the description seems slightly off. What we really want here is full alignment of the public and private API URIs: If the public is hitting https://en.wikipedia.org/api/rest_v1/foo, that should be what RB accepts as well (in Host: + URI terms). The description seems to leave out the /api/ part of it. Or in other words, make it unecessary for Varnish to do the transformation it does here: https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/production/modules/varnish/templates/text-backend.inc.vcl.erb#38 .

There's still a couple of things that can be done serially at present, one of which is necessary for the cert issuance later:

Tag edit because all of those are specific to WMF Ops and this ticket isn't!

BTW: https://gerrit.wikimedia.org/r/c/operations/dns/+/462693 is a good test job when it's flipped. This fails current linting because of the outdated gdnsd version there, but hypothetically should pass on the new Docker-based stuff with updated software.

So I see @Joe has merged up some Dockerfile stuff. What's our next step to flip operations/dns CI checks over to the new operations-dnslint? AFAIK we're ready for this at any time (current repo passes under the new checks and they're ready to use).

Some interesting stuff here (see also the Mailing Lists link there in the datatracker for discussion): https://datatracker.ietf.org/doc/draft-moura-dnsop-authoritative-recommendations/?include_text=1

^ Fixing it to be self-explanatory! :)

In T205439#4816740, @hashar wrote:

Out of curiosity: how do you ship the GeoDNS database? Is that relying on a package available through Debian?

Done, resolve?

@hashar - So where we're at now is that we just need our CI switched to a Docker with the following properties (which is probably simple, but non-obvious to me!):

@hashar - I'm re-working the tools for the linting checks on operations/dns in the commits linked above, and we should be able to get away from cloning/using operations/puppet completely and just run a few simple commands on a checkout of operations/dns from a Docker image. I'm sure we can add the trivial tab-checking into the main CI run as well.

No new EDAC errors reported since repooling, all we can do is assume it's ok for now I think.

Fixed again. Copying my whole terminal output for posterity. This runs a readonly command that md5sum's the zones directory to check whether all servers have the same exact zone data, then runs the same regeneration command that fixed them before, then confirms the hashes are aligned now:

bblack@cumin1001:~$ sudo cumin 'C:role::authdns::server' 'find /etc/gdnsd/zones -type f -exec md5sum {} \; |sort -k 2|md5sum'
3 hosts will be targeted:
authdns[1001,2001].wikimedia.org,multatuli.wikimedia.org
Confirm to continue [y/n]? y
===== NODE GROUP =====                                                                                                                                                        
(1) multatuli.wikimedia.org                                                                                                                                                   
----- OUTPUT of 'find /etc/gdnsd/...sort -k 2|md5sum' -----                                                                                                                   
ab66d08220b2475065d38c7c3bffc311  -                                                                                                                                           
===== NODE GROUP =====                                                                                                                                                        
(2) authdns[1001,2001].wikimedia.org                                                                                                                                          
----- OUTPUT of 'find /etc/gdnsd/...sort -k 2|md5sum' -----                                                                                                                   
749a6448e31706eab82740cfdab0cf5a  -                                                                                                                                           
================                                                                                                                                                              
PASS:  |█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100% (3/3) [00:01<00:00,  2.01hosts/s]     
FAIL:  |                                                                                                                                 |   0% (0/3) [00:01<?, ?hosts/s]     
100.0% (3/3) success ratio (>= 100.0% threshold) for command: 'find /etc/gdnsd/...sort -k 2|md5sum'.
100.0% (3/3) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
bblack@cumin1001:~$ sudo cumin 'C:role::authdns::server' 'authdns-gen-zones -f /srv/authdns/git/templates /etc/gdnsd/zones && gdnsdctl reload-zones'
3 hosts will be targeted:
authdns[1001,2001].wikimedia.org,multatuli.wikimedia.org
Confirm to continue [y/n]? y
===== NODE GROUP =====                                                                                                                                                        
(3) authdns[1001,2001].wikimedia.org,multatuli.wikimedia.org                                                                                                                  
----- OUTPUT of 'authdns-gen-zone...ctl reload-zones' -----                                                                                                                   
info: Zone data reloaded                                                                                                                                                      
================                                                                                                                                                              
PASS:  |█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100% (3/3) [00:05<00:00,  1.96s/hosts]     
FAIL:  |                                                                                                                                 |   0% (0/3) [00:05<?, ?hosts/s]     
100.0% (3/3) success ratio (>= 100.0% threshold) for command: 'authdns-gen-zone...ctl reload-zones'.
100.0% (3/3) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
bblack@cumin1001:~$ sudo cumin 'C:role::authdns::server' 'find /etc/gdnsd/zones -type f -exec md5sum {} \; |sort -k 2|md5sum'
3 hosts will be targeted:
authdns[1001,2001].wikimedia.org,multatuli.wikimedia.org
Confirm to continue [y/n]? y
===== NODE GROUP =====                                                                                                                                                        
(3) authdns[1001,2001].wikimedia.org,multatuli.wikimedia.org                                                                                                                  
----- OUTPUT of 'find /etc/gdnsd/...sort -k 2|md5sum' -----                                                                                                                   
edb7c18c736c92f6f34fd73850a001b5  -                                                                                                                                           
================                                                                                                                                                              
PASS:  |█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100% (3/3) [00:01<00:00,  2.01hosts/s]     
FAIL:  |                                                                                                                                 |   0% (0/3) [00:01<?, ?hosts/s]     
100.0% (3/3) success ratio (>= 100.0% threshold) for command: 'find /etc/gdnsd/...sort -k 2|md5sum'.
100.0% (3/3) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.

(But note that first hop from Ashburn to Chicago is our routers' choice, so it's possible some of our route engineering is at play here).

From bast1001 to the endpoints shown in line (2) above over v4 and v6:

bblack@bast1002:~$ mtr -c 10 -r -4 bottomless.aa.net.uk
Start: Tue Dec  4 12:23:35 2018
HOST: bast1002                    Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- ae3-1003.cr2-eqiad.wikime  0.0%    10    0.2   0.2   0.2   0.4   0.0
  2.|-- ae0.cr1-eqiad.wikimedia.o  0.0%    10    0.2   0.2   0.2   0.3   0.0
  3.|-- xe-0-0-28-0.a03.asbnva02.  0.0%    10    1.7   2.8   0.6  11.6   3.4
  4.|-- ae-70.r06.asbnva02.us.bb.  0.0%    10   72.3  72.4  72.3  72.6   0.0
  5.|-- ae-2.r22.asbnva02.us.bb.g  0.0%    10    1.5   2.8   0.6  10.0   3.2
  6.|-- ae-5.r25.nycmny01.us.bb.g  0.0%    10    6.1   6.1   6.1   6.4   0.0
  7.|-- ae-1.r24.nycmny01.us.bb.g  0.0%    10    6.7   6.9   6.7   7.6   0.0
  8.|-- ae-9.r24.londen12.uk.bb.g  0.0%    10   73.7  74.6  73.7  79.4   1.7
  9.|-- ae-1.r04.londen05.uk.bb.g  0.0%    10   73.7  73.6  73.5  73.9   0.0
 10.|-- e.aimless.aa.net.uk       50.0%    10   74.6  74.7  74.6  74.7   0.0
 11.|-- bottomless.aa.net.uk       0.0%    10   74.7  74.8  74.7  74.9   0.0
bblack@bast1002:~$ mtr -c 10 -r -6 bottomless.aa.net.uk
Start: Tue Dec  4 12:23:58 2018
HOST: bast1002                    Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- ae3-1003.cr2-eqiad.wikime  0.0%    10    0.3   0.3   0.2   0.3   0.0
  2.|-- xe-0-1-5.cr2-eqord.wikime  0.0%    10   66.5  32.2  28.3  66.5  12.0
  3.|-- 10gigabitethernet4-1.core  0.0%    10   25.8  25.1  25.0  25.8   0.0
  4.|-- 100ge16-1.core1.nyc4.he.n 10.0%    10   25.2  25.2  25.1  25.3   0.0
  5.|-- 100ge16-2.core1.lon2.he.n  0.0%    10   92.2  92.4  92.1  93.2   0.0
  6.|-- k.aimless.thn.aa.net.uk    0.0%    10   92.4  92.4  92.3  92.5   0.0
  7.|-- bottomless.aa.net.uk       0.0%    10   91.1  91.2  91.1  91.3   0.0
bblack@bast1002:~$ mtr -c 10 -r -4 a.gormless.thn.aa.net.uk
Start: Tue Dec  4 12:24:41 2018
HOST: bast1002                    Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- ae3-1003.cr2-eqiad.wikime  0.0%    10    7.7   1.2   0.3   7.7   2.3
  2.|-- ae0.cr1-eqiad.wikimedia.o  0.0%    10    0.2   0.2   0.2   0.3   0.0
  3.|-- ???                       100.0    10    0.0   0.0   0.0   0.0   0.0
bblack@bast1002:~$ mtr -c 10 -r -6 a.gormless.thn.aa.net.uk
Start: Tue Dec  4 12:25:03 2018
HOST: bast1002                    Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- ae3-1003.cr2-eqiad.wikime  0.0%    10    0.2   0.3   0.2   0.3   0.0
  2.|-- xe-0-1-5.cr2-eqord.wikime  0.0%    10   28.3  33.5  28.3  79.0  16.0
  3.|-- 10gigabitethernet4-1.core  0.0%    10   25.0  25.1  25.0  25.6   0.0
  4.|-- 100ge16-1.core1.nyc4.he.n  0.0%    10   25.1  25.1  25.1  25.3   0.0
  5.|-- 100ge16-2.core1.lon2.he.n  0.0%    10   92.1  94.1  92.1 111.1   5.9
  6.|-- k.aimless.thn.aa.net.uk   10.0%    10   92.4  92.4  92.2  92.5   0.0
  7.|-- a.gormless.thn.aa.net.uk   0.0%    10   91.4  91.3  91.2  91.4   0.0

I can't reproduce this anymore in my own testing. I'm assuming it's fixed for now, barring further reports of continuing breakage showing up.

I think the patch reverted above was at fault. What I can't be sure of is whether the reversion will help immediately, or will take some time. I suspect it will have a positive effect fairly quickly (as each failed ExpKill is going to nuke quite a few objects before it ultimately fails).

They seem different, as T190988 is about faulty uploads (which I presume would still look broken when fetched directly from Swift), and this is about ones that are correct on Swift but have issues fetched through Varnish?

Yeah I got busy and dropped this.

Seems reasonable to close this; the event itself is long over. There are still risks present for a followup event, but if we close up all the actionables that goes away eventually. Maybe add incident tag and move to follow-up column for T206951? (the other is already there)

Update from SRE meeting today - memtest was successful, and we're asked to put it back in production and see if the error happens again or not. Re-pooling!

Downtimes set, we shouldn't get cert alerts in icinga

Thanks for the data and the patch! We'll dig into the DNS patch next week and get it merged in so we're serving wikiba.se from our DNS as-is (as in, pointing at your existing server IPs). Then we can do handoff of the domain ownership/registration without causing any interruptions.

When looking at the latest MaxMind data, it locates this network as being in New Zealand, which we map to ulsfo as first choice, and esams as the last-resort choice. But the destination would've been set by geodns logic, so probably what really mattered was the location of the DNS cache in use. For future debugging, try a DNS lookup on reflect.wikimedia.org, which will show us what DNS cache exit IP our servers see, e.g.:

Yet another! cp1078 crash ticket above merged into here.

In T119366#4754978, @Bawolff wrote:

Fwiw: im of the opinion that date magic words should reduce varnish cache to at least 24 hours, maybe six hours.

Also, we should pre-downtime the unified ssl checks in icinga for cp3NNN and cp5NNN early next week before the US Thanksgiving holidays, so that nobody's pestered by a spam of WARNING alerts, which I believe are set to trigger 60 days out from expiry.

From IRC for posterity:

Seems to be testing fine on https://pinkunicorn.wikimedia.org/ , and the pre-deployment to all caches hosts and OCSP Stapling looks fine too.

Removing the ops/traffic/domains tags, as the Foundation doesn't operate anything about these domains (we don't own or operate the DNS, the IPs, or the servers). Whois says they belong to:

The dual RSA+ECDSA certs above have:

Not Before: Nov  8 21:37:02 2018 GMT
Not Before: Nov  8 21:21:04 2018 GMT

Must-Staple didn't turn out to be a realistic option for GlobalSign, we'll look at it again later/elsewhere!

Looking at the puppet level of this, the root of this problem is that this VM has no $facts['ipaddress6'], because there's no IPv6 in wmflabs I guess. Gaining some IPv6 might help reduce the prod<->wmcs diff in all things puppet! :) On the other hand, apparently very few other manifests directly reference that fact/global.

For now I've taken the pragmatic approach, and the merge above seems to do the trick. update-ocsp-all is now run as an ExecStartPre of nginx when tlsproxy does OCSP stapling, and nginx won't start if it exits non-zero, and traffic-pool won't repool things without nginx. There's a minor recursive glitch in all of this, which is that when the OCSP updater runs as an nginx ExecStartPre, it also runs its hook that tries to reload nginx, which isn't yet running. This results in log output:

In case parsing all those regexes gets annoying/confusing:

Resolving for now, it's been repooled for a while without issue

cp1085 hit this today, virtually identical in all respects with the sequence of events and kernel/log outputs, etc. That's 2/16 nodes, a little under 2 months apart. Not an epidemic, but maybe worth looking into. Most likely this is a kernel driver bug or firmware bug rather than bad hardware. A simple reboot brough cp1085 back into service afterwards.

So, with regard to the potential staging delays in this and T207295 , the reason they're not urgent or required for conversion of the existing legacy LE usages is that none of those legacy cases do OCSP stapling, and none of them are for a high-traffic production project domain where we're really worried about supporting awful client clocks.

So, a few things I can say along those lines:

(or alternatively, we could look at this as one of the clear examples where a separate WMCS puppetization would be far simpler).

Yeah I'd agree that's the direction we should go. We don't offer our ntp servers to the globe for good reasons, and we similarly probably shouldn't be offering them to WMCS, plus it's pretty easy to build new NTP servers there. There's a whole bunch of prod-specific stuff in modules/profile/manifests/ntp.pp that needs hiera-izing to override it for WMCS use-case. The structural stuff there is far more complex than the WMCS NTP servers would need, but I think it could be adapted to that case.