+1 on the latest patch, looks like the right fix.

@ayounsi - Yes, we're going to have some outbound recursive DNS needs from some ganeti-hosted services

Looking at that other ticket T250912 - would an in-band service ping or NOP event of some kind address detecting "hung" connections at the application layer? Does SSE have any kind of null requests or null events that could be sent for that purpose?

In T242767#6201754, @Ottomata wrote:

[reordering a little]

What happens right now if someone had to download a large file from I dunno, commons or dumps.wikimedia.org on a slow connection, such that the download takes more than 15 minutes? Would they be disconnected and never be able to download the file?

Even if we could experimentally verify option A, we probably can't trust it across future firmware differences (between sites, or between two routers in a site). Option B via MEDs sounds like a good path forward for now, though!

Interestingly, the mobile redirect code in varnish doesn't strip any parameters. The problem is that the analytics-side VCL code that consumes the wprov parameter also removes it from the URL (so that only analytics sees it, but it doesn't affect caching or get sent to MediaWiki, etc), so it's not present anymore at the time the redirect happens.

The kraz case is gone now (yay!) and hasn't recurred since the ircd restart above. What's left appears to be all infrastructure stuff: PDUs, switches, firewalls, etc. I've picked up quite a few of them in a few hours, so I'm going to let it run for a full 24h to try to capture them all, and then I'll make some sub-tasks to clean them up.

(correction - it's also internet-reachable via ulsfo only for now, in this interim state, just by chance because it's still advertising the whole original /23)

Update: the nsa authdns IP at 198.35.27.27 is live internally everywhere and monitored and working. There's some stuff to finish up later this week for the public side in T253196 , and then we can start really digging into the detailed bits about loadbancing and hashing with a live testbed.

Status Update: Worked through a bunch of the software-level complexities today with getting bird::anycast to advertise an authdns IP from all the auth and dnsbox hosts ( https://gerrit.wikimedia.org/r/#/q/topic:anycast-authdns+(status:open+OR+status:merged) ), and merged a DNS patch to set up a single anycast authdns IP in the unused /24 we had adjacent to ulsfo's space in: https://gerrit.wikimedia.org/r/#/c/operations/dns/+/597310/ . There's still one patch outstanding, which turns on the actual adverts and local healthchecks, at: https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/597311/ .

There's not much DoTLS adoption so far, and really our primary HTTPS termination needs this more than AuthDNS does, at which point we can just copy whatever solution emerges there.

Diving a little deeper on the symlink issue:

In T251726#6108138, @Dzahn wrote:

Even if we still use non-LE certs in some DCs i believe this is ok since we should also have other monitoring for the expiration of that cert. We do, right?

I like the root event timestamp info. We could potentially put in future rules to help by ignoring ancient purges, in some cases (e.g. if we can guarantee the cache's contents are no older then 24h, we can also ignore root events older than 24h, which might speed up replaying a backlog of data...).

Yeah, the linked Lua code is, I think, trying to emulate what our VCL has always traditionally done similarly as:

In T133821#6092865, @Joe wrote:

• Define a schema for a "url purge message".

I worry that removing a no-cache default might have all kinds of unintended consequences. We should probably do some research into what php outputs are even relying on this default and why.

We're probably going to add multiple purger connections to fan out the per-thread load from T241232 to help with T249325 . I've poked at that already, and the likely impact here is that the overall format will stay the same, but the Purger0 labels at the start of the lines will probably change, so don't rely on the PurgerN formatting of that leading label. I'm thinking right now that I'll give the purgers names in the config and then number them for the configured fanout factor, so we're likely to see something more like:

Probably-related: T241232

I had a chat with the author to make sure we understand the meaning of the fields:

So, the backend purging queues in esams are way behind. On the one node I'm staring at the most, there are currently about 87 million backlogged purge requests, which is probably somewhere in the ballpark of 10 hours of lag time. The backlog is in the local daemon on the esams hosts themselves (so this isn't a network issue with delivering the purges to the hosts over the WAN), so the culprit is likely the ATS daemon consuming them slowly.

Yeah, the varnish (frontend) code for this is in modules/varnish/templates/text-frontend.inc.vcl.erb:

# normalize all /static to the same hostname for caching
if (req.url ~ "^/static/") { set req.http.host = "<%= @vcl_config.fetch("static_host") %>"; }

Most likely the cause is that the Varnish rule for normalizing /static/ to the enwiki hostname hasn't been applied to ATS, and thus this isn't effectively purging /static from the ATS backend caches. Will check...

Resolving this, since it has become an undead tracker for too long. There are still two trailing issues, but having this over-arching tracking task above them isn't accomplishing anything:

We're not using nginx software for this functionality anymore, and everything else related to these parts of the software stack have changed and are still evolving, so this task doesn't make a ton of sense anymore as it stands.

Update: sometime since I last checked, they've changed the header to: strict-transport-security: max-age=31557600 (~1 year, vs ~90 days before). Still missing the other attributes (preload and includeSubDomains)...

Sorry, I missed this update when it flew through on a busy week. Will do today!

After the recent merger https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/571928/ - I'm having installer failure on dns2001 (we did its sibling dns2002 a few days before the merge without issue). It breaks out to promting with the installer GUI for netmask stuff and then disk partitioning. I saved the debug stuff over HTTP to cumin1001.eqiad.wmnet:~bblack/dns2001-logs/ ... we can live with it for a little bit, so leaving things as-is for debugging...

I think all that's left on our side is a DNS switch, which is pretty trivial and quick.

@RobH - Yes, let's edit this to include eqiad as well. We've had the same symptoms both places, and they're the same approximate generation of hardware configuration (IIRC, only the NVMe changed to a slightly newer/better model from eqiad to esams, but the base system is otherwise fairly identical).

Everything but the first few rows (system/repl stuff) and with all the Sleep rows removed:

Status update: what's missing here is codfw, which will happen when we finish its hardware upgrade switch to lvs2007-10

We also tested depooling just port 80 yesterday, which didn't affect anything (fd leak was still growing), which means this isn't driven by external->:80 traffic. cp4029 was at ~400K fds this morning, so I depooled its frontend services and restarted ats-tls + varnish-fe to reset the clock. I also repooled cp4032 while I was in there.

Most of this has been configured now, the remaining slightly difficult bit is configuring an alternate SNI cert for the domain on our new ats-tls termination.

Yes, more or less. The major caveat is some of our caches still have non-redirecting copies of various pages in https://transparency.wikimedia.org/ , but this will sort itself out over the next day or so at the most. To save anyone from trawling through the list of commits above, the changes in effect now are:

+1 from me, this was one of the many things we made the ganeti clusters for :)

Seems like a good plan!

So long as the registry's responses do all the standards-based things correctly (they contain Vary: Accept, and the matching Accept values also match the Content-Type values in the responses), this should Just Work on a functional level.

The wording issues here are actually a bit tricky. We've done several TLS standards upgrades over time, and there are still a few to go:

Or a patch to template this in. The problem is it's implemented from a standard template for the top 30-40 lines, which isn't specific to this case, in an attempt to standardize our error output templates.

External queries now working (note they all return a codfw IP without edns-client-subnet in play, because codfw is closest to my laptop and PROXYv2 is working for sending the "real" client IP from haproxy to gdnsd).

Actually we can't realistically do global monitoring from icinga either, because icinga isn't on Buster and so it doesn't have the right library/tool access to check a TLSv1.3-only service, so we'll have to settle for the per-server NRPE checks for now.

Refactoring the dependencies a little here: Really (2) above's sub-point about shared ticket key rotation won't matter until we're anycasting, so I've made a separate task (+subtask) in T240863 to go look at that stuff later, blocking the anycast work.

All of this is irregular and outside of policies we like to adhere to, but I'll push a zonefile to our nameservers which supports the bare minimum (existing Stanford-hosted IPs for the insecure site http://wikiworkshop.org and the same IP for redirects from http://www.wikiworkshop.org , and nothing else ). At some point after the holidays are over, I'd like to find out what the overall intent and/or plan is here so that we can provide some additional guidance and get this onto some kind of more-acceptable path though.

This is now mostly-working, with heira flag controlling test deployment (currently only on dns4002, which doesn't have any public authserver IPs routed into it at this time).

P9867 <- First internal test query on a prod dns box :)

I'm not even sure what the task is asking for, but yeah in general we're not going to make the sec-warning mechanism comply with all expected valid outputs from all possible APIs/URIs it's covering. It's designed to break things, in a way that at least provides some level of human info on what's going on if someone digs in and looks. The next step in the transition process after this is that whatever agent they're using which is getting the sec-warning output won't be able to establish a connection to our infrastructure at all, which is way more broken than this.

In T238038#5734955, @TheDJ wrote:

BTW. We no longer have the cipher stats grafana board ? Too bad, that one was hella interesting.

The way it works is that if the connection isn't using TLSv1.2, the user is served a 302 redirect to /sec-warning on the same domain, which in turn returns a cacheable 200 OK with the HTML warning content and the CT header as text/html; charset=utf-8. There are a lot of gory details in the compromises being made by that solution (vs. eg. we could have returned some kind of 4xx error immediately rather than 302->200), but we've learned this is the best pattern to avoid misbehavior of certain bots and scrapers out there in the world which spam-retry [45]xx return codes.

Yes, it's about that $notrack default. My hypothesis is that setting it to true wouldn't break any traffic, wouldn't change the security situation much, but would eliminate a bunch of potential for conntrack table size issues when various services get overwhelmed. Some thoughts about why that hypothesis might be false:

Status: The actual LVS portion of this is now completely removed globally. The IP addresses themselves are also completely unconfigured and removed from service at the all the edge sites, but not the core ones. What remains is that the legacy LVS recdns IPs 208.80.154.254 (eqiad) and 208.80.153.254 (codfw) are still statically-configured to avoid breaking any of the leftover dependencies on these IPs. Sniffer monitoring has shown at least the ircd instance on kraz is still using outdated resolv.conf data and hitting these IPs, several hardware PDUs are using them as well, and there are possibly other such cases which are rarer and thus harder to observe in short samples (I've done up to 1h samples).

I'm assuming that, for now, the hosting of the web service (and email?) is not moving, just the whois ownership and DNS service? We usually need a fair bit more information than this to handle such a case smoothly. At a glance it looks like there's potentially more to this (e.g. they have MX and SPF records, are there are also DMARC and such we need to copy?). Also, basic TLS doesn't seem to work on the target site, either. Is there a project-level task or something for whatever transition is happening here?