ssllabs confirms the expected changes above. The sslmate generator doesn't allow for custom entries to get globalsign.com in early (as a likely guess for when they flip their switch before the impending CA/B deadline).

I had to manually fix up salt keys and do final reboots on 4001+4003, all should be sane and consistent now (except for a couple of IPMI temp checks showing UNKNOWN in icinga).

Answering my own timeline question, it looks like it was announced that RCStream goes away July 7th!

Ok I think I was confused as to the state of the uselang hack. It looks like it already works, in uncacheable form, on all wikis? When I try it on enwiki with uselang set to es, I get an uncacheable copy of the article with the UI header stuff in Spanish, the content-language response header still set to en, and of course the article text still in en. On that front, it would still be an improvement if uselang outputs were cacheable, IMHO.

( Note also ori did a soft announce of HTTPS transition for it about a year ago, but with no target date for disabling plain HTTP: https://lists.gt.net/wiki/wikitech/719999 . This was around the same time RCStream's wikitech docs had their URLs switched to HTTPS as well ).

@Ottomata - Any high level new info about timetables for deprecating and then removing the RCStream stuff in favor of EventStreams ( T130651 )? If it looks like it might drag on a while, we might want to go back to the idea of announcing an HTTPS-only transition ahead of the removal, perhaps. The main issue there was that at least some RCStream clients don't seem to follow redirects to HTTPS, and therefore would need manual updates of their configs to use https:// or wss:// or they get broken.

The original point of this (now ~2 years old) tracking task was to track the very long tail of known but relatively-minor issues preventing us from reaching a full transition to modern HTTPS-only for all things public-facing that the Foundation has control over, in the wake the transition for our major public hostnames announced in https://blog.wikimedia.org/2015/06/12/securing-wikimedia-sites-with-https/ . This ticket has a danger of becoming an undead meta-task as more sub-tasks might topically accrete to it over time. It's not intended to replace the idea of a tag or a workboard column, after all, and we have such a thing in the TLS column of the Traffic workboard for tracking all other ongoing TLS improvements. The remaining set of valid open tasks is fairly small at this point and a few of them are near closing, so we're going to try to wind this ticket down completely over the next several months, and I'm going to unlink a few on categorical grounds today.

This has been working for some time, at least for the HTTPS issue at the root as tasked here! The other part about docs probably isn't relevant anymore, as the service is being replaced.

Currently this looks to be fixed. The relevant snippet on the live store site is now:

<script>
        if (window.location.protocol == "http:") {
                var restOfUrl = window.location.href.substr(5);
                window.location.replace("https:" + restOfUrl);
        }
</script>
<link rel="canonical" href="https://store.wikimedia.org/" />

(and yes, they do seem to properly 301-redirect HTTP to HTTPS, so I'm not sure why the hacky JS protocol redirect is still there, but it doesn't hurt anything)

@Jseddon @MBeat33 - ping again? The redirect appears to work currently, but still no HSTS header.

Yeah we can close this task if the sites are gone. We'll want to remove the current IP address mapping for these hostnames from our DNS when this happens (or now, if they're already no longer in use), to complete the removal of the concern. Is it just benefactorevents.wikimedia.org and eventdonations.wikimedia.org that are going away, or also others like benefactors.wikimedia.org?

In T166782#3307457, @Amire80 wrote:

I guess that the traffic to wikimediafoundation.org is relatively low, and anonymous selection can be enabled. There's a long discussion about enabling it on Commons, where the traffic is higher and there are concerns about caching issues, but it shouldn't be a problem on wikimediafoundation.org. If ops don't object, it should be done.

This task has gotten a bit confusing. Stepping back a bit from the specific case of Commons (because I think the same issues apply everywhere?)... let me try to recap here a little, and correct me please if I've gotten some of this wrong:

In T164768#3374447, @ema wrote:

On text, transient storage usage seems pretty reasonable; we could cap as follows, leaving plenty of room for spikes:

cache_type	layer	cap
text	frontend	5G
text	backend	2G

No updates yet, we're still finalizing DC vendor selection (one of several steps before APNIC will possibly give us new address space). Any firmer timeline on how long after acquisition of the address space the partner update process will take?

I think it's fair to at least put forward arguments for a 3rd point of view as well:

We've discussed the V5 upgrade a few times in the past (although not much recently or explicitly), so I'll try to recap here some related thoughts:

In T168033#3357737, @Yurivict wrote:

But I don't see how is it reasonable to fail requests when some metric is exceeded, vs. delaying responses.

We've also been tweaking and tuning our ratelimits in general to try to find a happy medium. Both of the API endpoints should now be limiting at the same rate of 1000 reqs per 10s per client IP (as a burstable token bucket filter).

In T167400#3355085, @Nemo_bis wrote:

Why restrict this mechanism to Zero, making Zero different from other access? We could instead deny access to unpatrolled files for users that aren't logged-in.

But then Commons would no longer be a wiki.

In T167400#3350622, @Bawolff wrote:

Wikipedia Zero traffic is tied to IP addresses, not users. So it definitely could be performant. Have MediaWiki set an unpatrolled header and purge on patrol. Then (somehow) configure Varnish to understand WP0 IP ranges and block if the unpatrolled header is set.

So the idea would be:

• MediaWiki sets something along the lines of MediaWiki-patrol-status: unpatrolled in File::getContentHeaders()
• Varnish looks for that header when getting files from swift. If the file is unpatrolled, and (maybe) its above a certain size, and the IP address is Zero-rated: Give a 403. Also make sure that cache varrying is set for unpatrolled files based on Zero-ratedness of IP
• On patrol, MediaWiki makes swift backend remove the header, and sends purge to varnish.
  
  Downsides: If anyone using a zero-rated connection is a file patroller, they won't be able to see the file.

Multicast has its uses in general. Even if we kill HTCP another use may pop up. I did quick survey to try to find active uses. Filtering for just v4, removing the standard references you'd see to 224.0.0.1 everywhere, and ignoring the excepted cp machines' multicast address as documented at https://wikitech.wikimedia.org/wiki/Multicast_HTCP_purging#Multicast_Addressing , these were the surprises:

That top client appears to be CrossRefEventDataBot from https://www.crossref.org/services/event-data/ , running on a hosted server at Hetzner in DE.

In T167920#3349772, @Haiku-narrative wrote:

we actually haven't been given the numbers yet, but I expect us to handle a few million requests over the course of a couple of hours. We only make calls to you on cache miss, so all of the traffic won't roll over, but from what we saw last night we tend to cache miss on 3-9% of requests.

What are the real pros and cons on this? We could even go in the other direction and have a unique ASN per region/continent. How does the impact future anycasting? Note https://tools.ietf.org/html/rfc6382 talks about best practice for anycast being to have distinct ASNs per region, but I don't pretend to understand all the finer details and arguments in that RFC.

No I don't think we need it for non-immediate analysis like this. We still zero, zeronet and proxy in the X-Analytics string too (which is also in webreq data)

The cp* should at least occasionally be sending normal ICMP responses correlated with their TCP flows, e.g. "Time Exceeded" and such. The LVSes are configured to schedule inbound ICMPs to the caches as well, IIRC. As for the original problem, the inbound Dest Unreach aren't necessarily in response to ICMP echo or similar. I think they could be the natural result of TCP SYN spoofing towards us (e.g. attacker sends spoofed SYN to us with unreachable source address, we send back SYN+ACK to said unreachable dest address, then some router sends us an ICMP Dest Unreach in response to our SYN+ACK).

I think this is because our mobile-redirect logic doesn't support dashes, but probably should due to these cases? Patch above proposed, if it makes logical sense to support mobile-redirects on "language" subdomains containing dashes (\w is only alphanumerics and underscore).

cp3003 is decomming for good in T167376

Stared at hashtable implementation some more, as well as the linux iptables hashlimit one (which I consider a sort of baseline canonical efficient implementation). The linux one is definitely more-configurable, but I think we can work with what vsthrottle gives us in terms of configurability of the bucket itself. The lack of a cost parameter is still an issue, but it's a long-term issue if/when we get to the point where we define a header that applications can send in their response to indicate heavier costs.

re: vsthrottle, my thoughts after a quick look this morning:

In T108435#3307340, @Gehel wrote:

My understanding from @BBlack is that we don't want to add application specific configuration in varnish, my understanding from @MaxSem is that this is something that is already done for multiple applications.

Another thought - could we be maxing out parallel connections to the kartotherian machines? We've always had a max_connections of 1000 (per varnish backend, to kartotherian), but the number of varnish backends has multiplied with this change, which could have multiplied the overall limit on connections opened from varnish to kartotherian, and then hit some limit causing further connections to stall?

Are you comparing cache hits to cache misses? From where? What was the timing like before?

What setting were we on before we moved to FF-only? Whatever the prior setting was, it tended to create spurious merge commits all over our actual git history which makes a mess of it, like: https://github.com/wikimedia/puppet/commit/1df0dab661e57fc73942defa4ba4a57436b1b240 . Is "Rebase if necessary" new-ish since then or did we just fail to know it was the optimal option?

[lvs1011 above just had some minor salt keying issues, fixed+rebooted]

Yeah that was the plan, for XKey to help here by consolidating that down to a single HTCP / PURGE per article touched. It's not useful for the mass-scale case (e.g. template/link references), as it doesn't scale well in that direction. But for the case like "1 article == 7 URLs for different formats/variants/derivatives" it should work great. The varnish module for it is deployed, but we haven't ever found/made the time to loop back to actually using it (defining standards for how to transmit it over the existing HTCP protocol or the new EventBus and pushing developers to make use of it). I think last we talked we were going to move cache-purge traffic over to EventBus before tackling this (with kafka consumers on the cache nodes pulling the purges), but I'm not sure what the relative timelines on all related projects look like anymore.

This is what we had before (copied from far above):

We can get broader averages by dividing the values seen in the aggregate client status code graphs using eqiad's text cluster (the remote sites would expect fewer due to some of the bursts being more likely to be dropped by the network)

We talked a bit on IRC. Probably the first step is to include all the canonical domains (the 14-domain set in our big unified cert):

The lack of graph data from falling off the history is a sad commentary on how long this has remained unresolved :(

In T155806#3300964, @BBlack wrote:

(even the non-canonicals, IMHO).

Why start with just wikipedia and wikimedia? We could go after our lower-traffic domains first as a test, but since we don't issue individual certs for them there's no real functional testing to happen there. Perhaps we could use a lesser domain just to validate that other tools parse and validate the CAA correctly from the public DNS view, though. Once we turn on the big two, we may as well turn on all the others as well (even the non-canonicals, IMHO).

In T147569#3294214, @Gilles wrote:

There is an apparent performance improvement that coincides in timing, but on a simulated slow internet connection:

T166373: Investigate apparent performance improvement around 2017-05-24

The improvement is only experienced on the large articles + slow connection combo. Could that be it?

So I've stared at NavTiming graphs, and honestly it's hard to read any notable difference in the tea leaves.

From my perspective, where we last stalled out is waiting for Analytics to say it's ok to merge https://gerrit.wikimedia.org/r/#/c/253474 (which removes XFF data from the webrequest stream). I'm not sure what blockers/deps/validation may be pending over in the Analytics side before that.

Agreed this should be 4xx rather than 5xx

acamar hit this again on Sunday, in spite of the (working) acpi_pad blacklist. A simple reboot seems to have cleared it. The next- best advice (based on that old Dell info) would be to blacklist mei. I've rmmod'd it on acamar for now to see if it causes additional issues before we try blacklisting it on all.

@Jgreen - re: civicrm, it needs to emit the HSTS header on all HTTPS responses.

Not resolved, as the purge graphs can attest!

So from the above, apache really has 3 different modes of operation:

The tentative and limited plan for now is to deploy 3x misc/infra hosts (meaning all the hosts other than lvs and cp) at each cache site and not use virtualization. We might revisit this at a later date. The basic layout looks like:

I'm probably backtracking into territory that was once known here, but after the long delay I felt I had to go back and re-validate what's going on with the ports and the thinking.

Answering for @ema I think this mostly came up as a consequence of trying to map out the data in T150256#3271004 using lldpcli to confirm port connections. That led to an in-depth conversation about how our racks and rows and switches and vlans are set up and how and where redundancy matters, which led to him looking at our caches and their port/rack/row mapping (also in racktables) and how they're not laid out very ideally in eqiad. Basically these are just questions spawned from exploration.

I think that almost universally, HT is a win for the host as a whole. There's always more things going on than there are cpu cores. If nothing else, picture it in your head as "puppet agent and stats outputting stuff can run in that extra headroom without impacting the important stuff" or whatever.

Edited the top part, re-ran excluding virtuals.

So, a couple points:

On the reboot issue: I've tested cp4021 and the existing puppetization works fine on reboot (even given the other stuff below).

I wonder if Chrome (which is the dominant browser now, not MSIE as indicated in that nginx source comment) sends the close notify?

Also notable: lvs1009 and lvs1012 connections to row B (eth2) are using 1GbE ports rather than 10GbE?

Also while I'm thinking about it - we should validate that the sysctl setting for fq as default qdisc "sticks" on reboot and isn't affected by some kind of ordering race...

There's not a lot of good data on how BBR behaves in datacenter-like networks (high bandwidth, low latency, low loss, etc). It's not really the use case it was designed for, and the reports from others have been mixed. It probably won't turn out completely awful or anything, but I don't know if it would actually fix the port saturation problem or not.

Re: ethernet port validation / config, the last table we had in the old ticket is here: T104458#1788478 . The idea was to try our best to ensure that a given vlan/row's LVS connections are FPC-redundant between the primaries and secondaries. e.g. if lvs1007 (primary for high-traffic1) connects to row C in asw-c-eqiad FPC 5, then lvs1010 (secondary for high-traffic1) needs to connect to row C / asw-c-eqiad in some FPC other than 5. Row D connections have probably changed entirely since that last table was made, and there were some pending moves/fixups listed there as well which may or may not have already happened.