Apologies @dcaro but I had less time for this than I expected this week, was only able to do some prep work but not be ready to touch anything in production before Friday.

@JMeybohm The best way to fix this is by adding a calico definition to the chart directory in a file whose name starts with wmf-, correct?

From discussion on IRC:

If we need to do a version of this for bare-metal hosts, we will, but for now let's not.

I think the last step to do here is to validate that any rsync failures will get reported on IRC. Then we can consider all the immediate followups of this incident done, and more slowly continue on with the larger work at T367119: Install a default timeout for systemd::timer::jobs.

Suggestions from discussion at I/F meeting:

• It's probably not necessary or desirable to add this to all of the contextmanager usages of alertmanager silences, as those are expected to be very short-lived
• "Manual" invocations of sre.hosts.downtime should almost certainly do this. Or in general any process where we don't have a pretty deterministic estimated-time-to-completion.
• We don't have an equivalent of "check optimal" for alertmananger, only Icinga. We should probably have this.
• Would be very good to have a dedicated dashboard for silences that are suppressing active alerts but being auto-extended

Alternatives to consider:

• Make this a required field instead of adding a default [harder up-front but potentially safer]
• Make omitting this field wmf puppet style guide violation [slower version of the above]

Mentioning T364280: Add jaeger-ui and other stuff to mwcli here.

Very helpful, thanks @dcaro and enjoy the pto!

Hi all. @joanna_borun asked me to do some looking into this. I promise I skimmed the above, but I'm sure I missed things, so please pardon me for the pretty basic questions.

Example trace as processed in codfw production:
https://trace.wikimedia.org/trace/06aabdeeb578a2663034270cf6d4accf

• Remove known PII (sessionstore URL)
• Filter out some very noisy spans (e.g. healthchecks, Special:Blankpage etc)

Both of these verified working in production :)

Patches as written depend upon otelcol v0.102.0, so upgrading again.

In T366272#9878290, @kostajh wrote:

@jijiki @CDanis as a follow-up, could someone from SRE please confirm that the GeoLite2 files at /usr/share/GeoIPInfo/GeoLite2- have been successfully updated today?

I discussed this with @Muehlenhoff in his evening/my morning.

I think you are right @Vgutierrez, thanks

Results after adding BR.ix are in.

Results after adding BR.ix are in.

In T353464#9854420, @JMeybohm wrote:

In T353464#9845947, @jijiki wrote:

Current status:

As I see it we're currently also still running the ganeti etcd instances in codfw and eqiad which I think does limit the performance of the etcd cluster by quite a bit. Was it a deliberate decision to not remove them?

As we expected/hoped, the increase in eqiad TX bytes was only about 10-15%.

In T366094#9842327, @akosiaris wrote:

I am gonna disagree on this one. This says that outgoing traffic from kubemasters can each 272MB/s, which is ~2.1Gbps, which is > 2/3 of the total NIC capacity of the 3 new control plane nodes.

In T366094#9841558, @Stashbot wrote:

Mentioned in SAL (#wikimedia-sre) [2024-05-29T11:23:04Z] <akosiaris> T366094 re-undeploy otel-collector, it being around increased traffic to the API >50%

• A few times we triggered a rolling restart of mw-api-int pods in eqiad to deliberately cause the CPU/network TX spikes on the two original masters, so we could observe more about the behavior. We wanted to figure out "what had changed" such that a scap deploy was now creating a ProbeDown page.
  • We captured pcaps on a few different k8s masters during such events, and then ran that through wireshark's Statistics > Conversations feature. Here's one such breakdown. I sorted the list of streams by bytes and manually tagged the top dozen or so streams. I stopped a few entries into a very long run of nearly-identical byte counts from local port 6443 (apiserver) to different node IPs; if you sum up the ones I tagged plus all of those, that makes up 97% of the bytes in the sample.
  • Of that 97% portion I inspected, everything sending packets to and from the apiserver machines was reasonable-looking: reading from an etcd's port 2380 was about 5% of overall bytes, then after that, in order, the apiserver sending lots of data to all of: the istiod pod, a calico-kube-controllers pod, a k8s-controller-sidecars pod, various different node IPs (so one of kubelet, kube-proxy, or rsyslogd)... all of them expected, known usages of the API.
  • This is 'just' absence of evidence but I'm gonna go ahead and call it evidence of absence here.

Posting a short comment now, before I start drafting a much longer comment (and possibly don't finish before my toddler ends my day):

I tested out a simple enabling of the k8s attributes processor in the values in the chart. The diffs all looked quite reasonable, but of course it doesn't work:

Traces are flowing again in eqiad.

helmfile apply went seamlessly, but unfortunately this broke trace collection: I realized only in retrospect that this effectively also changes the DNS name of the collector, and that's vendored into a lot of other charts with the full old name: main-opentelemetry-collector.opentelemetry-collector.svc.cluster.local

Hi Arzhel, for when I do have time to look at this, do you have a recommended way of reproducing without breaking anything or potentially actually affecting a network device?