I 've just re-enabled the filter, rejecting traffic, we are meeting issues with high latencies and decreased availability in the parsoid cluster.

@RobH, Switchover was done yesterday, we are now in codfw for the next 6 months, deploy1002 is no longer used. It can be powered off and moved whenever ops-eqiad feels like it.

I 've just disabled the rule. It's still present, but inactive. For other SREs having to re-enable it in an emergency:

We 'll schedule a scap deploy for RESTBase, thanks @jnuche

with overrides being configured within WMF production wiring, as opposed to provided by the software. That imho violates the separation of concerns and wouldn't scale for other MW users to know about and keep in sync across core and hundreds of extension repos, and across major version upgrades.

The range and sizes of buckets in the histogram can be defined per metric (actually group of metrics, e.g. via a regex). We already use this a lot, e.g. in various services in WikiKube, where each service configures statsd-exporter per metric they want. It is not possible to do this on demand, as in allow the producer to change it, on the fly, without shipping a config change.

I 'll admit I am a bit stumped here. This is clearly not the CDN's fault as RESTBase exhibits the same behavior while also violating what it advertises as the documentation of the API.

https://donate.wikipedia.org/.well-known/apple-developer-merchantid-domain-association now, in my checks, returns the contents of the file in this task. It might take a bit more (up to 30 minutes) to propagate everywhere. I am resolving this task, feel free to reopen to report issues.

I am gonna add one more data point. In all of these errors, the data.servedby stanza refers to an *eqiad* API server. I looked a bit at the distribution of those API servers to see if there is any pattern that would identify one or more specific ones, (thankfully?) that is not the case, apparently all eqiad API servers have the probability to appear in this dataset.

Should we resolve this?

In T337649#9162878, @Xover wrote:

In T337649#9162778, @akosiaris wrote:

I can sense the frustration pretty clearly […]

Oh, if that came across as venting frustration I have to apologize. It was an attempt at a funny way to communicate 1) that Thumbor needs kicking again (as it has needed periodically since the opening of this task), and 2) that tweaking the parameters for parallelism, queue size, and timeouts is probably not going to be sufficient since it keeps recurring (iow, we'll be back here in a couple of weeks, wasting Hnowlan's time with yet another temporary workaround).

I can sense the frustration pretty clearly and I appreciate the effort to illustrate it via this story, avoiding lashing out. As a data point, users aren't the only ones frustrated with the situation. Engineers (developers, software engineers, SREs) are frustrated (and have been for a long time) too.

Yes, it is safe, we haven't put yet those in production.

ICU67 images, built and pushed.

Editing the article will indeed issue cache purge events for both the CDN and RESTBase.

Given this isn't urgent and we have multiple ways of dealing with this, I 've re-enabled puppet and cadvisor has been started again. Sure enough, the latency has increased again.

In T345738#9148572, @fgiunchedi wrote:

In T345738#9148513, @akosiaris wrote:

Hi

TL;DR

cadvisor is to blame. Adding @fgiunchedi for his information and a thumbs up on disabling cadvisor on conf2* until we can bump their kernel version.

I'm ok to disable cadvisor there, though I gotta ask what are the plans for conf2* upgrades and/or reboot ?

There's a few more actionables here:

This isn't present in conf1* hosts, despite also running cadvisor and the same exact version, presumably because of a different kernel version. conf1* hosts are bullseye and conf2* hosts are buster. 5.10.0-15-amd64 vs 4.19.0-20-amd64

cadvisor is to blame. Adding @fgiunchedi for his information and a thumbs up on disabling cadvisor on conf2* until we can bump their kernel version.

I can reproduce internally, this has nothing to do with the CDN, it looks more like a RESTBase or a PCS service issue.

So, ethtool -G eno1 rx 1000 apparently did the trick

Adding a data point that just crossed my mind, just to rule it out.

I 've uploaded changes for icu67 php7.4 images for use with a shellbox deployment. I 'll also create a temporary shellbox deployment based on those.

In T345290#9139949, @Physikerwelt wrote:

In T345290#9139520, @akosiaris wrote:

I 've tried and did manage to find an owner in the past, but we are back to square 1 regarding this. I just hope that the plans to fully migrate math rendering in the browser will materialize and we 'll be able to undeploy this service at some point.

It will take a while. While we will be ready for opt-in very soon (maybe even this month), it will be a long way to fix all issues. To convince people that a browser is not TeX engine, will be even harder and to advocate for the advantages of inherent math display will take some time as well.

I 've merged and deployed the change mentioned in T344747, alongside the dependent changes. curl call listed in the Mathoid page returns succesfully an SVG, and openapi checks are apparently also fine. I 'll resolve this one but I 'll note that for

I think that the subject is misleading. Per https://gerrit.wikimedia.org/r/plugins/gitiles/operations/deployment-charts/+/33520a1a4409a9b0cef71a0b4baba148f64f2d40 (gerrit change is https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/809194) deployed mathoid version isn't 2023-02-21 but 2022-06-28-144716-production (which is even older).

@Ladsgroup @Marostegui , dependent T340843 is now resolved, you can proceed with the decom process. Thanks and sorry for waiting so long.

ipoid and toolhub done today. Resolving this. We 've chosen a path forward, implemented it and migrated the services that utilized hardcoded dbproxies networking rules to the new thing.

In T331699#9136475, @MoritzMuehlenhoff wrote:

One other option would be to simply start with a fresh, parallel setup and skip Bullseye entirely:

• Create new ldap-rw1001/ldap-rw2001 VMs using Bookworm and set profile::openldap::storage_backend to "mdb" and configure them as a synchronisation pair
• slapcat the existing data from serpens to an LDIF (ACLs, LDAP extensions are all distributed via Puppet)
• slapadd the LDIF on ldap-rw1001 and let it sync towards ldap-rw2001
• Create four additional ldap-replica VMs running Bookworm and sync them against ldap-rw1001/2001
• Test the new setup
• When everything works as expected in the parallel setup, revert the new Bookworm hosts to a clean state
• Setup a window (1-2 hours) during which no r/w changes are possible (disable Bitu temporarily, tell SREs to avoid LDAP changes, disable Horizon)
• Repeat the same import as above with current data, if all is well:
• Point ldap-rw.codfw.w.o to ldap-rw2001
• Point ldap-rw.eqiad.w.o to ldap-rw1001
• Depool all older readonly replicas in favour of the new bookworm ones
• If there are unforeseen issues we can simply revert to serpens/seaborgium/old replicas
• If all is well, decom serpens/seaborgium and the old replicas

linkrecommendation done today.

FYI, same mitigation applies to https://supportportal.juniper.net/s/article/2023-08-29-Out-of-Cycle-Security-Bulletin-Junos-OS-and-Junos-OS-Evolved-A-crafted-BGP-UPDATE-message-allows-a-remote-attacker-to-de-peer-reset-BGP-sessions-CVE-2023-4481?language=en_US, released yesterday

Those numbers are for summary quantiles, not histograms buckets. Summaries aren't aggregatable and in almost all cases regarding multi-instance (e.g. multiple servers, multiple pods) metrics you don't want to deal with them. Almost all efforts to run aggregation queries over them will result in wrong results.

Patch was merged and deployed for cxserver. Things went ok after a couple of roadbump and corresponding brown paper bag fixes. I 'll be deploying the changes to ipoid, toolhub and linkrecommendation in the next few days and remove the hardcoded references to dbproxies from those deployments.

Fix merged and deployed. Some hiccups aside, it works fine across all 3 environments (staging, production eqiad, production codfw). I 'll resolve this one.

In T340843#9123222, @Urbanecm_WMF wrote:

Hi all, is there any update on this please? linkrecommendation service had a second incident today caused by the same problem. I've updated the network policies again, but it'd be great to implement one of the solutions described in this task soon-ish to prevent future outages.

In T340036#9062519, @vadim-kovalenko wrote:

Hi there! I'm responsible for Kiwix migration to another API, but given the discussion above I'm curious whether you have plans to add MWOffliner to the allowed list to get access to /mobile-sections. And if so, how long it will be working? I assume that even though MWOffliner User-Agent was added earlier, MCS completely disabled already, because I've got 403 error page for this curl request:

curl -H "User-Agent: MWoffliner/1.13.0 (contact@kiwix.org)" https://en.wikipedia.org/api/rest_v1/page/mobile-sections

In T341122#9042792, @RLazarus wrote:

Those numbers don't immediately raise alarm bells for me -- "storage" doesn't mean anything persistent, only ephemeral data that can disappear when the script exits, right? As long as that's the case (and assuming you're using ~1 CPU), you should be fine. I'm tagging in @akosiaris to confirm the resource request is sensible.

Increase the threshold of the alert from 1s to 2s (or 1.5) as I'm not aware of any issues arising from this

I am gonna close this as declined. While we do have the ability to block requests based on user-agent, we don't do that on request.

I 've gone ahead and populated the Saturation panels. Traffic, Errors and Latencies will need more work, but I will not be able to help with that anytime soon.

I 've gone ahead and created https://grafana.wikimedia.org/d/FEkiKFqVk/wikifunctions?orgId=1

php7.4-fpm-multiversion-base rebuilt as well, should make it out to mw-on-k8s in the next deployments. I think we can resolve this now. Feel free to reopn.

The apparmor changes have been merged. I think the goal of this task is done. I 'll resolve, but feel free to reopen.

This is apparently stopped happening yesterday, the 23rd of July ~9:00am

OK, scheduling for tomorrow then, https://wikitech.wikimedia.org/wiki/Deployments#Tuesday,_July_25.

In T288198#9036874, @elukey wrote:

In T288198#9035228, @dancy wrote:

Another case of failing to push a large image: T342084. Is it possible to configure NGINX to use a different (large, on-disk) storage area for certain URLs?

Use case from ML - we are porting the recommendation-api Python service from wmf-cloud to k8s. It uses a dict file containing embeddings (basically a serialized blob) that weights around 3GB (so one big layer when we add it) and that it is loaded when the application boostraps. We are working with Research to figure out how to handle cases like this one, since "embeddings" could get multiple shapes and sizes, and they'd need to have flexibility when they experiment (like, using a serialized blob as starter and then think about some specialized datastore). This is not an "experimentation" use case of course, but the recommendation-api's setting may vary in the future. We could think about other options if raising the tmpfs mount is not an option (like fetching from swift or similar).

In T288198#9035228, @dancy wrote:

Another case of failing to push a large image: T342084. Is it possible to configure NGINX to use a different (large, on-disk) storage area for certain URLs?

@RobH mw hosts are 3 api servers and 3 appservers. You can do them anytime. Also it requires is a downtime and a poweroff per the description.

In T340087#9027714, @TheresNoTime wrote:

In T340087#9008668, @akosiaris wrote:

[...]
@TheresNoTime let me know when we should proceed with the next step of the deployment. Which should be across the whole fleet, unless you have a different idea.

Just to check, does the deployment to the whole fleet require a deployment window? It's sat on the canaries with no issues for about a week now, so we could probably start thinking about that full deployment..

A search says: https://svn.apache.org/viewvc/httpd/httpd/trunk/modules/loggers/mod_log_config.c?r1=98912&r2=98911&pathrev=98912

Just found something more important than User-Agent unfortunately

This is apparently due to transclusions per https://grafana.wikimedia.org/d/000300/change-propagation?orgId=1&from=1689418150653&to=1689449568835&viewPanel=27

@hnowlan @jijiki. nutcracker removal merged and deployed. I am gonna let you have the pleasure of resolving this task :-)

In T339102#9008337, @MSantos wrote:

@akosiaris as one of the maintainers of the Maps stack I'll leave my approval and ping @SLopes-WMF to complete as CTT manager.

In T340087#9008758, @MoritzMuehlenhoff wrote:

JFTR; I've also rebuilt/uploaded wikidiff 1.41.1 for component/icu67 (so that we don't regress when the ICU67 migration starts)

Looking quickly at mw-canaries and mwdebug, they all have 1.13.0-1+wmf1+buster1

Can I just say that this is pretty awesome? Especially the max latencies for kafka are pretty telling. Keep up the good work on this one!

Wikiwand/0.1 (https://www.wikiwand.com; admin@wikiwand.com) added to the list of user-agents. Please advise if it doesn't work, otherwise please resolve.

@Dzahn, Judging from the content of the task, this is for Infrastructure-Foundations, not serviceops, retagging.

Just to add my 2 cents as a generic observation.

PCC at https://puppet-compiler.wmflabs.org/output/936062/42341/ says 0 diff for alert hosts, lvs hosts see a comment change in configuration and there arguably the .discovery.wmnet approach is better anyway informationally. The alerting part of this task doesn't apply anymore anyway, I am gonna resolve this.

In T335770#8988938, @Brycehughes wrote:

@akosiaris Yep all clear now from Georgia (the country). However, this lasted much more than "several minutes". What do you think is the maximum expected time on the CDN cache purge? (FWIW I am impressed that you can even purge your CDN [individually?] so quickly... cool tbh)

Misconfiguration on our side @Brycehughes, thanks for noticing it. Now the 2 URLs above work fine. Resolving, but please do reopen and let us know if you see anything else.

In T136744#8991535, @Aklapper wrote:

That means this is not resolved but declined.

No, it's not relevant to icinga so much any more (and it's going to be less and less). It's still an interesting informational thing though and the replacement is easy enough that I am just gonna give it a try.

@MSantos, change deployed today. e.g. https://en.wikipedia.org/api/rest_v1/page/mobile-sections now returns a 403 with the above message. I 've just tested wikiwand.com, it still works fine and requests from it function.

Any objections to switching "svc.%{::site}.wmnet" to "discovery.wmnet" ?