In T405688#11781939, @Lucas_Werkmeister_WMDE wrote:

I don’t see what reliability has to do with it? We wanted to know the impact of adding a bunch of namespace aliases in many different languages (would this “shadow” any existing pages?), and it seemed to me that the best way to do that would be to dry-run namespaceDupes on all wikis, with the patch experimentally applied, and then inspect the maintenance script’s output (and compare it with a “regular” dry-run without the patch).

While investigating a different problem, I found that we have a similar(?) issue when mediawiki contacts the DBs T422489: rdbms errors in eqiad

I can't help noticing that MediaWiki periodic job update-special-pages-s5 failed failed twice for the same reason, which is either a very unfortunate coincidence related to T422489: rdbms errors in eqiad, or something worth investigating.

I am closing this, since we are tracking that in T422486

In T420336#11777591, @Jgiannelos wrote:

Update on the status of parsoid testreduce for rt-testing:

• In order to not break the current rt-testing infrastructure which is very critical for our weekly processes I created a dev env in cloudvps.
• Our testsuite is updated to use custom headers to access the parsoid instance in k8s
• From a quick run things look like that the basic functionality is there

Some things that need improvement

• I think that having 2 different envs and trying to keep track what the active DC is is getting complicated in our testsuite
  • Ideally we would like to have a single discovery instance that always points to the active DC

In T420223#11797805, @elukey wrote:

@jijiki shall we deploy mcrouter 2023.07.17.00-2 and test the keep alive options? I have the feeling that the TKOs will go down after it.

MW-Interfaces-Team same questions for you for changeprop/cpjobqueue /api-gateway/Ratelimit:

In T422455#11795500, @Scott_French wrote:

The changes in https://gerrit.wikimedia.org/r/1268569 were applied to codfw and eqiad at 15:10 and 15:20 UTC today respectively.

Following those changes, the rate of errors has plummeted (logstash):

Things looks quite well so far mw-cron (MediaWiki Periodic Jobs on k8s) after merging 1268569. More details in T422455#11795500

Infrastructure-Foundations two questions:

• how netbox will behave if it looses connectivity to its redis and then start with a cold cache?
• do we have any concerns updating to redis 8?

Take this with a grain of salt, it seems like something indeed changed during the week of March 17th, and if eqiad was not producing all those errors, we wouldn't have noticed.

failed jobs have been deleted, closing this too for T422486

failed jobs have been deleted, closing this too for T422486

failed jobs have been deleted, closing this too for T422486

I filtered timeouts from the mediamoderation-hourlyscan job in an attempt to establish if we are seeing those timouts more after switching to eqiad.

@A_smart_kitten Thank you!

Setting aside any mediawiki changes, the difference between the two DCs in the same time period (post codfw repooling), is alarming

It seems like EtcdConfig failed to fetch data: (curl error: 28) Timeout was reached started sometime around March 17th, with eqiad exhibiting

Seems like a temporary network error

The nearest (timestamp wise) long entry I found yielded a temp network problem. I am not aware how long it takes for @phaultfinder to create a task to be absolutely sure this is the one

This was due to a temp connection errors to the DB https://logstash.wikimedia.org/app/discover#/doc/logstash-*/logstash-k8s-1-7.0.0-1-2026.04.05?id=Z0EgXJ0BbI6kJ8WywbVL

In T420336#11777591, @Jgiannelos wrote:

Update on the status of parsoid testreduce for rt-testing:

• In order to not break the current rt-testing infrastructure which is very critical for our weekly processes I created a dev env in cloudvps.
• Our testsuite is updated to use custom headers to access the parsoid instance in k8s
• From a quick run things look like that the basic functionality is there

Some things that need improvement

• I think that having 2 different envs and trying to keep track what the active DC is is getting complicated in our testsuite
  • Ideally we would like to have a single discovery instance that always points to the active DC
• We might need to bump up the resources on the k8s instance we use. Currently it looks like if I crank up slightly the concurrency of our testing, it fails.
• Redirects wipe the headers so for pages in the testsuite that are redirects it currently fails so we need to fix it too

In T421168#11789662, @LucasWerkmeister wrote:

I don’t see how that’s a reason to close the task? I don’t really care whether you think it’s related to the DC switchover or not, I care about the issue getting fixed. And to me the Grafana graph linked in the task description, looking at badtoken errors in the last 30 days, doesn’t really look like a return to normal yet (though the logarithmic scale makes it hard to interpret, and AFAICT my volunteer account doesn’t have sufficient privileges to preview an edited version with a linear scale):

Due to an unfortunate coincidence, this issue caused a paging event.

I just realised this is actually two tasks

Updated, I also sorted the dashboard's variables to be inline with the other ones

In T422130#11784439, @Od1n wrote:

FWIW, I'm still currently encountering this error on frwiki, and it prevents my local custom JS/CSS files from loading.

Unexpectedly not loaded:

• Special:Mypage/common.css, Special:Mypage/common.js, Special:Mypage/vector.css, Special:Mypage/vector.js…

Not impacted — loading as expected:

• MediaWiki:Common.css, MediaWiki:Common.js, MediaWiki:Vector.css, MediaWiki:Vector.js…
• meta:Special:Mypage/global.css, meta:Special:Mypage/global.js

In T414486#11773036, @elukey wrote:

Other services:

• https://wikitech.wikimedia.org/wiki/Redioscope - @daniel @Clement_Goubert same question for you :)