From what I can see:

"[SQLBagOStuff] SqlBagOStuff::handleDBError: ignoring connection error"

dumping extra context that we miss in production on my dev machine results in:

"context":{"db_log_category":"performance","measure":"masterConns","maxSeconds":0,"by":"MediaWiki\\Actions\\ActionEntryPoint::execute","actualSeconds":1,"query":"[connect to sql-mariadb-primary.default.svc.cluster.local ()]","exception":{},"trxId":null,"fullQuery":"[connect to sql-mariadb-primary.default.svc.cluster.local ()]","dbHost":null}}

5 Wiki's that have some POST/PUT traffic now:
metabase.wikibase.cloud
eusterm.wikibase.cloud
lgbtdb.wikibase.cloud
data.r74n.com
blackbibliog.wikibase.cloud

I suspect we now see this error post I7ca996618e41b93f488cb5c4de82000bb36e0dd3 from T320873. Since this point the previously suppressed by us DBPerformance log is no longer used for the txnProfiler. Instead it uses the generic rdbms log.

We believe we have locally reproduced this error but opening an item page, purging the parser cache and then reparsing the page such that it is inserted into the parser cache. We then suppressed it by setting $wgParserCacheType = CACHE_NONE; and $wgParserCacheType = 'redis2';

To me it looked maybe a bit similar to T388165 but we aren't using SUL; maybe it is still some Auth Manager thing though?

I think I dug up this ticket T237143 as relevant to

[SQLBagOStuff] Duplicate get(): "mwdb_f036984f01-mwt_93f623752b_:echo:seen:message:time:1" fetched 2 times

I'm not actually sure the context here is that important but I wonder if we still want to include it at some point in this jsonPayload.

Got the context with a small change to this line in my dev env.

Only manage to look at this a tiny bit. The conclusion I immediately came to is that our error logging is missing sufficient context to understand what the error really is.

Perfect, this is now done and also we have started donating to the correct level for unbranded usage so we can go right ahead and follow the instructions. I hope we can figure this out relatively trivially; I believe we should automatically be added to some private repo (techarohq-botstopper) where we can then pull the image from.

I also saw failures for the custom domains:

will reopen if the user replies

@Anton.Kokh reached out to the user to confirm if they are able to use the Wikibase on Nov 28th 2025. We are yet to hear back.

I checked and can confirm all do appear to be running 1.43 apart from wbqc1.wikibase.dev which was deleted this morning

In T407499#11417121, @Pintoch wrote:

...

• why is Anubis enabled on an API endpoint at all?

Last I got to was probably trying to figure out how to use cloud-init (and the correct corresponding cloud-init debian image) was probably where we want to look. I think we could then make a proxmox VM template for this to spin up VM with ssh keys for all the devs added at the drop of a hat. I also think it should be possible to use cloud-init to ensure that these VM's have every package in we need.

Merged and tested on staging on ollie-test1.wikibase.dev. Tomorrow I think this is all ready for T411012

Turns out it was the validator expecting 0 / 1 rather than true / false.

waiting for T409530