Need more investigation

• https://console.cloud.google.com/errors/detail/COyO-aiI_5q2Zg;time=P7D;locations=global?project=wikibase-cloud occurs on mediawiki 1.43 ~ 500 occurences
• https://console.cloud.google.com/errors/detail/CLLJm-eviqOjaw;time=P7D;locations=global?project=wikibase-cloud need to be looked at more and could be related to the one above.
• https://console.cloud.google.com/errors/detail/CPbfnf7y8LGyaA;time=P7D;locations=global?project=wikibase-cloud needs a seperate ticket for more investigation.

To ignore

• https://console.cloud.google.com/errors/detail/CP2Tj_KIlOvNEQ;time=P7D;locations=global?project=wikibase-cloud solved by this https://github.com/wbstack/mediawiki/pull/492
• https://console.cloud.google.com/errors/detail/CIeK26H_6fWJRQ;time=P7D;locations=global?project=wikibase-cloud solved by https://github.com/wbstack/mediawiki/pull/493
• https://console.cloud.google.com/errors/detail/COTY79bQs5zkxQE;time=P7D;locations=global?project=wikibase-cloud occured once early on when we added 1.43, revisit if it happens again
• https://console.cloud.google.com/errors/detail/CPvV6dSKuZHJ4gE;time=P30D;locations=global?project=wikibase-cloud occured only on the 15th october , revisit if it happens again
• https://console.cloud.google.com/errors/detail/CKDn5ISY0bjMuAE;time=P30D;locations=global?project=wikibase-cloud same as the error directly above. happened at the same time and both related.

• All Error 1146 as they are caused by an attempt to read the wrong db pod

@Andrew-WMDE and I checked these 4 examples to make sure these were dues to 143 pods making requests on 139 databases https://console.cloud.google.com/errors/detail/CPmpkcnq-vWsmgE;time=P7D;locations=global?project=wikibase-cloud ~3300 occurences
https://console.cloud.google.com/errors/detail/CPesrdL0srieRQ;time=P7D;locations=global?project=wikibase-cloud ~42
https://console.cloud.google.com/errors/detail/CKW6i_LooYvaMA;time=P7D;locations=global?project=wikibase-cloud ~7
https://console.cloud.google.com/errors/detail/CL6vq9froNPIVg;time=P7D;locations=global?project=wikibase-cloud ~ 3700

wrong andrew, maybe?

In T407481#11298370, @Andrew wrote:

wrong andrew, maybe?

Oops! sorry

Investigate

[rdbms] Expectation (masterConns <= 0) by MediaWiki\MediaWikiEntryPoint::restInPeace not met (actual: 1): [connect to sql-mariadb-primary.default.svc.cluster.local ()]

from https://console.cloud.google.com/errors/detail/COyO-aiI_5q2Zg;time=P7D;locations=global?project=wikibase-cloud
see T408101

[SQLBagOStuff] Duplicate get(): "mwdb_f036984f01-mwt_93f623752b_:echo:seen:message:time:1" fetched 2 times

from https://console.cloud.google.com/errors/detail/CLLJm-eviqOjaw;time=P7D;locations=global?project=wikibase-cloud
see T408101

[RedisBagOStuff] MediaWiki\Extension\OAuth\Backend\MWOAuthToken value for 'OAUTH:mwdb_f036984f01-mwt_93f623752b_:token:TOKEN:request:REQUEST'; serialization is suspect.

from https://console.cloud.google.com/errors/detail/CPbfnf7y8LGyaA;time=P7D;locations=global?project=wikibase-cloud
see T408103

Monitor

Error 1146: Table 'mwdb_cc73c3ce73.mwt_edcc7ed9cb_block' doesn't exist Function

from https://console.cloud.google.com/errors/detail/CPmpkcnq-vWsmgE;time=P7D;locations=global?project=wikibase-cloud

• Likely caused by the Platform API attempting to use the MediaWiki 1.43 back-end to run jobs for instances that have not yet been upgraded from MediaWiki 1.39
• Should resolve itself once all instances are upgraded to MediaWiki 1.43 or sooner by T407504
• Check database version via SQL using: SELECT version FROM apidb.wiki_dbs WHERE name = '';

[rdbms] Error 1146 from MediaWiki\Block\DatabaseBlockStore::newLoad, Table 'mwdb_cc73c3ce73.mwt_edcc7ed9cb_block' doesn't exist

from https://console.cloud.google.com/errors/detail/CL6vq9froNPIVg;time=P7D;locations=global?project=wikibase-cloud

• Same as Error 1146 above

Error 1054: Unknown column 'pl_namespace' in 'where clause' Function: BacklinkCache::queryLinks Query

from https://console.cloud.google.com/errors/detail/CPvV6dSKuZHJ4gE;time=P30D;locations=global?project=wikibase-cloud

• Has not occurred since the initial deploy of MediaWiki 1.43 on staging
• I suspect a straggling job with a mismatched MediaWiki back-end

[runJobs] htmlCacheUpdate Main_Page table=pagelinks recursive=1 rootJobIsSelf=1 rootJobSignature= rootJobTimestamp=causeAction=page-touch namespace=0 title=Main_Page requestId= causeAgent=unknown (id=3,timestamp=20251015090341) t=9 error=Wikimedia\Rdbms\DBQueryError: Error 1054: Unknown column 'pl_namespace' in 'where clause' Function: BacklinkCache::queryLinks

from https://console.cloud.google.com/errors/detail/CKDn5ISY0bjMuAE;time=P30D;locations=global?project=wikibase-cloud

• Same as Error 1054 above

[ParserCache] Inconsistent revision ID

from https://console.cloud.google.com/errors/detail/CLTtg5LK_d7fFQ;locations=global?project=wikibase-cloud

• Has not occurred again for over a week

Resolved

Undefined constant MediaWiki\Extension\OAuth\Backend\ConsumerAcceptance::READ_NORMAL

from https://console.cloud.google.com/errors/detail/CP2Tj_KIlOvNEQ;time=P7D;locations=global?project=wikibase-cloud
see wbstack/mediawiki/pull/492 | Call IDBAccessObject::READ_NORMAL directly

Call to undefined method MWHttpRequest::factory()

from https://console.cloud.google.com/errors/detail/CIeK26H_6fWJRQ;time=P7D;locations=global?project=wikibase-cloud
see wbstack/mediawiki/pull/493 | Replace call to removed MWHttpRequest::factory

In T407481#11322507, @Ollie.Shotton_WMDE wrote:

Unassigning and marking as stalled until we can confirm that we don't need to take further action on the "monitor" errors.

After talking about it in the daily today I created T408789 as a follow up, so we can move this task here to done