While doing a backport for https://gerrit.wikimedia.org/r/c/1127458/ , httpb test suite failed three times in a row, each time on the baremetal mwdebug servers.
The baremetal debug servers timing out is a regular issue (T360867, T371427), but I don't know whether that could be a more general issue and thus mark it as a train blocker.
==== Error ====
* mwversion: 1.44.0-wmf.21
* reqId: `1ac78293-5f0b-992b-a2a4-a90c026ae66d`
* [[ https://logstash.wikimedia.org/app/dashboards#/view/AXFV7JE83bOlOASGccsT?_g=(time:(from:'2025-03-17T03:33:02.380Z',to:'2025-03-18T08:24:51.650Z'))&_a=(query:(query_string:(query:'reqId:%221ac78293-5f0b-992b-a2a4-a90c026ae66d%22'))) | Find reqId in Logstash ]]
```name=normalized_message,lines=10
[{reqId}] {exception_url} UnexpectedValueException: Invalid server index #
```
| Frame | Location | Call
| -- | -- | --
| from | [[ https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/core/+blame/refs/heads/wmf/1.44.0-wmf.21/includes/objectcache/SqlBagOStuff.php#371 | /srv/mediawiki/php-1.44.0-wmf.21/includes/objectcache/SqlBagOStuff.php(371) ]] |
| #0 | [[ https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/core/+blame/refs/heads/wmf/1.44.0-wmf.21/includes/objectcache/SqlBagOStuff.php#535 | /srv/mediawiki/php-1.44.0-wmf.21/includes/objectcache/SqlBagOStuff.php(535) ]] | SqlBagOStuff->getConnection(null)
| #1 | [[ https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/core/+blame/refs/heads/wmf/1.44.0-wmf.21/includes/objectcache/SqlBagOStuff.php#192 | /srv/mediawiki/php-1.44.0-wmf.21/includes/objectcache/SqlBagOStuff.php(192) ]] | SqlBagOStuff->fetchBlobs(array, bool)
| #2 | [[ https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/core/+blame/refs/heads/wmf/1.44.0-wmf.21/includes/libs/objectcache/MediumSpecificBagOStuff.php#122 | /srv/mediawiki/php-1.44.0-wmf.21/includes/libs/objectcache/MediumSpecificBagOStuff.php(122) ]] | SqlBagOStuff->doGet(string, int)
| #3 | [[ https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/core/+blame/refs/heads/wmf/1.44.0-wmf.21/includes/libs/objectcache/BagOStuff.php#783 | /srv/mediawiki/php-1.44.0-wmf.21/includes/libs/objectcache/BagOStuff.php(783) ]] | Wikimedia\ObjectCache\MediumSpecificBagOStuff->get(string, int)
| #4 | [[ https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/core/+blame/refs/heads/wmf/1.44.0-wmf.21/includes/libs/objectcache/MultiWriteBagOStuff.php#396 | /srv/mediawiki/php-1.44.0-wmf.21/includes/libs/objectcache/MultiWriteBagOStuff.php(396) ]] | Wikimedia\ObjectCache\BagOStuff->proxyCall(string, int, int, array, Wikimedia\ObjectCache\MultiWriteBagOStuff)
| #5 | [[ https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/core/+blame/refs/heads/wmf/1.44.0-wmf.21/includes/libs/objectcache/MultiWriteBagOStuff.php#187 | /srv/mediawiki/php-1.44.0-wmf.21/includes/libs/objectcache/MultiWriteBagOStuff.php(187) ]] | Wikimedia\ObjectCache\MultiWriteBagOStuff->callKeyMethodOnTierCache(int, string, int, int, array)
| #6 | [[ https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/core/+blame/refs/heads/wmf/1.44.0-wmf.21/includes/parser/ParserCache.php#278 | /srv/mediawiki/php-1.44.0-wmf.21/includes/parser/ParserCache.php(278) ]] | Wikimedia\ObjectCache\MultiWriteBagOStuff->get(string, int)
| #7 | [[ https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/core/+blame/refs/heads/wmf/1.44.0-wmf.21/includes/parser/ParserCache.php#377 | /srv/mediawiki/php-1.44.0-wmf.21/includes/parser/ParserCache.php(377) ]] | MediaWiki\Parser\ParserCache->getMetadata(WikiPage, int)
| #8 | [[ https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/core/+blame/refs/heads/wmf/1.44.0-wmf.21/includes/page/ParserOutputAccess.php#239 | /srv/mediawiki/php-1.44.0-wmf.21/includes/page/ParserOutputAccess.php(239) ]] | MediaWiki\Parser\ParserCache->get(WikiPage, MediaWiki\Parser\ParserOptions)
| #9 | [[ https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/core/+blame/refs/heads/wmf/1.44.0-wmf.21/includes/page/Article.php#729 | /srv/mediawiki/php-1.44.0-wmf.21/includes/page/Article.php(729) ]] | MediaWiki\Page\ParserOutputAccess->getCachedParserOutput(WikiPage, MediaWiki\Parser\ParserOptions, null, int)
| #10 | [[ https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/core/+blame/refs/heads/wmf/1.44.0-wmf.21/includes/page/Article.php#545 | /srv/mediawiki/php-1.44.0-wmf.21/includes/page/Article.php(545) ]] | Article->generateContentOutput(MediaWiki\User\User, MediaWiki\Parser\ParserOptions, int, MediaWiki\Output\OutputPage, array)
| #11 | [[ https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/extensions/Wikibase/+blame/refs/heads/wmf/1.44.0-wmf.21/repo/includes/Actions/ViewEntityAction.php#80 | /srv/mediawiki/php-1.44.0-wmf.21/extensions/Wikibase/repo/includes/Actions/ViewEntityAction.php(80) ]] | Article->view()
| #12 | [[ https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/extensions/Wikibase/+blame/refs/heads/wmf/1.44.0-wmf.21/repo/includes/Actions/ViewEntityAction.php#55 | /srv/mediawiki/php-1.44.0-wmf.21/extensions/Wikibase/repo/includes/Actions/ViewEntityAction.php(55) ]] | Wikibase\Repo\Actions\ViewEntityAction->showEntityPage()
| #13 | [[ https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/core/+blame/refs/heads/wmf/1.44.0-wmf.21/includes/actions/ActionEntryPoint.php#730 | /srv/mediawiki/php-1.44.0-wmf.21/includes/actions/ActionEntryPoint.php(730) ]] | Wikibase\Repo\Actions\ViewEntityAction->show()
| #14 | [[ https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/core/+blame/refs/heads/wmf/1.44.0-wmf.21/includes/actions/ActionEntryPoint.php#507 | /srv/mediawiki/php-1.44.0-wmf.21/includes/actions/ActionEntryPoint.php(507) ]] | MediaWiki\Actions\ActionEntryPoint->performAction(Article, MediaWiki\Title\Title)
| #15 | [[ https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/core/+blame/refs/heads/wmf/1.44.0-wmf.21/includes/actions/ActionEntryPoint.php#145 | /srv/mediawiki/php-1.44.0-wmf.21/includes/actions/ActionEntryPoint.php(145) ]] | MediaWiki\Actions\ActionEntryPoint->performRequest()
| #16 | [[ https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/core/+blame/refs/heads/wmf/1.44.0-wmf.21/includes/MediaWikiEntryPoint.php#202 | /srv/mediawiki/php-1.44.0-wmf.21/includes/MediaWikiEntryPoint.php(202) ]] | MediaWiki\Actions\ActionEntryPoint->execute()
| #17 | [[ https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/core/+blame/refs/heads/wmf/1.44.0-wmf.21/index.php#58 | /srv/mediawiki/php-1.44.0-wmf.21/index.php(58) ]] | MediaWiki\MediaWikiEntryPoint->run()
| #18 | [[ https://gerrit.wikimedia.org/r/plugins/gitiles/operations/mediawiki-config/+blame/refs/heads/master/w/index.php#3 | /srv/mediawiki/w/index.php(3) ]] | require(string)
| #19 | {main} |
----
Console outputs from this morning:
```
07:22:54 Executing check 'check_testservers_baremetal-1_of_1'
07:22:54 Executing check 'check_testservers_k8s-1_of_2'
07:22:54 Executing check 'check_testservers_k8s-2_of_2'
07:23:42 Check 'check_testservers_baremetal-1_of_1' failed: Sending to 4 hosts...
https://test.wikidata.org/wiki/Property:P13344 (/srv/deployment/httpbb-tests/appserver/test_main.yaml:133)
mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet
Status code: expected 200, got 500.
Body: expected to contain 'rbkKZufs', got '<!DOCTYPE html>\n<html class="client-nojs vector-fe'... (34883 characters total).
https://test.wikidata.org/wiki/Wikidata:Main_Page (/srv/deployment/httpbb-tests/appserver/test_main.yaml:136)
mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet
Status code: expected 200, got 500.
Body: expected to contain 'test instance', got '<!DOCTYPE html>\n<html class="client-nojs vector-fe'... (37308 characters total).
===
FAIL: 147 requests sent to each of 4 hosts. 2 requests with failed assertions.
```
```
07:29:01 Executing check 'check_testservers_baremetal-1_of_1'
07:29:01 Executing check 'check_testservers_k8s-1_of_2'
07:29:01 Executing check 'check_testservers_k8s-2_of_2'
07:29:37 Check 'check_testservers_baremetal-1_of_1' failed: Sending to 4 hosts...
https://test.wikidata.org/wiki/Property:P13344 (/srv/deployment/httpbb-tests/appserver/test_main.yaml:133)
mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet
Status code: expected 200, got 500.
Body: expected to contain 'rbkKZufs', got '<!DOCTYPE html>\n<html class="client-nojs vector-fe'... (34883 characters total).
https://test.wikidata.org/wiki/Wikidata:Main_Page (/srv/deployment/httpbb-tests/appserver/test_main.yaml:136)
mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet
Status code: expected 200, got 500.
Body: expected to contain 'test instance', got '<!DOCTYPE html>\n<html class="client-nojs vector-fe'... (37308 characters total).
===
FAIL: 147 requests sent to each of 4 hosts. 2 requests with failed assertions.
```
```
07:30:22 Executing check 'check_testservers_baremetal-1_of_1'
07:30:22 Executing check 'check_testservers_k8s-1_of_2'
07:30:22 Executing check 'check_testservers_k8s-2_of_2'
07:30:57 Check 'check_testservers_baremetal-1_of_1' failed: Sending to 4 hosts...
https://test.wikidata.org/wiki/Property:P13344 (/srv/deployment/httpbb-tests/appserver/test_main.yaml:133)
mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet
Status code: expected 200, got 500.
Body: expected to contain 'rbkKZufs', got '<!DOCTYPE html>\n<html class="client-nojs vector-fe'... (34883 characters total).
https://test.wikidata.org/wiki/Wikidata:Main_Page (/srv/deployment/httpbb-tests/appserver/test_main.yaml:136)
mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet
Status code: expected 200, got 500.
Body: expected to contain 'test instance', got '<!DOCTYPE html>\n<html class="client-nojs vector-fe'... (37308 characters total).
https://wikimedia.org/research (/srv/deployment/httpbb-tests/appserver/test_redirects.yaml:3)
mwdebug1002.eqiad.wmnet
Status code: expected 301, got 503.
Location header: expected 'https://research.wikimedia.org/', was missing.
===
FAIL: 147 requests sent to each of 4 hosts. 3 requests with failed assertions.
```