constraint-regex-checker
Closed, ResolvedPublicPRODUCTION ERROR
Actions

Assigned To

Authored By

	dancy
	Aug 1 2024, 6:51 PM

Description

logspam-watch output always has a burst of entries like the following during mediawiki scap deployments:

About 35 hits in total in a 1-2 minute window.

For details on a recent burst, see https://logstash.wikimedia.org/goto/ba8b9d9a4174a8b8bc2a3cc86a066a0c (wider time frame: https://logstash.wikimedia.org/goto/bdad52a58aec86bc5f46657506c9c1c0)

In T371633#10078899, @Krinkle wrote:

It seems to revolve around these three messages all for the same http://localhost:6025/call/ endpoint. This endpoint is defined in mf-config and Envoy (via Codsearch) as shellbox-constraints. All calls (see stack traces) are from the WikibaseQualityConstraints extension. […]

Last 7 days:

300x cURL error 7: Failed to connect to localhost port 6025: Connection refused
280x cURL error 56: Recv failure: Connection reset by peer
80x cURL error 52: Empty reply from server

[27d39b50-8d32-4e7e-864e-a43a1d522dc6] /w/api.php?action=wbcheckconstraints&format=json&formatversion=2&id=Q706575&status=violation%7Cwarning%7Csuggestion%7Cbad-parameters&uselang=nb   GuzzleHttp\Exception\ConnectException: cURL error 7: Failed to connect to localhost port 6025: Connection refused (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for http://localhost:6025/call/constraint-regex-checker

from /srv/mediawiki/php-1.43.0-wmf.18/vendor/guzzlehttp/guzzle/src/Handler/CurlFactory.php(210)
#0 /srv/mediawiki/php-1.43.0-wmf.18/vendor/guzzlehttp/guzzle/src/Handler/CurlFactory.php(158): GuzzleHttp\Handler\CurlFactory::createRejection(GuzzleHttp\Handler\EasyHandle, array)
#1 /srv/mediawiki/php-1.43.0-wmf.18/vendor/guzzlehttp/guzzle/src/Handler/CurlFactory.php(110): GuzzleHttp\Handler\CurlFactory::finishError(GuzzleHttp\Handler\CurlHandler, GuzzleHttp\Handler\EasyHandle, GuzzleHttp\Handler\CurlFactory)
#2 /srv/mediawiki/php-1.43.0-wmf.18/vendor/guzzlehttp/guzzle/src/Handler/CurlHandler.php(47): GuzzleHttp\Handler\CurlFactory::finish(GuzzleHttp\Handler\CurlHandler, GuzzleHttp\Handler\EasyHandle, GuzzleHttp\Handler\CurlFactory)
#3 /srv/mediawiki/php-1.43.0-wmf.18/vendor/guzzlehttp/guzzle/src/Handler/Proxy.php(28): GuzzleHttp\Handler\CurlHandler->__invoke(GuzzleHttp\Psr7\Request, array)
#4 /srv/mediawiki/php-1.43.0-wmf.18/vendor/guzzlehttp/guzzle/src/Handler/Proxy.php(48): GuzzleHttp\Handler\Proxy::GuzzleHttp\Handler\{closure}(GuzzleHttp\Psr7\Request, array)
#5 /srv/mediawiki/php-1.43.0-wmf.18/vendor/guzzlehttp/guzzle/src/PrepareBodyMiddleware.php(64): GuzzleHttp\Handler\Proxy::GuzzleHttp\Handler\{closure}(GuzzleHttp\Psr7\Request, array)
#6 /srv/mediawiki/php-1.43.0-wmf.18/vendor/guzzlehttp/guzzle/src/Middleware.php(31): GuzzleHttp\PrepareBodyMiddleware->__invoke(GuzzleHttp\Psr7\Request, array)
#7 /srv/mediawiki/php-1.43.0-wmf.18/vendor/guzzlehttp/guzzle/src/RedirectMiddleware.php(55): GuzzleHttp\Middleware::GuzzleHttp\{closure}(GuzzleHttp\Psr7\Request, array)
#8 /srv/mediawiki/php-1.43.0-wmf.18/vendor/guzzlehttp/guzzle/src/Middleware.php(63): GuzzleHttp\RedirectMiddleware->__invoke(GuzzleHttp\Psr7\Request, array)
#9 /srv/mediawiki/php-1.43.0-wmf.18/vendor/guzzlehttp/guzzle/src/HandlerStack.php(75): GuzzleHttp\Middleware::GuzzleHttp\{closure}(GuzzleHttp\Psr7\Request, array)
#10 /srv/mediawiki/php-1.43.0-wmf.18/vendor/guzzlehttp/guzzle/src/Client.php(333): GuzzleHttp\HandlerStack->__invoke(GuzzleHttp\Psr7\Request, array)
#11 /srv/mediawiki/php-1.43.0-wmf.18/vendor/guzzlehttp/guzzle/src/Client.php(107): GuzzleHttp\Client->transfer(GuzzleHttp\Psr7\Request, array)
#12 /srv/mediawiki/php-1.43.0-wmf.18/vendor/guzzlehttp/guzzle/src/Client.php(138): GuzzleHttp\Client->sendAsync(GuzzleHttp\Psr7\Request, array)
#13 /srv/mediawiki/php-1.43.0-wmf.18/vendor/wikimedia/shellbox/src/Client.php(162): GuzzleHttp\Client->sendRequest(GuzzleHttp\Psr7\Request)
#14 /srv/mediawiki/php-1.43.0-wmf.18/vendor/wikimedia/shellbox/src/Client.php(107): Shellbox\Client->sendRequest(string, array)
#15 /srv/mediawiki/php-1.43.0-wmf.18/extensions/WikibaseQualityConstraints/src/ConstraintCheck/Checker/FormatChecker.php(199): Shellbox\Client->call(string, string, array)
#16 /srv/mediawiki/php-1.43.0-wmf.18/extensions/WikibaseQualityConstraints/src/ConstraintCheck/Checker/FormatChecker.php(181): WikibaseQuality\ConstraintReport\ConstraintCheck\Checker\FormatChecker->runRegexCheckUsingShellbox(string, string)
#17 /srv/mediawiki/php-1.43.0-wmf.18/extensions/WikibaseQualityConstraints/src/ConstraintCheck/Checker/FormatChecker.php(139): WikibaseQuality\ConstraintReport\ConstraintCheck\Checker\FormatChecker->runRegexCheck(string, string)
#18 /srv/mediawiki/php-1.43.0-wmf.18/extensions/WikibaseQualityConstraints/src/ConstraintCheck/DelegatingConstraintChecker.php(568): WikibaseQuality\ConstraintReport\ConstraintCheck\Checker\FormatChecker->checkConstraint(WikibaseQuality\ConstraintReport\ConstraintCheck\Context\MainSnakContext, WikibaseQuality\ConstraintReport\Constraint)
#19 /srv/mediawiki/php-1.43.0-wmf.18/extensions/WikibaseQualityConstraints/src/ConstraintCheck/DelegatingConstraintChecker.php(469): WikibaseQuality\ConstraintReport\ConstraintCheck\DelegatingConstraintChecker->getCheckResultFor(WikibaseQuality\ConstraintReport\ConstraintCheck\Context\MainSnakContext, WikibaseQuality\ConstraintReport\Constraint)
#20 /srv/mediawiki/php-1.43.0-wmf.18/extensions/WikibaseQualityConstraints/src/ConstraintCheck/DelegatingConstraintChecker.php(370): WikibaseQuality\ConstraintReport\ConstraintCheck\DelegatingConstraintChecker->checkConstraintsForMainSnak(Wikibase\DataModel\Entity\Item, Wikibase\DataModel\Statement\Statement, null, array)
#21 /srv/mediawiki/php-1.43.0-wmf.18/extensions/WikibaseQualityConstraints/src/ConstraintCheck/DelegatingConstraintChecker.php(342): WikibaseQuality\ConstraintReport\ConstraintCheck\DelegatingConstraintChecker->checkStatement(Wikibase\DataModel\Entity\Item, Wikibase\DataModel\Statement\Statement, null, array)
#22 /srv/mediawiki/php-1.43.0-wmf.18/extensions/WikibaseQualityConstraints/src/ConstraintCheck/DelegatingConstraintChecker.php(132): WikibaseQuality\ConstraintReport\ConstraintCheck\DelegatingConstraintChecker->checkEveryStatement(Wikibase\DataModel\Entity\Item, null, array)
#23 /srv/mediawiki/php-1.43.0-wmf.18/extensions/WikibaseQualityConstraints/src/Api/CheckingResultsSource.php(53): WikibaseQuality\ConstraintReport\ConstraintCheck\DelegatingConstraintChecker->checkAgainstConstraintsOnEntityId(Wikibase\DataModel\Entity\ItemId, null, array, array)
#24 /srv/mediawiki/php-1.43.0-wmf.18/extensions/WikibaseQualityConstraints/src/Api/CachingResultsSource.php(235): WikibaseQuality\ConstraintReport\Api\CheckingResultsSource->getResults(array, array, null, array)
#25 /srv/mediawiki/php-1.43.0-wmf.18/extensions/WikibaseQualityConstraints/src/Api/CachingResultsSource.php(167): WikibaseQuality\ConstraintReport\Api\CachingResultsSource->getAndStoreResults(array, array, null, array)
#26 /srv/mediawiki/php-1.43.0-wmf.18/extensions/WikibaseQualityConstraints/src/Api/CheckConstraints.php(121): WikibaseQuality\ConstraintReport\Api\CachingResultsSource->getResults(array, array, null, array)
#27 /srv/mediawiki/php-1.43.0-wmf.18/includes/api/ApiMain.php(1948): WikibaseQuality\ConstraintReport\Api\CheckConstraints->execute()
#28 /srv/mediawiki/php-1.43.0-wmf.18/includes/api/ApiMain.php(926): ApiMain->executeAction()
#29 /srv/mediawiki/php-1.43.0-wmf.18/includes/api/ApiMain.php(897): ApiMain->executeActionWithErrorHandling()
#30 /srv/mediawiki/php-1.43.0-wmf.18/includes/api/ApiEntryPoint.php(153): ApiMain->execute()
#31 /srv/mediawiki/php-1.43.0-wmf.18/includes/MediaWikiEntryPoint.php(200): MediaWiki\Api\ApiEntryPoint->execute()
#32 /srv/mediawiki/php-1.43.0-wmf.18/api.php(44): MediaWiki\MediaWikiEntryPoint->run()
#33 /srv/mediawiki/w/api.php(3): require(string)
#34 {main}

[90652139-0d8e-4b3e-a0ef-15cf1d36e145] /rpc/RunSingleJob.php   GuzzleHttp\Exception\RequestException: cURL error 56: Recv failure: Connection reset by peer (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for http://localhost:6025/call/constraint-regex-checker

from /srv/mediawiki/php-1.43.0-wmf.18/vendor/guzzlehttp/guzzle/src/Handler/CurlFactory.php(211)
#0 /srv/mediawiki/php-1.43.0-wmf.18/vendor/guzzlehttp/guzzle/src/Handler/CurlFactory.php(158): GuzzleHttp\Handler\CurlFactory::createRejection(GuzzleHttp\Handler\EasyHandle, array)
#1 /srv/mediawiki/php-1.43.0-wmf.18/vendor/guzzlehttp/guzzle/src/Handler/CurlFactory.php(110): GuzzleHttp\Handler\CurlFactory::finishError(GuzzleHttp\Handler\CurlHandler, GuzzleHttp\Handler\EasyHandle, GuzzleHttp\Handler\CurlFactory)
#2 /srv/mediawiki/php-1.43.0-wmf.18/vendor/guzzlehttp/guzzle/src/Handler/CurlHandler.php(47): GuzzleHttp\Handler\CurlFactory::finish(GuzzleHttp\Handler\CurlHandler, GuzzleHttp\Handler\EasyHandle, GuzzleHttp\Handler\CurlFactory)
#3 /srv/mediawiki/php-1.43.0-wmf.18/vendor/guzzlehttp/guzzle/src/Handler/Proxy.php(28): GuzzleHttp\Handler\CurlHandler->__invoke(GuzzleHttp\Psr7\Request, array)
#4 /srv/mediawiki/php-1.43.0-wmf.18/vendor/guzzlehttp/guzzle/src/Handler/Proxy.php(48): GuzzleHttp\Handler\Proxy::GuzzleHttp\Handler\{closure}(GuzzleHttp\Psr7\Request, array)
#5 /srv/mediawiki/php-1.43.0-wmf.18/vendor/guzzlehttp/guzzle/src/PrepareBodyMiddleware.php(64): GuzzleHttp\Handler\Proxy::GuzzleHttp\Handler\{closure}(GuzzleHttp\Psr7\Request, array)
#6 /srv/mediawiki/php-1.43.0-wmf.18/vendor/guzzlehttp/guzzle/src/Middleware.php(31): GuzzleHttp\PrepareBodyMiddleware->__invoke(GuzzleHttp\Psr7\Request, array)
#7 /srv/mediawiki/php-1.43.0-wmf.18/vendor/guzzlehttp/guzzle/src/RedirectMiddleware.php(55): GuzzleHttp\Middleware::GuzzleHttp\{closure}(GuzzleHttp\Psr7\Request, array)
#8 /srv/mediawiki/php-1.43.0-wmf.18/vendor/guzzlehttp/guzzle/src/Middleware.php(63): GuzzleHttp\RedirectMiddleware->__invoke(GuzzleHttp\Psr7\Request, array)
#9 /srv/mediawiki/php-1.43.0-wmf.18/vendor/guzzlehttp/guzzle/src/HandlerStack.php(75): GuzzleHttp\Middleware::GuzzleHttp\{closure}(GuzzleHttp\Psr7\Request, array)
#10 /srv/mediawiki/php-1.43.0-wmf.18/vendor/guzzlehttp/guzzle/src/Client.php(333): GuzzleHttp\HandlerStack->__invoke(GuzzleHttp\Psr7\Request, array)
#11 /srv/mediawiki/php-1.43.0-wmf.18/vendor/guzzlehttp/guzzle/src/Client.php(107): GuzzleHttp\Client->transfer(GuzzleHttp\Psr7\Request, array)
#12 /srv/mediawiki/php-1.43.0-wmf.18/vendor/guzzlehttp/guzzle/src/Client.php(138): GuzzleHttp\Client->sendAsync(GuzzleHttp\Psr7\Request, array)
#13 /srv/mediawiki/php-1.43.0-wmf.18/vendor/wikimedia/shellbox/src/Client.php(162): GuzzleHttp\Client->sendRequest(GuzzleHttp\Psr7\Request)
#14 /srv/mediawiki/php-1.43.0-wmf.18/vendor/wikimedia/shellbox/src/Client.php(107): Shellbox\Client->sendRequest(string, array)
#15 /srv/mediawiki/php-1.43.0-wmf.18/extensions/WikibaseQualityConstraints/src/ConstraintCheck/Checker/FormatChecker.php(199): Shellbox\Client->call(string, string, array)
#16 /srv/mediawiki/php-1.43.0-wmf.18/extensions/WikibaseQualityConstraints/src/ConstraintCheck/Checker/FormatChecker.php(181): WikibaseQuality\ConstraintReport\ConstraintCheck\Checker\FormatChecker->runRegexCheckUsingShellbox(string, string)
#17 /srv/mediawiki/php-1.43.0-wmf.18/extensions/WikibaseQualityConstraints/src/ConstraintCheck/Checker/FormatChecker.php(139): WikibaseQuality\ConstraintReport\ConstraintCheck\Checker\FormatChecker->runRegexCheck(string, string)
#18 /srv/mediawiki/php-1.43.0-wmf.18/extensions/WikibaseQualityConstraints/src/ConstraintCheck/DelegatingConstraintChecker.php(568): WikibaseQuality\ConstraintReport\ConstraintCheck\Checker\FormatChecker->checkConstraint(WikibaseQuality\ConstraintReport\ConstraintCheck\Context\MainSnakContext, WikibaseQuality\ConstraintReport\Constraint)
#19 /srv/mediawiki/php-1.43.0-wmf.18/extensions/WikibaseQualityConstraints/src/ConstraintCheck/DelegatingConstraintChecker.php(469): WikibaseQuality\ConstraintReport\ConstraintCheck\DelegatingConstraintChecker->getCheckResultFor(WikibaseQuality\ConstraintReport\ConstraintCheck\Context\MainSnakContext, WikibaseQuality\ConstraintReport\Constraint)
#20 /srv/mediawiki/php-1.43.0-wmf.18/extensions/WikibaseQualityConstraints/src/ConstraintCheck/DelegatingConstraintChecker.php(370): WikibaseQuality\ConstraintReport\ConstraintCheck\DelegatingConstraintChecker->checkConstraintsForMainSnak(Wikibase\DataModel\Entity\Item, Wikibase\DataModel\Statement\Statement, null, array)
#21 /srv/mediawiki/php-1.43.0-wmf.18/extensions/WikibaseQualityConstraints/src/ConstraintCheck/DelegatingConstraintChecker.php(342): WikibaseQuality\ConstraintReport\ConstraintCheck\DelegatingConstraintChecker->checkStatement(Wikibase\DataModel\Entity\Item, Wikibase\DataModel\Statement\Statement, null, array)
#22 /srv/mediawiki/php-1.43.0-wmf.18/extensions/WikibaseQualityConstraints/src/ConstraintCheck/DelegatingConstraintChecker.php(132): WikibaseQuality\ConstraintReport\ConstraintCheck\DelegatingConstraintChecker->checkEveryStatement(Wikibase\DataModel\Entity\Item, null, array)
#23 /srv/mediawiki/php-1.43.0-wmf.18/extensions/WikibaseQualityConstraints/src/Api/CheckingResultsSource.php(53): WikibaseQuality\ConstraintReport\ConstraintCheck\DelegatingConstraintChecker->checkAgainstConstraintsOnEntityId(Wikibase\DataModel\Entity\ItemId, null, array, array)
#24 /srv/mediawiki/php-1.43.0-wmf.18/extensions/WikibaseQualityConstraints/src/Api/CachingResultsSource.php(235): WikibaseQuality\ConstraintReport\Api\CheckingResultsSource->getResults(array, array, null, array)
#25 /srv/mediawiki/php-1.43.0-wmf.18/extensions/WikibaseQualityConstraints/src/Api/CachingResultsSource.php(167): WikibaseQuality\ConstraintReport\Api\CachingResultsSource->getAndStoreResults(array, array, null, array)
#26 /srv/mediawiki/php-1.43.0-wmf.18/extensions/WikibaseQualityConstraints/src/Job/CheckConstraintsJob.php(84): WikibaseQuality\ConstraintReport\Api\CachingResultsSource->getResults(array, array, null, array)
#27 /srv/mediawiki/php-1.43.0-wmf.18/extensions/WikibaseQualityConstraints/src/Job/CheckConstraintsJob.php(74): WikibaseQuality\ConstraintReport\Job\CheckConstraintsJob->checkConstraints(Wikibase\DataModel\Entity\ItemId)
#28 /srv/mediawiki/php-1.43.0-wmf.18/extensions/EventBus/includes/JobExecutor.php(80): WikibaseQuality\ConstraintReport\Job\CheckConstraintsJob->run()
#29 /srv/mediawiki/rpc/RunSingleJob.php(60): MediaWiki\Extension\EventBus\JobExecutor->execute(array)
#30 {main}

[990d4fc2-a75d-42d3-923c-7e8831ff8121] /rpc/RunSingleJob.php   GuzzleHttp\Exception\ConnectException: cURL error 52: Empty reply from server (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for http://localhost:6025/call/constraint-regex-checker
from /srv/mediawiki/php-1.43.0-wmf.18/vendor/guzzlehttp/guzzle/src/Handler/CurlFactory.php(210)

#0 /srv/mediawiki/php-1.43.0-wmf.18/vendor/guzzlehttp/guzzle/src/Handler/CurlFactory.php(158): GuzzleHttp\Handler\CurlFactory::createRejection(GuzzleHttp\Handler\EasyHandle, array)
#1 /srv/mediawiki/php-1.43.0-wmf.18/vendor/guzzlehttp/guzzle/src/Handler/CurlFactory.php(110): GuzzleHttp\Handler\CurlFactory::finishError(GuzzleHttp\Handler\CurlHandler, GuzzleHttp\Handler\EasyHandle, GuzzleHttp\Handler\CurlFactory)
#2 /srv/mediawiki/php-1.43.0-wmf.18/vendor/guzzlehttp/guzzle/src/Handler/CurlHandler.php(47): GuzzleHttp\Handler\CurlFactory::finish(GuzzleHttp\Handler\CurlHandler, GuzzleHttp\Handler\EasyHandle, GuzzleHttp\Handler\CurlFactory)
#3 /srv/mediawiki/php-1.43.0-wmf.18/vendor/guzzlehttp/guzzle/src/Handler/Proxy.php(28): GuzzleHttp\Handler\CurlHandler->__invoke(GuzzleHttp\Psr7\Request, array)
#4 /srv/mediawiki/php-1.43.0-wmf.18/vendor/guzzlehttp/guzzle/src/Handler/Proxy.php(48): GuzzleHttp\Handler\Proxy::GuzzleHttp\Handler\{closure}(GuzzleHttp\Psr7\Request, array)
#5 /srv/mediawiki/php-1.43.0-wmf.18/vendor/guzzlehttp/guzzle/src/PrepareBodyMiddleware.php(64): GuzzleHttp\Handler\Proxy::GuzzleHttp\Handler\{closure}(GuzzleHttp\Psr7\Request, array)
#6 /srv/mediawiki/php-1.43.0-wmf.18/vendor/guzzlehttp/guzzle/src/Middleware.php(31): GuzzleHttp\PrepareBodyMiddleware->__invoke(GuzzleHttp\Psr7\Request, array)
#7 /srv/mediawiki/php-1.43.0-wmf.18/vendor/guzzlehttp/guzzle/src/RedirectMiddleware.php(55): GuzzleHttp\Middleware::GuzzleHttp\{closure}(GuzzleHttp\Psr7\Request, array)
#8 /srv/mediawiki/php-1.43.0-wmf.18/vendor/guzzlehttp/guzzle/src/Middleware.php(63): GuzzleHttp\RedirectMiddleware->__invoke(GuzzleHttp\Psr7\Request, array)
#9 /srv/mediawiki/php-1.43.0-wmf.18/vendor/guzzlehttp/guzzle/src/HandlerStack.php(75): GuzzleHttp\Middleware::GuzzleHttp\{closure}(GuzzleHttp\Psr7\Request, array)
#10 /srv/mediawiki/php-1.43.0-wmf.18/vendor/guzzlehttp/guzzle/src/Client.php(333): GuzzleHttp\HandlerStack->__invoke(GuzzleHttp\Psr7\Request, array)
#11 /srv/mediawiki/php-1.43.0-wmf.18/vendor/guzzlehttp/guzzle/src/Client.php(107): GuzzleHttp\Client->transfer(GuzzleHttp\Psr7\Request, array)
#12 /srv/mediawiki/php-1.43.0-wmf.18/vendor/guzzlehttp/guzzle/src/Client.php(138): GuzzleHttp\Client->sendAsync(GuzzleHttp\Psr7\Request, array)
#13 /srv/mediawiki/php-1.43.0-wmf.18/vendor/wikimedia/shellbox/src/Client.php(162): GuzzleHttp\Client->sendRequest(GuzzleHttp\Psr7\Request)
#14 /srv/mediawiki/php-1.43.0-wmf.18/vendor/wikimedia/shellbox/src/Client.php(107): Shellbox\Client->sendRequest(string, array)
#15 /srv/mediawiki/php-1.43.0-wmf.18/extensions/WikibaseQualityConstraints/src/ConstraintCheck/Checker/FormatChecker.php(199): Shellbox\Client->call(string, string, array)
#16 /srv/mediawiki/php-1.43.0-wmf.18/extensions/WikibaseQualityConstraints/src/ConstraintCheck/Checker/FormatChecker.php(181): WikibaseQuality\ConstraintReport\ConstraintCheck\Checker\FormatChecker->runRegexCheckUsingShellbox(string, string)
#17 /srv/mediawiki/php-1.43.0-wmf.18/extensions/WikibaseQualityConstraints/src/ConstraintCheck/Checker/FormatChecker.php(139): WikibaseQuality\ConstraintReport\ConstraintCheck\Checker\FormatChecker->runRegexCheck(string, string)
#18 /srv/mediawiki/php-1.43.0-wmf.18/extensions/WikibaseQualityConstraints/src/ConstraintCheck/DelegatingConstraintChecker.php(568): WikibaseQuality\ConstraintReport\ConstraintCheck\Checker\FormatChecker->checkConstraint(WikibaseQuality\ConstraintReport\ConstraintCheck\Context\ReferenceContext, WikibaseQuality\ConstraintReport\Constraint)
#19 /srv/mediawiki/php-1.43.0-wmf.18/extensions/WikibaseQualityConstraints/src/ConstraintCheck/DelegatingConstraintChecker.php(546): WikibaseQuality\ConstraintReport\ConstraintCheck\DelegatingConstraintChecker->getCheckResultFor(WikibaseQuality\ConstraintReport\ConstraintCheck\Context\ReferenceContext, WikibaseQuality\ConstraintReport\Constraint)
#20 /srv/mediawiki/php-1.43.0-wmf.18/extensions/WikibaseQualityConstraints/src/ConstraintCheck/DelegatingConstraintChecker.php(389): WikibaseQuality\ConstraintReport\ConstraintCheck\DelegatingConstraintChecker->checkConstraintsForReferences(Wikibase\DataModel\Entity\Item, Wikibase\DataModel\Statement\Statement, null, array)
#21 /srv/mediawiki/php-1.43.0-wmf.18/extensions/WikibaseQualityConstraints/src/ConstraintCheck/DelegatingConstraintChecker.php(342): WikibaseQuality\ConstraintReport\ConstraintCheck\DelegatingConstraintChecker->checkStatement(Wikibase\DataModel\Entity\Item, Wikibase\DataModel\Statement\Statement, null, array)
#22 /srv/mediawiki/php-1.43.0-wmf.18/extensions/WikibaseQualityConstraints/src/ConstraintCheck/DelegatingConstraintChecker.php(132): WikibaseQuality\ConstraintReport\ConstraintCheck\DelegatingConstraintChecker->checkEveryStatement(Wikibase\DataModel\Entity\Item, null, array)
#23 /srv/mediawiki/php-1.43.0-wmf.18/extensions/WikibaseQualityConstraints/src/Api/CheckingResultsSource.php(53): WikibaseQuality\ConstraintReport\ConstraintCheck\DelegatingConstraintChecker->checkAgainstConstraintsOnEntityId(Wikibase\DataModel\Entity\ItemId, null, array, array)
#24 /srv/mediawiki/php-1.43.0-wmf.18/extensions/WikibaseQualityConstraints/src/Api/CachingResultsSource.php(235): WikibaseQuality\ConstraintReport\Api\CheckingResultsSource->getResults(array, array, null, array)
#25 /srv/mediawiki/php-1.43.0-wmf.18/extensions/WikibaseQualityConstraints/src/Api/CachingResultsSource.php(167): WikibaseQuality\ConstraintReport\Api\CachingResultsSource->getAndStoreResults(array, array, null, array)
#26 /srv/mediawiki/php-1.43.0-wmf.18/extensions/WikibaseQualityConstraints/src/Job/CheckConstraintsJob.php(84): WikibaseQuality\ConstraintReport\Api\CachingResultsSource->getResults(array, array, null, array)
#27 /srv/mediawiki/php-1.43.0-wmf.18/extensions/WikibaseQualityConstraints/src/Job/CheckConstraintsJob.php(74): WikibaseQuality\ConstraintReport\Job\CheckConstraintsJob->checkConstraints(Wikibase\DataModel\Entity\ItemId)
#28 /srv/mediawiki/php-1.43.0-wmf.18/extensions/EventBus/includes/JobExecutor.php(80): WikibaseQuality\ConstraintReport\Job\CheckConstraintsJob->run()
#29 /srv/mediawiki/rpc/RunSingleJob.php(60): MediaWiki\Extension\EventBus\JobExecutor->execute(array)
#30 {main}

Details

Related Changes in Gerrit:

Subject	Repo	Branch	Lines +/-
Revert "logspam: Consolidate CurlFactory cURL errors"	operations/puppet	production	+0 -2
logspam: Consolidate CurlFactory cURL errors	operations/puppet	production	+2 -0
Catch ShellboxError and ClientExceptions in FormatChecker	mediawiki/extensions/WikibaseQualityConstraints	master	+81 -27

Customize query in gerrit

Related Objects

Mentioned In: T388460: Frequent Kask read errors
T388180: GuzzleHttp\Exception\ConnectException: cURL error 52: Empty reply from server (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for http://localhost:6025/call/constraint-regex-checker
T388179: GuzzleHttp\Exception\RequestException: cURL error 56: Recv failure: Connection reset by peer (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for http://localhost:6025/call/constraint-regex-checker
T388178: GuzzleHttp\Exception\ConnectException: cURL error 7: Failed to connect to localhost port 6025: Connection refused (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for http://localhost:6025/call/constraint-regex-checker
T377038: Migrate production Shellbox variants to PHP 8.1
T374117: Shellbox bubbles GuzzleHttp\Exception\ConnectException when it should probably wrap it in a ShellboxError?
Mentioned Here: T374117: Shellbox bubbles GuzzleHttp\Exception\ConnectException when it should probably wrap it in a ShellboxError?
T304084: [WBQC] GuzzleHttp\Exception\ConnectException: cURL error 28: Operation timed out after 5000 milliseconds with 0 bytes received (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for http://localhost:6025/call/constraint-regex-checker
T362084: [SW] [WBQC] shellbox-constraints returning 500 on preg_match error
T331609: Gracefully handle pod termination in mw-on-k8s
T249700: k8s/mw: traffic to eventgate dropped by iptables
T341634: MediaWiki frequently receives HTTP 500 from AQS (via PageViewInfo extension)

Event Timeline

dancy created this task.Aug 1 2024, 6:51 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 1 2024, 6:51 PM

dancy updated the task description. (Show Details)Aug 1 2024, 6:55 PM

Krinkle added a project: MediaWiki-libs-HTTP.Aug 20 2024, 4:53 PM

@dancy wrote in the task description:

CurlFactory:211  cURL error 56: Recv failure: Connection reset by peer (see https://curl.haxx.s...
CurlFactory:210  cURL error 7: Failed to connect to localhost port 6025: Connection refused (se...
CurlFactory:210  cURL error 52: Empty reply from server (see https://curl.haxx.se/libcurl/c/lib...
CurlFactory:210  cURL error 7: Failed to connect to localhost port 6027: Connection refused (se...
CurlFactory:211  cURL error 56: Recv failure: Connection reset by peer (see https://curl.haxx.s...

This is a category of errors, rather than a specific problem or source of problems, akin to "Timeout", "Variable undefined", or "Out of memory". These might not be related. I'll repurpose this task to be about the most common one, based on the complete error message, which mentions the http://localhost:6025 service.

I'm going to make the assumption that this isn't a problem with the Guzzle library or the general HTTP request abstraction layer in MediaWiki, but rather due to one of these three:

An issue with how WikibaseQualityConstrains calls the service (e.g. sending bad input).
An issue with how WikibaseQualityConstrains handles failure (e.g. failing to catch or handle "expected" bad responses for bad inputs).
An issue with how the shellbox-constraints service is configured or operated (e.g. this has a certain smell that reminds me of T341634, and T249700).

Last 7 days:

300x cURL error 7: Failed to connect to localhost port 6025: Connection refused
280x cURL error 56: Recv failure: Connection reset by peer
80x cURL error 52: Empty reply from server

[27d39b50-8d32-4e7e-864e-a43a1d522dc6] /w/api.php?action=wbcheckconstraints&format=json&formatversion=2&id=Q706575&status=violation%7Cwarning%7Csuggestion%7Cbad-parameters&uselang=nb   GuzzleHttp\Exception\ConnectException: cURL error 7: Failed to connect to localhost port 6025: Connection refused (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for http://localhost:6025/call/constraint-regex-checker
from /srv/mediawiki/php-1.43.0-wmf.18/vendor/guzzlehttp/guzzle/src/Handler/CurlFactory.php(210)
#0 /srv/mediawiki/php-1.43.0-wmf.18/vendor/guzzlehttp/guzzle/src/Handler/CurlFactory.php(158): GuzzleHttp\Handler\CurlFactory::createRejection(GuzzleHttp\Handler\EasyHandle, array)
#1 /srv/mediawiki/php-1.43.0-wmf.18/vendor/guzzlehttp/guzzle/src/Handler/CurlFactory.php(110): GuzzleHttp\Handler\CurlFactory::finishError(GuzzleHttp\Handler\CurlHandler, GuzzleHttp\Handler\EasyHandle, GuzzleHttp\Handler\CurlFactory)
#2 /srv/mediawiki/php-1.43.0-wmf.18/vendor/guzzlehttp/guzzle/src/Handler/CurlHandler.php(47): GuzzleHttp\Handler\CurlFactory::finish(GuzzleHttp\Handler\CurlHandler, GuzzleHttp\Handler\EasyHandle, GuzzleHttp\Handler\CurlFactory)
#3 /srv/mediawiki/php-1.43.0-wmf.18/vendor/guzzlehttp/guzzle/src/Handler/Proxy.php(28): GuzzleHttp\Handler\CurlHandler->__invoke(GuzzleHttp\Psr7\Request, array)
#4 /srv/mediawiki/php-1.43.0-wmf.18/vendor/guzzlehttp/guzzle/src/Handler/Proxy.php(48): GuzzleHttp\Handler\Proxy::GuzzleHttp\Handler\{closure}(GuzzleHttp\Psr7\Request, array)
#5 /srv/mediawiki/php-1.43.0-wmf.18/vendor/guzzlehttp/guzzle/src/PrepareBodyMiddleware.php(64): GuzzleHttp\Handler\Proxy::GuzzleHttp\Handler\{closure}(GuzzleHttp\Psr7\Request, array)
#6 /srv/mediawiki/php-1.43.0-wmf.18/vendor/guzzlehttp/guzzle/src/Middleware.php(31): GuzzleHttp\PrepareBodyMiddleware->__invoke(GuzzleHttp\Psr7\Request, array)
#7 /srv/mediawiki/php-1.43.0-wmf.18/vendor/guzzlehttp/guzzle/src/RedirectMiddleware.php(55): GuzzleHttp\Middleware::GuzzleHttp\{closure}(GuzzleHttp\Psr7\Request, array)
#8 /srv/mediawiki/php-1.43.0-wmf.18/vendor/guzzlehttp/guzzle/src/Middleware.php(63): GuzzleHttp\RedirectMiddleware->__invoke(GuzzleHttp\Psr7\Request, array)
#9 /srv/mediawiki/php-1.43.0-wmf.18/vendor/guzzlehttp/guzzle/src/HandlerStack.php(75): GuzzleHttp\Middleware::GuzzleHttp\{closure}(GuzzleHttp\Psr7\Request, array)
#10 /srv/mediawiki/php-1.43.0-wmf.18/vendor/guzzlehttp/guzzle/src/Client.php(333): GuzzleHttp\HandlerStack->__invoke(GuzzleHttp\Psr7\Request, array)
#11 /srv/mediawiki/php-1.43.0-wmf.18/vendor/guzzlehttp/guzzle/src/Client.php(107): GuzzleHttp\Client->transfer(GuzzleHttp\Psr7\Request, array)
#12 /srv/mediawiki/php-1.43.0-wmf.18/vendor/guzzlehttp/guzzle/src/Client.php(138): GuzzleHttp\Client->sendAsync(GuzzleHttp\Psr7\Request, array)
#13 /srv/mediawiki/php-1.43.0-wmf.18/vendor/wikimedia/shellbox/src/Client.php(162): GuzzleHttp\Client->sendRequest(GuzzleHttp\Psr7\Request)
#14 /srv/mediawiki/php-1.43.0-wmf.18/vendor/wikimedia/shellbox/src/Client.php(107): Shellbox\Client->sendRequest(string, array)
#15 /srv/mediawiki/php-1.43.0-wmf.18/extensions/WikibaseQualityConstraints/src/ConstraintCheck/Checker/FormatChecker.php(199): Shellbox\Client->call(string, string, array)
#16 /srv/mediawiki/php-1.43.0-wmf.18/extensions/WikibaseQualityConstraints/src/ConstraintCheck/Checker/FormatChecker.php(181): WikibaseQuality\ConstraintReport\ConstraintCheck\Checker\FormatChecker->runRegexCheckUsingShellbox(string, string)
#17 /srv/mediawiki/php-1.43.0-wmf.18/extensions/WikibaseQualityConstraints/src/ConstraintCheck/Checker/FormatChecker.php(139): WikibaseQuality\ConstraintReport\ConstraintCheck\Checker\FormatChecker->runRegexCheck(string, string)
#18 /srv/mediawiki/php-1.43.0-wmf.18/extensions/WikibaseQualityConstraints/src/ConstraintCheck/DelegatingConstraintChecker.php(568): WikibaseQuality\ConstraintReport\ConstraintCheck\Checker\FormatChecker->checkConstraint(WikibaseQuality\ConstraintReport\ConstraintCheck\Context\MainSnakContext, WikibaseQuality\ConstraintReport\Constraint)
#19 /srv/mediawiki/php-1.43.0-wmf.18/extensions/WikibaseQualityConstraints/src/ConstraintCheck/DelegatingConstraintChecker.php(469): WikibaseQuality\ConstraintReport\ConstraintCheck\DelegatingConstraintChecker->getCheckResultFor(WikibaseQuality\ConstraintReport\ConstraintCheck\Context\MainSnakContext, WikibaseQuality\ConstraintReport\Constraint)
#20 /srv/mediawiki/php-1.43.0-wmf.18/extensions/WikibaseQualityConstraints/src/ConstraintCheck/DelegatingConstraintChecker.php(370): WikibaseQuality\ConstraintReport\ConstraintCheck\DelegatingConstraintChecker->checkConstraintsForMainSnak(Wikibase\DataModel\Entity\Item, Wikibase\DataModel\Statement\Statement, null, array)
#21 /srv/mediawiki/php-1.43.0-wmf.18/extensions/WikibaseQualityConstraints/src/ConstraintCheck/DelegatingConstraintChecker.php(342): WikibaseQuality\ConstraintReport\ConstraintCheck\DelegatingConstraintChecker->checkStatement(Wikibase\DataModel\Entity\Item, Wikibase\DataModel\Statement\Statement, null, array)
#22 /srv/mediawiki/php-1.43.0-wmf.18/extensions/WikibaseQualityConstraints/src/ConstraintCheck/DelegatingConstraintChecker.php(132): WikibaseQuality\ConstraintReport\ConstraintCheck\DelegatingConstraintChecker->checkEveryStatement(Wikibase\DataModel\Entity\Item, null, array)
#23 /srv/mediawiki/php-1.43.0-wmf.18/extensions/WikibaseQualityConstraints/src/Api/CheckingResultsSource.php(53): WikibaseQuality\ConstraintReport\ConstraintCheck\DelegatingConstraintChecker->checkAgainstConstraintsOnEntityId(Wikibase\DataModel\Entity\ItemId, null, array, array)
#24 /srv/mediawiki/php-1.43.0-wmf.18/extensions/WikibaseQualityConstraints/src/Api/CachingResultsSource.php(235): WikibaseQuality\ConstraintReport\Api\CheckingResultsSource->getResults(array, array, null, array)
#25 /srv/mediawiki/php-1.43.0-wmf.18/extensions/WikibaseQualityConstraints/src/Api/CachingResultsSource.php(167): WikibaseQuality\ConstraintReport\Api\CachingResultsSource->getAndStoreResults(array, array, null, array)
#26 /srv/mediawiki/php-1.43.0-wmf.18/extensions/WikibaseQualityConstraints/src/Api/CheckConstraints.php(121): WikibaseQuality\ConstraintReport\Api\CachingResultsSource->getResults(array, array, null, array)
#27 /srv/mediawiki/php-1.43.0-wmf.18/includes/api/ApiMain.php(1948): WikibaseQuality\ConstraintReport\Api\CheckConstraints->execute()
#28 /srv/mediawiki/php-1.43.0-wmf.18/includes/api/ApiMain.php(926): ApiMain->executeAction()
#29 /srv/mediawiki/php-1.43.0-wmf.18/includes/api/ApiMain.php(897): ApiMain->executeActionWithErrorHandling()
#30 /srv/mediawiki/php-1.43.0-wmf.18/includes/api/ApiEntryPoint.php(153): ApiMain->execute()
#31 /srv/mediawiki/php-1.43.0-wmf.18/includes/MediaWikiEntryPoint.php(200): MediaWiki\Api\ApiEntryPoint->execute()
#32 /srv/mediawiki/php-1.43.0-wmf.18/api.php(44): MediaWiki\MediaWikiEntryPoint->run()
#33 /srv/mediawiki/w/api.php(3): require(string)
#34 {main}

[90652139-0d8e-4b3e-a0ef-15cf1d36e145] /rpc/RunSingleJob.php   GuzzleHttp\Exception\RequestException: cURL error 56: Recv failure: Connection reset by peer (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for http://localhost:6025/call/constraint-regex-checker
from /srv/mediawiki/php-1.43.0-wmf.18/vendor/guzzlehttp/guzzle/src/Handler/CurlFactory.php(211)
#0 /srv/mediawiki/php-1.43.0-wmf.18/vendor/guzzlehttp/guzzle/src/Handler/CurlFactory.php(158): GuzzleHttp\Handler\CurlFactory::createRejection(GuzzleHttp\Handler\EasyHandle, array)
#1 /srv/mediawiki/php-1.43.0-wmf.18/vendor/guzzlehttp/guzzle/src/Handler/CurlFactory.php(110): GuzzleHttp\Handler\CurlFactory::finishError(GuzzleHttp\Handler\CurlHandler, GuzzleHttp\Handler\EasyHandle, GuzzleHttp\Handler\CurlFactory)
#2 /srv/mediawiki/php-1.43.0-wmf.18/vendor/guzzlehttp/guzzle/src/Handler/CurlHandler.php(47): GuzzleHttp\Handler\CurlFactory::finish(GuzzleHttp\Handler\CurlHandler, GuzzleHttp\Handler\EasyHandle, GuzzleHttp\Handler\CurlFactory)
#3 /srv/mediawiki/php-1.43.0-wmf.18/vendor/guzzlehttp/guzzle/src/Handler/Proxy.php(28): GuzzleHttp\Handler\CurlHandler->__invoke(GuzzleHttp\Psr7\Request, array)
#4 /srv/mediawiki/php-1.43.0-wmf.18/vendor/guzzlehttp/guzzle/src/Handler/Proxy.php(48): GuzzleHttp\Handler\Proxy::GuzzleHttp\Handler\{closure}(GuzzleHttp\Psr7\Request, array)
#5 /srv/mediawiki/php-1.43.0-wmf.18/vendor/guzzlehttp/guzzle/src/PrepareBodyMiddleware.php(64): GuzzleHttp\Handler\Proxy::GuzzleHttp\Handler\{closure}(GuzzleHttp\Psr7\Request, array)
#6 /srv/mediawiki/php-1.43.0-wmf.18/vendor/guzzlehttp/guzzle/src/Middleware.php(31): GuzzleHttp\PrepareBodyMiddleware->__invoke(GuzzleHttp\Psr7\Request, array)
#7 /srv/mediawiki/php-1.43.0-wmf.18/vendor/guzzlehttp/guzzle/src/RedirectMiddleware.php(55): GuzzleHttp\Middleware::GuzzleHttp\{closure}(GuzzleHttp\Psr7\Request, array)
#8 /srv/mediawiki/php-1.43.0-wmf.18/vendor/guzzlehttp/guzzle/src/Middleware.php(63): GuzzleHttp\RedirectMiddleware->__invoke(GuzzleHttp\Psr7\Request, array)
#9 /srv/mediawiki/php-1.43.0-wmf.18/vendor/guzzlehttp/guzzle/src/HandlerStack.php(75): GuzzleHttp\Middleware::GuzzleHttp\{closure}(GuzzleHttp\Psr7\Request, array)
#10 /srv/mediawiki/php-1.43.0-wmf.18/vendor/guzzlehttp/guzzle/src/Client.php(333): GuzzleHttp\HandlerStack->__invoke(GuzzleHttp\Psr7\Request, array)
#11 /srv/mediawiki/php-1.43.0-wmf.18/vendor/guzzlehttp/guzzle/src/Client.php(107): GuzzleHttp\Client->transfer(GuzzleHttp\Psr7\Request, array)
#12 /srv/mediawiki/php-1.43.0-wmf.18/vendor/guzzlehttp/guzzle/src/Client.php(138): GuzzleHttp\Client->sendAsync(GuzzleHttp\Psr7\Request, array)
#13 /srv/mediawiki/php-1.43.0-wmf.18/vendor/wikimedia/shellbox/src/Client.php(162): GuzzleHttp\Client->sendRequest(GuzzleHttp\Psr7\Request)
#14 /srv/mediawiki/php-1.43.0-wmf.18/vendor/wikimedia/shellbox/src/Client.php(107): Shellbox\Client->sendRequest(string, array)
#15 /srv/mediawiki/php-1.43.0-wmf.18/extensions/WikibaseQualityConstraints/src/ConstraintCheck/Checker/FormatChecker.php(199): Shellbox\Client->call(string, string, array)
#16 /srv/mediawiki/php-1.43.0-wmf.18/extensions/WikibaseQualityConstraints/src/ConstraintCheck/Checker/FormatChecker.php(181): WikibaseQuality\ConstraintReport\ConstraintCheck\Checker\FormatChecker->runRegexCheckUsingShellbox(string, string)
#17 /srv/mediawiki/php-1.43.0-wmf.18/extensions/WikibaseQualityConstraints/src/ConstraintCheck/Checker/FormatChecker.php(139): WikibaseQuality\ConstraintReport\ConstraintCheck\Checker\FormatChecker->runRegexCheck(string, string)
#18 /srv/mediawiki/php-1.43.0-wmf.18/extensions/WikibaseQualityConstraints/src/ConstraintCheck/DelegatingConstraintChecker.php(568): WikibaseQuality\ConstraintReport\ConstraintCheck\Checker\FormatChecker->checkConstraint(WikibaseQuality\ConstraintReport\ConstraintCheck\Context\MainSnakContext, WikibaseQuality\ConstraintReport\Constraint)
#19 /srv/mediawiki/php-1.43.0-wmf.18/extensions/WikibaseQualityConstraints/src/ConstraintCheck/DelegatingConstraintChecker.php(469): WikibaseQuality\ConstraintReport\ConstraintCheck\DelegatingConstraintChecker->getCheckResultFor(WikibaseQuality\ConstraintReport\ConstraintCheck\Context\MainSnakContext, WikibaseQuality\ConstraintReport\Constraint)
#20 /srv/mediawiki/php-1.43.0-wmf.18/extensions/WikibaseQualityConstraints/src/ConstraintCheck/DelegatingConstraintChecker.php(370): WikibaseQuality\ConstraintReport\ConstraintCheck\DelegatingConstraintChecker->checkConstraintsForMainSnak(Wikibase\DataModel\Entity\Item, Wikibase\DataModel\Statement\Statement, null, array)
#21 /srv/mediawiki/php-1.43.0-wmf.18/extensions/WikibaseQualityConstraints/src/ConstraintCheck/DelegatingConstraintChecker.php(342): WikibaseQuality\ConstraintReport\ConstraintCheck\DelegatingConstraintChecker->checkStatement(Wikibase\DataModel\Entity\Item, Wikibase\DataModel\Statement\Statement, null, array)
#22 /srv/mediawiki/php-1.43.0-wmf.18/extensions/WikibaseQualityConstraints/src/ConstraintCheck/DelegatingConstraintChecker.php(132): WikibaseQuality\ConstraintReport\ConstraintCheck\DelegatingConstraintChecker->checkEveryStatement(Wikibase\DataModel\Entity\Item, null, array)
#23 /srv/mediawiki/php-1.43.0-wmf.18/extensions/WikibaseQualityConstraints/src/Api/CheckingResultsSource.php(53): WikibaseQuality\ConstraintReport\ConstraintCheck\DelegatingConstraintChecker->checkAgainstConstraintsOnEntityId(Wikibase\DataModel\Entity\ItemId, null, array, array)
#24 /srv/mediawiki/php-1.43.0-wmf.18/extensions/WikibaseQualityConstraints/src/Api/CachingResultsSource.php(235): WikibaseQuality\ConstraintReport\Api\CheckingResultsSource->getResults(array, array, null, array)
#25 /srv/mediawiki/php-1.43.0-wmf.18/extensions/WikibaseQualityConstraints/src/Api/CachingResultsSource.php(167): WikibaseQuality\ConstraintReport\Api\CachingResultsSource->getAndStoreResults(array, array, null, array)
#26 /srv/mediawiki/php-1.43.0-wmf.18/extensions/WikibaseQualityConstraints/src/Job/CheckConstraintsJob.php(84): WikibaseQuality\ConstraintReport\Api\CachingResultsSource->getResults(array, array, null, array)
#27 /srv/mediawiki/php-1.43.0-wmf.18/extensions/WikibaseQualityConstraints/src/Job/CheckConstraintsJob.php(74): WikibaseQuality\ConstraintReport\Job\CheckConstraintsJob->checkConstraints(Wikibase\DataModel\Entity\ItemId)
#28 /srv/mediawiki/php-1.43.0-wmf.18/extensions/EventBus/includes/JobExecutor.php(80): WikibaseQuality\ConstraintReport\Job\CheckConstraintsJob->run()
#29 /srv/mediawiki/rpc/RunSingleJob.php(60): MediaWiki\Extension\EventBus\JobExecutor->execute(array)
#30 {main}

[990d4fc2-a75d-42d3-923c-7e8831ff8121] /rpc/RunSingleJob.php   GuzzleHttp\Exception\ConnectException: cURL error 52: Empty reply from server (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for http://localhost:6025/call/constraint-regex-checker
from /srv/mediawiki/php-1.43.0-wmf.18/vendor/guzzlehttp/guzzle/src/Handler/CurlFactory.php(210)
#0 /srv/mediawiki/php-1.43.0-wmf.18/vendor/guzzlehttp/guzzle/src/Handler/CurlFactory.php(158): GuzzleHttp\Handler\CurlFactory::createRejection(GuzzleHttp\Handler\EasyHandle, array)
#1 /srv/mediawiki/php-1.43.0-wmf.18/vendor/guzzlehttp/guzzle/src/Handler/CurlFactory.php(110): GuzzleHttp\Handler\CurlFactory::finishError(GuzzleHttp\Handler\CurlHandler, GuzzleHttp\Handler\EasyHandle, GuzzleHttp\Handler\CurlFactory)
#2 /srv/mediawiki/php-1.43.0-wmf.18/vendor/guzzlehttp/guzzle/src/Handler/CurlHandler.php(47): GuzzleHttp\Handler\CurlFactory::finish(GuzzleHttp\Handler\CurlHandler, GuzzleHttp\Handler\EasyHandle, GuzzleHttp\Handler\CurlFactory)
#3 /srv/mediawiki/php-1.43.0-wmf.18/vendor/guzzlehttp/guzzle/src/Handler/Proxy.php(28): GuzzleHttp\Handler\CurlHandler->__invoke(GuzzleHttp\Psr7\Request, array)
#4 /srv/mediawiki/php-1.43.0-wmf.18/vendor/guzzlehttp/guzzle/src/Handler/Proxy.php(48): GuzzleHttp\Handler\Proxy::GuzzleHttp\Handler\{closure}(GuzzleHttp\Psr7\Request, array)
#5 /srv/mediawiki/php-1.43.0-wmf.18/vendor/guzzlehttp/guzzle/src/PrepareBodyMiddleware.php(64): GuzzleHttp\Handler\Proxy::GuzzleHttp\Handler\{closure}(GuzzleHttp\Psr7\Request, array)
#6 /srv/mediawiki/php-1.43.0-wmf.18/vendor/guzzlehttp/guzzle/src/Middleware.php(31): GuzzleHttp\PrepareBodyMiddleware->__invoke(GuzzleHttp\Psr7\Request, array)
#7 /srv/mediawiki/php-1.43.0-wmf.18/vendor/guzzlehttp/guzzle/src/RedirectMiddleware.php(55): GuzzleHttp\Middleware::GuzzleHttp\{closure}(GuzzleHttp\Psr7\Request, array)
#8 /srv/mediawiki/php-1.43.0-wmf.18/vendor/guzzlehttp/guzzle/src/Middleware.php(63): GuzzleHttp\RedirectMiddleware->__invoke(GuzzleHttp\Psr7\Request, array)
#9 /srv/mediawiki/php-1.43.0-wmf.18/vendor/guzzlehttp/guzzle/src/HandlerStack.php(75): GuzzleHttp\Middleware::GuzzleHttp\{closure}(GuzzleHttp\Psr7\Request, array)
#10 /srv/mediawiki/php-1.43.0-wmf.18/vendor/guzzlehttp/guzzle/src/Client.php(333): GuzzleHttp\HandlerStack->__invoke(GuzzleHttp\Psr7\Request, array)
#11 /srv/mediawiki/php-1.43.0-wmf.18/vendor/guzzlehttp/guzzle/src/Client.php(107): GuzzleHttp\Client->transfer(GuzzleHttp\Psr7\Request, array)
#12 /srv/mediawiki/php-1.43.0-wmf.18/vendor/guzzlehttp/guzzle/src/Client.php(138): GuzzleHttp\Client->sendAsync(GuzzleHttp\Psr7\Request, array)
#13 /srv/mediawiki/php-1.43.0-wmf.18/vendor/wikimedia/shellbox/src/Client.php(162): GuzzleHttp\Client->sendRequest(GuzzleHttp\Psr7\Request)
#14 /srv/mediawiki/php-1.43.0-wmf.18/vendor/wikimedia/shellbox/src/Client.php(107): Shellbox\Client->sendRequest(string, array)
#15 /srv/mediawiki/php-1.43.0-wmf.18/extensions/WikibaseQualityConstraints/src/ConstraintCheck/Checker/FormatChecker.php(199): Shellbox\Client->call(string, string, array)
#16 /srv/mediawiki/php-1.43.0-wmf.18/extensions/WikibaseQualityConstraints/src/ConstraintCheck/Checker/FormatChecker.php(181): WikibaseQuality\ConstraintReport\ConstraintCheck\Checker\FormatChecker->runRegexCheckUsingShellbox(string, string)
#17 /srv/mediawiki/php-1.43.0-wmf.18/extensions/WikibaseQualityConstraints/src/ConstraintCheck/Checker/FormatChecker.php(139): WikibaseQuality\ConstraintReport\ConstraintCheck\Checker\FormatChecker->runRegexCheck(string, string)
#18 /srv/mediawiki/php-1.43.0-wmf.18/extensions/WikibaseQualityConstraints/src/ConstraintCheck/DelegatingConstraintChecker.php(568): WikibaseQuality\ConstraintReport\ConstraintCheck\Checker\FormatChecker->checkConstraint(WikibaseQuality\ConstraintReport\ConstraintCheck\Context\ReferenceContext, WikibaseQuality\ConstraintReport\Constraint)
#19 /srv/mediawiki/php-1.43.0-wmf.18/extensions/WikibaseQualityConstraints/src/ConstraintCheck/DelegatingConstraintChecker.php(546): WikibaseQuality\ConstraintReport\ConstraintCheck\DelegatingConstraintChecker->getCheckResultFor(WikibaseQuality\ConstraintReport\ConstraintCheck\Context\ReferenceContext, WikibaseQuality\ConstraintReport\Constraint)
#20 /srv/mediawiki/php-1.43.0-wmf.18/extensions/WikibaseQualityConstraints/src/ConstraintCheck/DelegatingConstraintChecker.php(389): WikibaseQuality\ConstraintReport\ConstraintCheck\DelegatingConstraintChecker->checkConstraintsForReferences(Wikibase\DataModel\Entity\Item, Wikibase\DataModel\Statement\Statement, null, array)
#21 /srv/mediawiki/php-1.43.0-wmf.18/extensions/WikibaseQualityConstraints/src/ConstraintCheck/DelegatingConstraintChecker.php(342): WikibaseQuality\ConstraintReport\ConstraintCheck\DelegatingConstraintChecker->checkStatement(Wikibase\DataModel\Entity\Item, Wikibase\DataModel\Statement\Statement, null, array)
#22 /srv/mediawiki/php-1.43.0-wmf.18/extensions/WikibaseQualityConstraints/src/ConstraintCheck/DelegatingConstraintChecker.php(132): WikibaseQuality\ConstraintReport\ConstraintCheck\DelegatingConstraintChecker->checkEveryStatement(Wikibase\DataModel\Entity\Item, null, array)
#23 /srv/mediawiki/php-1.43.0-wmf.18/extensions/WikibaseQualityConstraints/src/Api/CheckingResultsSource.php(53): WikibaseQuality\ConstraintReport\ConstraintCheck\DelegatingConstraintChecker->checkAgainstConstraintsOnEntityId(Wikibase\DataModel\Entity\ItemId, null, array, array)
#24 /srv/mediawiki/php-1.43.0-wmf.18/extensions/WikibaseQualityConstraints/src/Api/CachingResultsSource.php(235): WikibaseQuality\ConstraintReport\Api\CheckingResultsSource->getResults(array, array, null, array)
#25 /srv/mediawiki/php-1.43.0-wmf.18/extensions/WikibaseQualityConstraints/src/Api/CachingResultsSource.php(167): WikibaseQuality\ConstraintReport\Api\CachingResultsSource->getAndStoreResults(array, array, null, array)
#26 /srv/mediawiki/php-1.43.0-wmf.18/extensions/WikibaseQualityConstraints/src/Job/CheckConstraintsJob.php(84): WikibaseQuality\ConstraintReport\Api\CachingResultsSource->getResults(array, array, null, array)
#27 /srv/mediawiki/php-1.43.0-wmf.18/extensions/WikibaseQualityConstraints/src/Job/CheckConstraintsJob.php(74): WikibaseQuality\ConstraintReport\Job\CheckConstraintsJob->checkConstraints(Wikibase\DataModel\Entity\ItemId)
#28 /srv/mediawiki/php-1.43.0-wmf.18/extensions/EventBus/includes/JobExecutor.php(80): WikibaseQuality\ConstraintReport\Job\CheckConstraintsJob->run()
#29 /srv/mediawiki/rpc/RunSingleJob.php(60): MediaWiki\Extension\EventBus\JobExecutor->execute(array)
#30 {main}

Krinkle added projects: Wikimedia-production-error, Wikidata-Campsite.Aug 20 2024, 6:22 PM

Krinkle renamed this task from Burst of curlFactory cURL errors during mediawiki deployments to Burst of GuzzleHttp Exception for http://localhost:6025/call/constraint-regex-checker.Aug 20 2024, 6:24 PM

Krinkle updated the task description. (Show Details)

Krinkle moved this task from Untriaged to Aug 2024 on the Wikimedia-production-error board.

This is most likely caused by envoy terminating before mediawiki.

These are generated by a terminating pod, for instance this sequence of kubernetes events picked up from this burst of errors shows that the mediawiki container has stopped *after* envoy has terminated, meaning there were probably requests that were still being processed by mediawiki while the service mesh was unavailable.

We tried to address this problem in T331609, but apart from playing with timings I don't think there is a lot we can do until we have a kubernetes version that supports the SidecarContainers feature gate.

It's also a bit of a chicken and egg problem, as envoy handles both tls termination for incoming connections and the service mesh. We could try to drain inbound listeners on pod termination which, if I understand correctly, would stop accepting new connections to the tls termination side while finishing up serving outbound connections.

To summarize, I don't think we can completely eliminate dropping requests at some point in the path when terminating pods for a deployment, the question is where do we drop them.

Lydia_Pintscher edited projects, added wmde-wikidata-tech, Wikidata; removed Wikidata-Campsite.Aug 21 2024, 4:16 PM

Lydia_Pintscher added a project: [Archived]Wikidata Dev Team.

karapayneWMDE moved this task from Incoming to [DOT] Tech Backlog on the [Archived]Wikidata Dev Team board.Aug 27 2024, 2:20 PM

Is this somehow related to T362084: [SW] [WBQC] shellbox-constraints returning 500 on preg_match error?

In T371633#10102614, @ItamarWMDE wrote:

Is this somehow related to T362084: [SW] [WBQC] shellbox-constraints returning 500 on preg_match error?

No.

Restricted Application added a project: User-ItamarWMDE. · View Herald TranscriptAug 30 2024, 1:13 PM

Let me add some prospect, as I've heard people are complaining about this.

When we started not revalidating opcache, we had to restart php-fpm on bare metal during deployments. This meant that, in order to avoid deployments lasting too long, we'd use a 3 seconds grace period between when the node was not receiving more traffic from the load balancer and when we killed the process, thus killing all live requests that were lasting more than 3 seconds.

On kubernetes, we do the same, with the only difference being that, being k8s much better at deploying payloads, we can afford a longer grace period. Sometimes requests last longer than this grace period and we start tearing down containers in the pod, and in that situation you could see this kind of errors. On bare metal you would not see them, but the result would've been the same: the original HTTP request will be served a 503. If anything, the situation is better now than it was on bare metal.

I'll leave the task open because when we can use SidecarContainers, we can verify this doesn't happen anymore. But let me reiterate: this has no actual effect on the fate of HTTP requests, which if anything are allowed more time to complete on kubernetes.

ItamarWMDE moved this task from Incoming to Radar on the wmde-wikidata-tech board.Sep 3 2024, 8:02 AM

In T371633#10078899, @Krinkle wrote:

I'm going to make the assumption that this isn't a problem with the Guzzle library or the general HTTP request abstraction layer in MediaWiki, but rather due to one of these three:

An issue with how WikibaseQualityConstrains calls the service (e.g. sending bad input).

An issue with how WikibaseQualityConstrains handles failure (e.g. failing to catch or handle "expected" bad responses for bad inputs).

An issue with how the shellbox-constraints service is configured or operated (e.g. this has a certain smell that reminds me of T341634, and T249700).

I still think the problem is in Shellbox, and is very similar to that described in T304084#8094707 / T304084#8094758 / T304084#8561863: Shellbox\RPC\RpcClient::call() is documented to throw ShellboxError; WikibaseQualityConstraints catches ShellboxError; shellbox actually throws various underlying Guzzle exceptions without wrapping them in ShellboxError, so we end up with uncaught exceptions. (All the exceptions seen here seem to extend Psr\Http\Client\ClientExceptionInterface, so it should be possible to cover them all with one catch block, it’s just not clear who’s supposed to be catching them.)

Lucas_Werkmeister_WMDE mentioned this in T374117: Shellbox bubbles GuzzleHttp\Exception\ConnectException when it should probably wrap it in a ShellboxError?.Sep 5 2024, 2:08 PM

I have filed T374117 which is about extension usually catching ShellboxError while ShellBox lets low level exception pass through when they should probably be wrapped instead.

removed from dev board until its can go through our prioritization pipeline

Restricted Application changed the subtype of this task from "Task" to "Production Error". · View Herald TranscriptNov 29 2024, 11:51 AM

Change #1056221 had a related patch set uploaded (by Ahmon Dancy; author: Ahmon Dancy):

[operations/puppet@production] logspam: Consolidate CurlFactory cURL errors

https://gerrit.wikimedia.org/r/1056221

gerritbot added a project: Patch-For-Review.Feb 18 2025, 8:07 PM

Change #1056221 merged by Dzahn:

[operations/puppet@production] logspam: Consolidate CurlFactory cURL errors

https://gerrit.wikimedia.org/r/1056221

Maintenance_bot removed a project: Patch-For-Review.Feb 19 2025, 6:31 PM

dancy triaged this task as Low priority.Feb 19 2025, 7:55 PM

Scott_French mentioned this in T377038: Migrate production Shellbox variants to PHP 8.1.Mar 6 2025, 11:03 PM

Lucas_Werkmeister_WMDE mentioned this in T388178: GuzzleHttp\Exception\ConnectException: cURL error 7: Failed to connect to localhost port 6025: Connection refused (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for http://localhost:6025/call/constraint-regex-checker.Mar 7 2025, 10:13 AM

Lucas_Werkmeister_WMDE mentioned this in T388179: GuzzleHttp\Exception\RequestException: cURL error 56: Recv failure: Connection reset by peer (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for http://localhost:6025/call/constraint-regex-checker.

Lucas_Werkmeister_WMDE mentioned this in T388180: GuzzleHttp\Exception\ConnectException: cURL error 52: Empty reply from server (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for http://localhost:6025/call/constraint-regex-checker.

Change #1125397 had a related patch set uploaded (by Lucas Werkmeister (WMDE); author: Lucas Werkmeister (WMDE)):

[mediawiki/extensions/WikibaseQualityConstraints@master] Catch ShellboxError in FormatChecker again

https://gerrit.wikimedia.org/r/1125397

gerritbot added a project: Patch-For-Review.Mar 7 2025, 10:50 AM

jijiki moved this task from Incoming 🐫 to 🌻Mediawiki on the serviceops-deprecated board.Mar 11 2025, 10:14 AM

Lucas_Werkmeister_WMDE updated the task description. (Show Details)Mar 12 2025, 2:24 PM

To do: Update the gerrit change to catch the ClientExceptionInterface

A/C

Error with "constraint-regex-checker" no longer appears
Error no longer appears after 7 days of the fix

https://logstash.wikimedia.org/app/dashboards#/view/mediawiki-errors?_g=h@75c69a9&_a=h@1257990

karapayneWMDE added a project: Wikidata-Omega.Mar 12 2025, 2:26 PM

karapayneWMDE moved this task from Incoming to The Backlog on the Wikidata-Omega board.

Eevans mentioned this in T388460: Frequent Kask read errors.Mar 13 2025, 7:05 PM

ArthurTaylor claimed this task.Mar 24 2025, 11:55 AM

ArthurTaylor moved this task from The Backlog to The Board on the Wikidata-Omega board.

ArthurTaylor edited projects, added Wikidata-Omega (The Board); removed Wikidata-Omega.

ArthurTaylor moved this task from new to the board to In Development on the Wikidata-Omega (The Board) board.Mar 24 2025, 11:58 AM

ArthurTaylor removed ArthurTaylor as the assignee of this task.Mar 24 2025, 1:04 PM

ArthurTaylor moved this task from In Development to Ready for Peer Review on the Wikidata-Omega (The Board) board.

ArthurTaylor subscribed.

AudreyPenven_WMDE claimed this task.Mar 28 2025, 2:04 PM

AudreyPenven_WMDE moved this task from Ready for Peer Review to In Peer Review on the Wikidata-Omega (The Board) board.

Change #1125397 merged by jenkins-bot:

[mediawiki/extensions/WikibaseQualityConstraints@master] Catch ShellboxError and ClientExceptions in FormatChecker

https://gerrit.wikimedia.org/r/1125397

AudreyPenven_WMDE removed AudreyPenven_WMDE as the assignee of this task.Mar 28 2025, 2:16 PM

AudreyPenven_WMDE moved this task from In Peer Review to Ready for Tech Verification on the Wikidata-Omega (The Board) board.

AudreyPenven_WMDE subscribed.

Maintenance_bot removed a project: Patch-For-Review.Mar 28 2025, 2:30 PM

ReleaseTaggerBot added a project: MW-1.44-notes (1.44.0-wmf.23; 2025-04-01).Mar 28 2025, 3:00 PM

karapayneWMDE closed this task as Resolved.Apr 4 2025, 10:15 AM

karapayneWMDE claimed this task.

karapayneWMDE moved this task from Ready for Tech Verification to In Tech Verification on the Wikidata-Omega (The Board) board.

karapayneWMDE moved this task from In Tech Verification to Done - Moving a task here will move it to the completed tasks workboard, resolve the ticket, and assign it to whomever made the move on the Wikidata-Omega (The Board) board.

karapayneWMDE edited projects, added Wikidata-Omega (Completed Tasks); removed Wikidata-Omega (The Board).

karapayneWMDE moved this task from Newly completed tasks to 2025 Q1 on the Wikidata-Omega (Completed Tasks) board.Apr 4 2025, 10:39 AM

In T371633#10628754, @karapayneWMDE wrote:

Error no longer appears after 7 days of the fix

Now that it’s been 7 days since the fix was deployed, I can confirm that the errors are gone from logstash.

Change #1135690 had a related patch set uploaded (by Lucas Werkmeister (WMDE); author: Lucas Werkmeister (WMDE)):

[operations/puppet@production] Revert "logspam: Consolidate CurlFactory cURL errors"

https://gerrit.wikimedia.org/r/1135690

gerritbot added a project: Patch-For-Review.Apr 10 2025, 10:54 AM

Thanks to everyone who worked to take care of these errors!

Change #1135690 merged by Dzahn: