Page MenuHomePhabricator

[WBQC] GuzzleHttp\Exception\ConnectException: cURL error 28: Operation timed out after 5000 milliseconds with 0 bytes received (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for http://localhost:6025/call/constraint-regex-checker
Open, Needs TriagePublicPRODUCTION ERROR

Description

Error
normalized_message
[{reqId}] {exception_url}   GuzzleHttp\Exception\ConnectException: cURL error 28: Operation timed out after 5000 milliseconds with 0 bytes received (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for http://localhost:6025/call/constraint-regex-ch
exception.trace
from /srv/mediawiki/php-1.38.0-wmf.25/vendor/guzzlehttp/guzzle/src/Handler/CurlFactory.php(210)
#0 /srv/mediawiki/php-1.38.0-wmf.25/vendor/guzzlehttp/guzzle/src/Handler/CurlFactory.php(158): GuzzleHttp\Handler\CurlFactory::createRejection(GuzzleHttp\Handler\EasyHandle, array)
#1 /srv/mediawiki/php-1.38.0-wmf.25/vendor/guzzlehttp/guzzle/src/Handler/CurlFactory.php(110): GuzzleHttp\Handler\CurlFactory::finishError(GuzzleHttp\Handler\CurlHandler, GuzzleHttp\Handler\EasyHandle, GuzzleHttp\Handler\CurlFactory)
#2 /srv/mediawiki/php-1.38.0-wmf.25/vendor/guzzlehttp/guzzle/src/Handler/CurlHandler.php(47): GuzzleHttp\Handler\CurlFactory::finish(GuzzleHttp\Handler\CurlHandler, GuzzleHttp\Handler\EasyHandle, GuzzleHttp\Handler\CurlFactory)
#3 /srv/mediawiki/php-1.38.0-wmf.25/vendor/guzzlehttp/guzzle/src/Handler/Proxy.php(28): GuzzleHttp\Handler\CurlHandler->__invoke(GuzzleHttp\Psr7\Request, array)
#4 /srv/mediawiki/php-1.38.0-wmf.25/vendor/guzzlehttp/guzzle/src/Handler/Proxy.php(48): GuzzleHttp\Handler\Proxy::GuzzleHttp\Handler\{closure}(GuzzleHttp\Psr7\Request, array)
#5 /srv/mediawiki/php-1.38.0-wmf.25/vendor/guzzlehttp/guzzle/src/PrepareBodyMiddleware.php(64): GuzzleHttp\Handler\Proxy::GuzzleHttp\Handler\{closure}(GuzzleHttp\Psr7\Request, array)
#6 /srv/mediawiki/php-1.38.0-wmf.25/vendor/guzzlehttp/guzzle/src/Middleware.php(31): GuzzleHttp\PrepareBodyMiddleware->__invoke(GuzzleHttp\Psr7\Request, array)
#7 /srv/mediawiki/php-1.38.0-wmf.25/vendor/guzzlehttp/guzzle/src/RedirectMiddleware.php(55): GuzzleHttp\Middleware::GuzzleHttp\{closure}(GuzzleHttp\Psr7\Request, array)
#8 /srv/mediawiki/php-1.38.0-wmf.25/vendor/guzzlehttp/guzzle/src/Middleware.php(61): GuzzleHttp\RedirectMiddleware->__invoke(GuzzleHttp\Psr7\Request, array)
#9 /srv/mediawiki/php-1.38.0-wmf.25/vendor/guzzlehttp/guzzle/src/HandlerStack.php(75): GuzzleHttp\Middleware::GuzzleHttp\{closure}(GuzzleHttp\Psr7\Request, array)
#10 /srv/mediawiki/php-1.38.0-wmf.25/vendor/guzzlehttp/guzzle/src/Client.php(331): GuzzleHttp\HandlerStack->__invoke(GuzzleHttp\Psr7\Request, array)
#11 /srv/mediawiki/php-1.38.0-wmf.25/vendor/guzzlehttp/guzzle/src/Client.php(108): GuzzleHttp\Client->transfer(GuzzleHttp\Psr7\Request, array)
#12 /srv/mediawiki/php-1.38.0-wmf.25/vendor/guzzlehttp/guzzle/src/Client.php(137): GuzzleHttp\Client->sendAsync(GuzzleHttp\Psr7\Request, array)
#13 /srv/mediawiki/php-1.38.0-wmf.25/vendor/wikimedia/shellbox/src/Client.php(162): GuzzleHttp\Client->sendRequest(GuzzleHttp\Psr7\Request)
#14 /srv/mediawiki/php-1.38.0-wmf.25/vendor/wikimedia/shellbox/src/Client.php(107): Shellbox\Client->sendRequest(string, array)
#15 /srv/mediawiki/php-1.38.0-wmf.25/extensions/WikibaseQualityConstraints/src/ConstraintCheck/Checker/FormatChecker.php(201): Shellbox\Client->call(string, string, array)
#16 /srv/mediawiki/php-1.38.0-wmf.25/extensions/WikibaseQualityConstraints/src/ConstraintCheck/Checker/FormatChecker.php(183): WikibaseQuality\ConstraintReport\ConstraintCheck\Checker\FormatChecker->runRegexCheckUsingShellbox(string, string)
#17 /srv/mediawiki/php-1.38.0-wmf.25/extensions/WikibaseQualityConstraints/src/ConstraintCheck/Checker/FormatChecker.php(141): WikibaseQuality\ConstraintReport\ConstraintCheck\Checker\FormatChecker->runRegexCheck(string, string)
#18 /srv/mediawiki/php-1.38.0-wmf.25/extensions/WikibaseQualityConstraints/src/ConstraintCheck/DelegatingConstraintChecker.php(591): WikibaseQuality\ConstraintReport\ConstraintCheck\Checker\FormatChecker->checkConstraint(WikibaseQuality\ConstraintReport\ConstraintCheck\Context\ReferenceContext, WikibaseQuality\ConstraintReport\Constraint)
#19 /srv/mediawiki/php-1.38.0-wmf.25/extensions/WikibaseQualityConstraints/src/ConstraintCheck/DelegatingConstraintChecker.php(563): WikibaseQuality\ConstraintReport\ConstraintCheck\DelegatingConstraintChecker->getCheckResultFor(WikibaseQuality\ConstraintReport\ConstraintCheck\Context\ReferenceContext, WikibaseQuality\ConstraintReport\Constraint)
#20 /srv/mediawiki/php-1.38.0-wmf.25/extensions/WikibaseQualityConstraints/src/ConstraintCheck/DelegatingConstraintChecker.php(408): WikibaseQuality\ConstraintReport\ConstraintCheck\DelegatingConstraintChecker->checkConstraintsForReferences(Wikibase\DataModel\Entity\Item, Wikibase\DataModel\Statement\Statement, NULL, array)
#21 /srv/mediawiki/php-1.38.0-wmf.25/extensions/WikibaseQualityConstraints/src/ConstraintCheck/DelegatingConstraintChecker.php(361): WikibaseQuality\ConstraintReport\ConstraintCheck\DelegatingConstraintChecker->checkStatement(Wikibase\DataModel\Entity\Item, Wikibase\DataModel\Statement\Statement, NULL, array)
#22 /srv/mediawiki/php-1.38.0-wmf.25/extensions/WikibaseQualityConstraints/src/ConstraintCheck/DelegatingConstraintChecker.php(155): WikibaseQuality\ConstraintReport\ConstraintCheck\DelegatingConstraintChecker->checkEveryStatement(Wikibase\DataModel\Entity\Item, NULL, array)
#23 /srv/mediawiki/php-1.38.0-wmf.25/extensions/WikibaseQualityConstraints/src/Api/CheckingResultsSource.php(53): WikibaseQuality\ConstraintReport\ConstraintCheck\DelegatingConstraintChecker->checkAgainstConstraintsOnEntityId(Wikibase\DataModel\Entity\ItemId, NULL, array, array)
#24 /srv/mediawiki/php-1.38.0-wmf.25/extensions/WikibaseQualityConstraints/src/Api/CachingResultsSource.php(234): WikibaseQuality\ConstraintReport\Api\CheckingResultsSource->getResults(array, array, NULL, array)
#25 /srv/mediawiki/php-1.38.0-wmf.25/extensions/WikibaseQualityConstraints/src/Api/CachingResultsSource.php(166): WikibaseQuality\ConstraintReport\Api\CachingResultsSource->getAndStoreResults(array, array, NULL, array)
#26 /srv/mediawiki/php-1.38.0-wmf.25/extensions/WikibaseQualityConstraints/src/Job/CheckConstraintsJob.php(84): WikibaseQuality\ConstraintReport\Api\CachingResultsSource->getResults(array, array, NULL, array)
#27 /srv/mediawiki/php-1.38.0-wmf.25/extensions/WikibaseQualityConstraints/src/Job/CheckConstraintsJob.php(74): WikibaseQuality\ConstraintReport\Job\CheckConstraintsJob->checkConstraints(Wikibase\DataModel\Entity\ItemId)
#28 /srv/mediawiki/php-1.38.0-wmf.25/extensions/EventBus/includes/JobExecutor.php(79): WikibaseQuality\ConstraintReport\Job\CheckConstraintsJob->run()
#29 /srv/mediawiki/rpc/RunSingleJob.php(76): MediaWiki\Extension\EventBus\JobExecutor->execute(array)
#30 {main}
Notes
  • Happened 6 times in the past seven days all clustered around the same time
  • All for 1.38.0-wmf.25
  • All from jobrunners — unclear if this is the same request

Details

MediaWiki Version
1.38.0-wmf.25
Request URL
https://jobrunner.discovery.wmnet/rpc/RunSingleJob.php

Event Timeline

Hm, does that mean Shellbox isn’t throwing the ShellboxError exception that we’re trying to catch?

FormatChecker::runRegexCheckUsingShellbox()
		try {
			$pattern = '/^(?:' . str_replace( '/', '\/', $format ) . ')$/u';
			$shellboxResponse = $this->shellboxClientFactory->getClient( [
				'timeout' => $this->config->get( 'WBQualityConstraintsSparqlMaxMillis' ) / 1000,
				'service' => 'constraint-regex-checker',
			] )->call(
				'constraint-regex-checker',
				'preg_match',
				[ $pattern, $text ]
			);
		} catch ( ShellboxError $exception ) {
			throw new ConstraintParameterException(
				( new ViolationMessage( 'wbqc-violation-message-parameter-regex' ) )
					->withInlineCode( $pattern, Role::CONSTRAINT_PARAMETER_VALUE )
			);
		}

In general, a timeout is more or less what we want to happen here, but it shouldn’t crash the whole request/job.

I guess the comment that @Pchelolo added in Use standard PSR-18 ClientInterface suggests that it’s intentional for Shellbox to not catch this exception (emphasis added):

@param ClientInterface $httpClient An object which requests an HTTP resource. It is permissible to throw an exception for propagation back to the caller. However, a successfully received response with a status code of >=400 should ideally be returned to Shellbox as a ResponseInterface, so that Shellbox can parse and rethrow its own error messages. With Guzzle this could be achieved by passing setting RequestOptions::HTTP_ERROR option to false when creating the client.

@tstarling @Reedy (reviewers of that patch): do you agree that shellbox callers should also catch ClientExceptionInterface (if they’re catching anything)? Or should Shellbox rethrow request exceptions as ShellboxError after all?

As far as I’m concerned, it’s still unclear if Shellbox callers are expected to catch PSR-18’s ClientExceptionInterface in addition to ShellboxError, or if shellbox should rethrow request exceptions as ShellboxError.

ItamarWMDE renamed this task from GuzzleHttp\Exception\ConnectException: cURL error 28: Operation timed out after 5000 milliseconds with 0 bytes received (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for http://localhost:6025/call/constraint-regex-checker to [WBQC] GuzzleHttp\Exception\ConnectException: cURL error 28: Operation timed out after 5000 milliseconds with 0 bytes received (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for http://localhost:6025/call/constraint-regex-checker.Nov 8 2023, 10:52 AM
ItamarWMDE moved this task from Incoming to [DOT] By Project on the wmde-wikidata-tech board.
ItamarWMDE added a project: Wikidata Dev Team.