The exception that was supposed to be thrown is Unable to open CDB file. There is no obvious reason why the exception class couldn't be loaded - CDB code hasn't changed in a long time, the class is at the right place, in my local setup the Composer class does include it. Also no obvious reason while the file couldn't load. So I'd assume some sort of deployment hiccup where some of the MediaWiki files were missing from some job runner.

It happens on more servers, and more wikis, and over a larger time period than seems explained by a mere deployment race condition.

Initial investigation shows that this happens relatively rarely, but exclusively as part of RefreshUserImpactJob in GrowthExperiments. This might mean the issue is in the way how GrowthExperiments calls PageViewInfo, but judging by CodeSearch, the more likely explanation is that no other extension calls PageViewInfo in a job context.

Checking the logs at mwlog1002, I can see that this exception is always followed by a PHP Fatal Error. Failed opening required '/srv/mediawiki/php-1.41.0-wmf.18/includes/libs/HttpStatus.php'. By looking at the file, I can see no obvious reason why it should fail this way.

Around the exception, I can also see a bunch of errors in PageViewInfo channel, around the lines of Error fetching URL: Could not resolve host: localhost and There was a problem during the HTTP request: 503 Service Unavailable (LogStash view). I do not understand why localhost would be unresolvable. Those errors are much more frequent and colerate with the Cdb exception reported in this task.

I see no obvious patterns in terms of which servers it impacts; I think the exception is related to cases when the service proxy is not working properly, which appears to happen fairly regularly for some reason.

Most of these errors (failing to load the Exception and HttpStatus classes, failing to open the CDB file) are about file access. The remaining ones are about internet connections, which are sometimes handled in a way similar to files (e.g. sockets as virtual filesystem entries). Given that the issue seems to be limited to a job which does a ton of HTTP requests to AQS, it's probably overrunning some sort of quota that applies to open files + connections?

Could not resolve host: localhost is only happening on mw1437 (the jobrunner canary) so maybe that's a different issue.

In T341658#9041935, @Tgr wrote:

Most of these errors […] are about file access. […] it's probably overrunning some sort of quota that applies to open files + connections?

Nice catch, and yes, I believe this is exactly what's happening. See also T230245, where a maintenance scripts that generates captcha images, gets a fatal error:

Fatal error: require(/srv/mediawiki/php-1.34.0-wmf.8/includes/json/FormatJson.php): File not found in /srv/mediawiki/php-1.34.0-wmf.8/includes/AutoLoader.php on line 109

Notably, this fatal error happens as part of error reporting, after it fails to upload a file to Swift over HTTP. Both are happening for the same reason — EMFILE (Too many open files), which is a system-level restriction typically set by the operating system.

In case of T230245 the problem was that there was no concurrency limit set. It was correctly uploading things in batches, but it was preparing the temp files all in parallel and keeping the file handles open.

I think the requests are not parallel (but Guzzle certainly has the ability to do them in parallel and it comes down to request configuration, which can get injected in a number of different ways, so I might have misread the code). It's also possible some HTTP-related code is missing a close() call.

In T341658#9042593, @Tgr wrote:

It's also possible some HTTP-related code is missing a close() call.

Guzzle seems to handle that internally (I think these requests end up with CurlHandler and then CurlFactory::release()). It reuses curl handles and only closes open handles once it's above its max handle limit but that's very low (3 if I read the code correctly), so that's probably not it.

As part of running the train I found some errors coming from the PageViewInfo job which somehow have the autoloader unable to open includes/libs/rdbms/exception/DBConnectionError.php or includes/libs/HttpStatus.php. I have filed it as T348614

The trace for the missing includes/libs/HttpStatus.php also have errors Class 'Cdb\Exception' not found which points me back to this task.