DBConnectionError.php' (include_path=
Closed, ResolvedPublicPRODUCTION ERROR
Actions

Assigned To

Authored By

	hashar
	Oct 11 2023, 8:58 AM

Description

Error

service.version: 1.41.0-wmf.29
trace.id: e2073812e72ba6a057f36b73
Find trace.id in Logstash

labels.normalized_message

[{reqId}] {exception_url}   PHP Fatal Error from line 221 of /srv/mediawiki/php-1.41.0-wmf.29/includes/AutoLoader.php: require(): Failed opening required '/srv/mediawiki/php-1.41.0-wmf.29/includes/libs/rdbms/exception/DBConnectionError.php' (include_path=

error.stack_trace

from /srv/mediawiki/php-1.41.0-wmf.29/includes/AutoLoader.php(221)
#0 [internal function]: MWExceptionHandler::handleFatalError()
#1 {main}

Impact

I can't tell about the impact, looks like some job fails with a database error. The big unknown is why DBConnectionError.php could not be opened?!

There are similar issues with Failed opening required '/srv/mediawiki/php-1.41.0-wmf.29/includes/libs/HttpStatus.php

Notes

Looking at the error log, the errors appears between 7:50 UTC and 9:50 UTC possibly due to a scheduled job? Here is the view over 7 days with the filter log.origin.file.name: "/srv/mediawiki/php-1.41.0-wmf.29/includes/AutoLoader.php"

autoload_fatal_7days.png (273×628 px, 16 KB)

Details

Request URL: https://jobrunner.discovery.wmnet/rpc/RunSingleJob.php

Related Objects
Search...

Status	Subtype	Assigned	Task
Open		KStoller-WMF	T248330 [EPIC] Growth: Positive reinforcement
Resolved		KStoller-WMF	T222310 [EPIC] Positive reinforcement: New Impact Module
Open		KStoller-WMF	T342303 [EPIC] Growth: Deploy Positive reinforcement features to all Wikipedias
Resolved	Nov 1 2023	Urbanecm_WMF	T336203 Positive reinforcement: Deploy the new Impact module to all Wikipedias
Resolved		Urbanecm_WMF	T344143 New Impact module: Run backend updating logic on all Wikipedias
Resolved		Urbanecm_WMF	T344428 refreshUserImpactJob logs mysterious fatal errors
Resolved	PRODUCTION ERROR	Urbanecm_WMF	T348614 PHP Fatal Error from line 221 of /srv/mediawiki/php-1.41.0-wmf.29/includes/AutoLoader.php: require(): Failed opening required '/srv/mediawiki/php-1.41.0-wmf.29/includes/libs/rdbms/exception/DBConnectionError.php' (include_path=

Event Timeline

hashar created this task.Oct 11 2023, 8:58 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 11 2023, 8:59 AM

When looking at the log entries for the trace id used to generate this task ( Find trace.id in Logstash ), there are close to 20k messages from PageViewInfo Failed fetching {requesturl}: {error}.

We found an issue with PageViewInfo yesterday and filed it as T348517

But the devil is why the Autoloader.php can not find the requested files (either DbConnectionError.php or HttpStatus.php. Maybe that is because PHP is shutting down and the autoloader entries have already been deleted.

hashar mentioned this in T348517: Daily pageview/PageViewInfo errors on jobrunners.Oct 11 2023, 9:05 AM

The similar report for HttpStatus.php not being found:

Error

mwversion: 1.41.0-wmf.29
reqId: 4673633442b751f3f992c01a
Find reqId in Logstash

normalized_message

[{reqId}] {exception_url}   PHP Fatal Error from line 221 of /srv/mediawiki/php-1.41.0-wmf.29/includes/AutoLoader.php: require(): Failed opening required '/srv/mediawiki/php-1.41.0-wmf.29/includes/libs/HttpStatus.php' (include_path='/srv/mediawiki/php-1.4

exception.trace

from /srv/mediawiki/php-1.41.0-wmf.29/includes/AutoLoader.php(221)
#0 [internal function]: MWExceptionHandler::handleFatalError()
#1 {main}

That trace id also have a bunch of Class 'Cdb\Exception' not found which got filed as T341658. So that might be a duplicate.

hashar mentioned this in T341658: Error: Class 'Cdb\Exception' not found.Oct 11 2023, 9:20 AM

hashar mentioned this in T344929: Maintenance script appservers running code from wmf.20 when version not live.Oct 11 2023, 9:25 AM

Based on the timing and the fact it's happening on the jobrunners, I'm fairly certain this is related to T344428#9279661 in some way. It also seems to no longer happen since the T344428#9279661 change was deployed, which also signalizes the root cause is the same.

In any case, since logstash says this doesn't happen since yesterday, I'm closing this one (and linking to T344428 for future reference).

PHP Fatal Error from line 221 of /srv/mediawiki/php-1.41.0-wmf.29/includes/AutoLoader.php: require(): Failed opening required '/srv/mediawiki/php-1.41.0-wmf.29/includes/libs/rdbms/exception/DBConnectionError.php' (include_path=Closed, ResolvedPublicPRODUCTION ERRORActions

Description

Error

Impact

Notes

Details

Related ObjectsSearch...

Event Timeline

Error

PHP Fatal Error from line 221 of /srv/mediawiki/php-1.41.0-wmf.29/includes/AutoLoader.php: require(): Failed opening required '/srv/mediawiki/php-1.41.0-wmf.29/includes/libs/rdbms/exception/DBConnectionError.php' (include_path=
Closed, ResolvedPublicPRODUCTION ERROR
Actions

Related Objects
Search...