Page MenuHomePhabricator
Create Task
Maniphest T341058

Failures for Cirrus Search dumps for wikidatawiki, zhwiki, enwiki for the 20230703 run
Closed, ResolvedPublic3 Estimated Story Points
Actions

Description

I got the following messages in email:

Jul  4 09:16:50 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20230703/wikidatawiki-20230703-cirrussearch-content.json.gz
Jul  4 09:16:50 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20230703/zhwiki-20230703-cirrussearch-content.json.gz
Jul  4 09:16:50 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20230703/enwiki-20230703-cirrussearch-general.json.gz

These three files are indeed missing from the output files. Didn't see anything in logstash right off, in the logs on snapshot1008 I see

Elastica\Exception\ClientException from line 26 of /srv/mediawiki/php-1.41.0-wmf.15/vendor/ruflin/elastica/src/Connection/Strategy/Simple.php: No enabled connection
#0 /srv/mediawiki/php-1.41.0-wmf.15/vendor/ruflin/elastica/src/Connection/ConnectionPool.php(86): Elastica\Connection\Strategy\Simple->getConnection(Array)
#1 /srv/mediawiki/php-1.41.0-wmf.15/vendor/ruflin/elastica/src/Client.php(396): Elastica\Connection\ConnectionPool->getConnection()
#2 /srv/mediawiki/php-1.41.0-wmf.15/vendor/ruflin/elastica/src/Client.php(512): Elastica\Client->getConnection()
#3 /srv/mediawiki/php-1.41.0-wmf.15/vendor/ruflin/elastica/src/Search.php(348): Elastica\Client->request('zhwiki_content/...', 'POST', Array, Array)
#4 /srv/mediawiki/php-1.41.0-wmf.15/extensions/CirrusSearch/includes/Elastica/SearchAfter.php(96): Elastica\Search->search()
#5 /srv/mediawiki/php-1.41.0-wmf.15/extensions/CirrusSearch/includes/Elastica/SearchAfter.php(72): CirrusSearch\Elastica\SearchAfter->runSearch()
#6 /srv/mediawiki/php-1.41.0-wmf.15/extensions/CirrusSearch/maintenance/DumpIndex.php(163): CirrusSearch\Elastica\SearchAfter->next()
#7 /srv/mediawiki/php-1.41.0-wmf.15/maintenance/includes/MaintenanceRunner.php(681): CirrusSearch\Maintenance\DumpIndex->execute()
#8 /srv/mediawiki/php-1.41.0-wmf.15/maintenance/run.php(51): MediaWiki\Maintenance\MaintenanceRunner->run()
#9 /srv/mediawiki/multiversion/MWScript.php(160): require_once('/srv/mediawiki/...')
#10 {main}

and similar for the other two files.

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
Don't disable the last connection after http error (part deux)Ebernhardsonmediawiki/extensions/Elasticamaster+4 -0
Customize query in gerrit

Event Timeline

ArielGlenn created this task.Jul 4 2023, 12:29 PM
ArielGlenn triaged this task as High priority.
Restricted Application added a subscriber: Stang. · View Herald TranscriptJul 4 2023, 12:29 PM
ArielGlenn moved this task from Backlog to Other teams on the Dumps-Generation board.Jul 4 2023, 12:37 PM
Shizhao added a project: Chinese-Sites.Jul 5 2023, 1:42 AM
ArielGlenn added a comment.Jul 5 2023, 12:19 PM

Adding two more failures:

Jul  4 09:16:50 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20230703/mlwiki-20230703-cirrussearch-content.json.gz
Jul  4 09:16:50 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20230703/metawiki-20230703-cirrussearch-general.json.gz

These came from different db sections so the script running those finished later, and hence the error report was sent later.

ArielGlenn added a comment.Jul 9 2023, 3:38 PM

One more to add:

Jul  4 09:16:50 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20230703/commonswiki-20230703-cirrussearch-content.json.gz
bking set the point value for this task to 3.Jul 10 2023, 3:34 PM
bking moved this task from Incoming to Ready for Dev -- SWE on the Discovery-Search (Current work) board.
EBernhardson claimed this task.Jul 10 2023, 5:46 PM
EBernhardson moved this task from Ready for Dev -- SWE to In Progress on the Discovery-Search (Current work) board.

Sigh, I thought we resolved the no enabled connection problem but clearly not. Looking into it.

gerritbot added a comment.Jul 10 2023, 8:42 PM

Change 936806 had a related patch set uploaded (by Ebernhardson; author: Ebernhardson):

[mediawiki/extensions/Elastica@master] Don't disable the last connection after http error (part deux)

https://gerrit.wikimedia.org/r/936806

gerritbot added a project: Patch-For-Review.Jul 10 2023, 8:42 PM
EBernhardson added a comment.Jul 10 2023, 8:44 PM

It looks like this time we got both the CURLE_COULDNT_CONNECT error, which we handle, along with CURLE_GOT_NOTHING which we do not handle. Rather than playing wack-a-mole with curl error codes I think our best bet is going to be ensuring the connection remains available regardless of which error it was.

EBernhardson moved this task from In Progress to Needs review on the Discovery-Search (Current work) board.Jul 10 2023, 8:44 PM
ArielGlenn added a comment.Jul 13 2023, 11:14 AM

Just for my understanding, it looks like the new patch would exception out in the case where there is a failure with the last connection of whatever sort. Am I reading that right? And if so, how does that help us in the current situation? Sorry for whatever I am missing here. Thanks!

EBernhardson added a comment.Jul 13 2023, 2:53 PM

The error is No enabled connection. Connections are disabled by the client library any time there is an http error, unless we explicitly re-enable it. In prod we only have one connection, the LVS one, so a single unhandled http failure kills any attempt to talk to elasticsearch for the rest of the script execution.

pfischer claimed this task.Jul 17 2023, 7:04 AM
pfischer added a subscriber: EBernhardson.
Gehel reassigned this task from pfischer to EBernhardson.Jul 17 2023, 3:09 PM
Gehel added a subscriber: pfischer.
gerritbot added a comment.Jul 18 2023, 8:55 AM

Change 936806 merged by jenkins-bot:

[mediawiki/extensions/Elastica@master] Don't disable the last connection after http error (part deux)

https://gerrit.wikimedia.org/r/936806

ReleaseTaggerBot added a project: MW-1.41-notes (1.41.0-wmf.19; 2023-07-25).Jul 18 2023, 9:00 AM
Maintenance_bot removed a project: Patch-For-Review.Jul 18 2023, 9:10 AM
dcausse moved this task from Needs review to Needs Reporting on the Discovery-Search (Current work) board.Jul 24 2023, 3:16 PM
Gehel closed this task as Resolved.Jul 28 2023, 9:51 AM
Stang moved this task from Backlog to Closed on the Chinese-Sites board.Jul 28 2023, 10:01 AM
Stang unsubscribed.Jul 31 2023, 10:33 PM
FriedrickMILBarbarossa added a project: Wikidata.Aug 8 2023, 3:03 AM
ArielGlenn moved this task from Other teams to Done on the Dumps-Generation board.Aug 28 2023, 9:56 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits