Page MenuHomePhabricator
Create Task
Maniphest T330936

Missing Cirrussearch dump (enwiki and wikidata)
Closed, ResolvedPublic3 Estimated Story Points
Actions

Description

Hello @EBernhardson, the script failed yesterday; here is the email we got and the stack trace;

email
Systemd timer ran the following command:
    `/usr/local/bin/dumpcirrussearch.sh --config /etc/dumps/confs/wikidump.conf.other --dblist /srv/mediawiki/dblists/s1.dblist`
its return value was 1 and emitted the following output:
<13>Feb 27 19:03:59 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20230227/enwiki-20230227-cirrussearch-content.json.gz
stack.trace
Dumping 6624383 documents (6624383 in the index)
        6% done...
        8% done...
        ......
        58% done...
Elastica\Exception\ClientException from line 26 of /srv/mediawiki/php-1.40.0-wmf.24/vendor/ruflin/elastica/src/Connection/Strategy/Simple.php: No enabled connection
#0 /srv/mediawiki/php-1.40.0-wmf.24/vendor/ruflin/elastica/src/Connection/ConnectionPool.php(86): Elastica\Connection\Strategy\Simple->getConnection(Array)
#1 /srv/mediawiki/php-1.40.0-wmf.24/vendor/ruflin/elastica/src/Client.php(396): Elastica\Connection\ConnectionPool->getConnection()
#2 /srv/mediawiki/php-1.40.0-wmf.24/vendor/ruflin/elastica/src/Client.php(512): Elastica\Client->getConnection()
#3 /srv/mediawiki/php-1.40.0-wmf.24/vendor/ruflin/elastica/src/Search.php(348): Elastica\Client->request('enwiki_content/...', 'POST', Array, Array)
#4 /srv/mediawiki/php-1.40.0-wmf.24/extensions/CirrusSearch/includes/Elastica/SearchAfter.php(90): Elastica\Search->search()
#5 /srv/mediawiki/php-1.40.0-wmf.24/extensions/CirrusSearch/includes/Elastica/SearchAfter.php(70): CirrusSearch\Elastica\SearchAfter->runSearch()
#6 /srv/mediawiki/php-1.40.0-wmf.24/extensions/CirrusSearch/maintenance/DumpIndex.php(163): CirrusSearch\Elastica\SearchAfter->next()
#7 /srv/mediawiki/php-1.40.0-wmf.24/maintenance/includes/MaintenanceRunner.php(609): CirrusSearch\Maintenance\DumpIndex->execute()
#8 /srv/mediawiki/php-1.40.0-wmf.24/maintenance/doMaintenance.php(99): MediaWiki\Maintenance\MaintenanceRunner->run()
#9 /srv/mediawiki/php-1.40.0-wmf.24/extensions/CirrusSearch/maintenance/DumpIndex.php(288): require_once('/srv/mediawiki/...')
#10 /srv/mediawiki/multiversion/MWScript.php(118): require_once('/srv/mediawiki/...')
#11 {main}

We checked to see if enwiki-20230227-cirrussearch-content.json.gz was later completed; sadly, the file is still missing. Additionally, the wikidata files for the last run are missing; we didn’t get any errors or anything related to why the wikidata files weren’t completed.

The cirrussearch wikidata log on that day(20230220) looks like this; It shows the dump didn’t complete

Dumping 101983781 documents (101983781 in the index)
        2% done...
        ......
        30% done...
        32% done...
        34% done...
        36% done...

Details

SubjectRepoBranchLines +/-
Don't disable the last connection after http errormediawiki/extensions/Elasticamaster+12 -4
Customize query in gerrit

Event Timeline

Hokwelum created this task.Mar 1 2023, 8:26 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 1 2023, 8:26 PM
Hokwelum added projects: Dumps-Generation, CirrusSearch.Mar 1 2023, 8:28 PM
Restricted Application added a project: Discovery-Search. · View Herald TranscriptMar 1 2023, 8:28 PM
Hokwelum updated the task description. (Show Details)Mar 1 2023, 8:29 PM
Hokwelum updated the task description. (Show Details)Mar 1 2023, 8:43 PM
ArielGlenn moved this task from Backlog to Other teams on the Dumps-Generation board.Mar 2 2023, 11:49 AM
MPhamWMF moved this task from needs triage to Current work on the Discovery-Search board.Mar 6 2023, 4:35 PM
MPhamWMF edited projects, added Discovery-Search (Current work); removed Discovery-Search.
EBernhardson added a comment.Mar 6 2023, 4:55 PM

Not sure yet, enwiki failed with curl error 23 (Failed writing body). The client library disabled the connection after this failure, meaning no retries were attempted and we went to the failure condition.

wikidata is more curious, i don't see any logs generated by mediawiki around the time that it stopped running. Will have to look closer to see whats going on there.

EBernhardson added a comment.Mar 6 2023, 6:13 PM

Took a closer look for the wikidata failure, but i've turned up nothing. With the output ending without any failure messages it suggests to me that the process died, perhaps a force kill or a segfault. I couldn't find anything in the system logs that correlate. Sadly syslog doesn't go back that far (oldest syslog entry is feb 27, this died on feb 22) but theres no certainty it would have had useful information.

Getting back to the enwiki failure, with error 23 (CURLE_WRITE_ERROR), this is also a bit curious. Typically, per stack overflow, this error occurs when attempting to write to disk and it's rejected for permissions or available space. Thats not the case here, the output goes to an in-memory buffer which shouldn't be able to have such issues. I poked the libcurl source a bit to see what other conditions can generate this error, there are actually quite a few. Too many to review for relevance here without spending significant time.

I'm indecisive on the correct fix for the CURLE_WRITE_ERROR, will talk with the team in our wednesday meeting and see what we think.

EBernhardson added a comment.Mar 20 2023, 4:35 PM

We talked a bit about this, the plan right now is to prevent the library we use from disabling the last available connection. That should allow the retries to work as we expect regardless of the error type. It seems this connection disabling is more suited for cases that have many instances in their pool, rather than a single DNS backed by LVS.

MPhamWMF set the point value for this task to 3.Mar 20 2023, 4:39 PM
MPhamWMF moved this task from Incoming to Ready for Dev -- SWE on the Discovery-Search (Current work) board.
EBernhardson claimed this task.Mar 30 2023, 5:12 PM
EBernhardson moved this task from Ready for Dev -- SWE to In Progress on the Discovery-Search (Current work) board.
gerritbot added a comment.Mar 30 2023, 6:52 PM

Change 904615 had a related patch set uploaded (by Ebernhardson; author: Ebernhardson):

[mediawiki/extensions/Elastica@master] Don't disable the last connection after http error

https://gerrit.wikimedia.org/r/904615

gerritbot added a project: Patch-For-Review.Mar 30 2023, 6:52 PM
EBernhardson moved this task from In Progress to Needs review on the Discovery-Search (Current work) board.Mar 30 2023, 6:52 PM
gerritbot added a comment.Apr 11 2023, 9:31 AM

Change 904615 merged by jenkins-bot:

[mediawiki/extensions/Elastica@master] Don't disable the last connection after http error

https://gerrit.wikimedia.org/r/904615

ReleaseTaggerBot added a project: MW-1.41-notes (1.41.0-wmf.5; 2023-04-17).Apr 11 2023, 10:00 AM
Maintenance_bot removed a project: Patch-For-Review.Apr 11 2023, 10:10 AM
EBernhardson moved this task from Needs review to Blocked/Waiting on the Discovery-Search (Current work) board.Apr 24 2023, 5:58 PM
EBernhardson moved this task from Blocked/Waiting to Needs Reporting on the Discovery-Search (Current work) board.May 15 2023, 3:11 PM

has been deployed for a month without issues, lets hope this is resolved.

Gehel closed this task as Resolved.Jun 2 2023, 9:43 AM
ArielGlenn moved this task from Other teams to Done on the Dumps-Generation board.Jun 22 2023, 4:57 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL