Page MenuHomePhabricator

CirrusSearch dumps are broken since Mar 18 2019
Closed, ResolvedPublic


Last valid dumps are from 2019/03/18.
Recent ones are all empty files.

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript
dcausse claimed this task.
dcausse triaged this task as High priority.

Fixed by (all cirrus scripts are broken because of this problem)

I expect the dumps generated today to be correct, I'll keep an eye on it and re-open if needed.

dumpsgen@snapshot1008:/mnt/dumpsdata/temp/dumpsgen$ /usr/bin/php7.2 /srv/mediawiki/multiversion/MWScript.php extensions/CirrusSearch/maintenance/dumpIndex.php --wiki=elwiki --indexType=content | gzip > elwiki-cirrus-content.gz
[1a17456c9633fdaeb63a3c83] [no req]   Elastica\Exception\Connection\HttpException from line 189 of /srv/mediawiki/php-1.33.0-wmf.25/vendor/ruflin/elastica/lib/Elastica/Transport/Http.php: Couldn't connect to host, Elasticsearch down?
#0 /srv/mediawiki/php-1.33.0-wmf.25/vendor/ruflin/elastica/lib/Elastica/Request.php(193): Elastica\Transport\Http->exec(Elastica\Request, array)
#1 /srv/mediawiki/php-1.33.0-wmf.25/vendor/ruflin/elastica/lib/Elastica/Client.php(688): Elastica\Request->send()
#2 /srv/mediawiki/php-1.33.0-wmf.25/vendor/ruflin/elastica/lib/Elastica/Client.php(699): Elastica\Client->request(string, string, array, array)
#3 /srv/mediawiki/php-1.33.0-wmf.25/vendor/ruflin/elastica/lib/Elastica/Client.php(699): Elastica\Client->request(string, string, array, array)
#4 /srv/mediawiki/php-1.33.0-wmf.25/vendor/ruflin/elastica/lib/Elastica/Search.php(462): Elastica\Client->request(string, string, array, array)
#5 /srv/mediawiki/php-1.33.0-wmf.25/vendor/ruflin/elastica/lib/Elastica/Scroll.php(130): Elastica\Search->search()
#6 /srv/mediawiki/php-1.33.0-wmf.25/extensions/CirrusSearch/maintenance/dumpIndex.php(147): Elastica\Scroll->rewind()
#7 /srv/mediawiki/php-1.33.0-wmf.25/maintenance/doMaintenance.php(96): CirrusSearch\Maintenance\DumpIndex->execute()
#8 /srv/mediawiki/php-1.33.0-wmf.25/extensions/CirrusSearch/maintenance/dumpIndex.php(268): require_once(string)
#9 /srv/mediawiki/multiversion/MWScript.php(100): require_once(string)
#10 {main}

Is this expected (fix not yet deployed) or is my attempt to test incorrect?

So it's likely that snapshot machines are lacking the reverse proxy used for connection pooling needed to connect to elastic when using PHP, reopening.

I think that ::profile::services_proxy is lacking on these machines, @ArielGlenn would it be possible to add it?

dcausse added a project: Dumps-Generation.
dcausse moved this task from needs triage to elastic / cirrus on the Discovery-Search board.
dcausse moved this task from elastic / cirrus to Ops / SRE on the Discovery-Search board.

Change 503934 had a related patch set uploaded (by ArielGlenn; owner: ArielGlenn):
[operations/puppet@production] add services proxy setup for snapshot hosts

Change 503934 merged by ArielGlenn:
[operations/puppet@production] add services proxy setup for snapshot hosts

I've done a test for elwiki and the content looks reasonable.

This can remain open until today's run completes and it's verified that the dumps have proper content.

I think the data for 20190415 is vaild. <s>But what about the previous data?</s>

Wait... Only a few sites have the data.

The previous data is no longer available; we get a dump of what the indexes hold at the time of the dump. It's not like article history where there are separate revisions showing the state at any given time.

ArielGlenn claimed this task.

I'm going to close this since the reported problem was resolved.