Page MenuHomePhabricator

Make Cirrus Search dump script more resilient to failures (elasticsearch restarts)
Open, MediumPublic

Description

Problem: The Cirrus Search dump script is not resilient enough to failures (elasticsearch restarts), causing cirrus dumps for some wikis to be missing for some weeks.

Solution probably involves stopping use of elastic "srcoll" API

AC:

  • Make the dump script more resilient, so that cirrus dumps no longer fail when restarting elasticsearch

original log below, and additional logs in comments.

For this week's run:

<13>Oct  7 08:09:36 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20201005/eswiki-20201005-cirrussearch-general.json.gz
<13>Oct  7 08:09:37 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20201005/eswikibooks-20201005-cirrussearch-content.json.gz
<13>Oct  7 08:09:37 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20201005/eswikibooks-20201005-cirrussearch-general.json.gz

Event Timeline

ArielGlenn triaged this task as Medium priority.Oct 8 2020, 4:41 PM
ArielGlenn created this task.

capturing some logs before they vanish:

eswiki-20201005-cirrussearch-general is:

Dumping 3630181 documents (3630181 in the index)
        2% done...
        4% done...
        6% done...
Elastica\Exception\Connection\HttpException from line 190 of /srv/mediawiki/php-1.36.0-wmf.10/vendor/ruflin/elastica/lib/Elastica/Transport/Http.php: Unknown error:56
#0 /srv/mediawiki/php-1.36.0-wmf.10/vendor/ruflin/elastica/lib/Elastica/Request.php(194): Elastica\Transport\Http->exec(Object(Elastica\Request), Array)
#1 /srv/mediawiki/php-1.36.0-wmf.10/vendor/ruflin/elastica/lib/Elastica/Client.php(689): Elastica\Request->send()
#2 /srv/mediawiki/php-1.36.0-wmf.10/vendor/ruflin/elastica/lib/Elastica/Search.php(463): Elastica\Client->request('_search/scroll', 'GET', Array, Array)
#3 /srv/mediawiki/php-1.36.0-wmf.10/vendor/ruflin/elastica/lib/Elastica/Scroll.php(82): Elastica\Search->search()
#4 /srv/mediawiki/php-1.36.0-wmf.10/extensions/CirrusSearch/maintenance/DumpIndex.php(154): Elastica\Scroll->next()
#5 /srv/mediawiki/php-1.36.0-wmf.10/maintenance/doMaintenance.php(107): CirrusSearch\Maintenance\DumpIndex->execute()
#6 /srv/mediawiki/php-1.36.0-wmf.10/extensions/CirrusSearch/maintenance/DumpIndex.php(269): require_once('/srv/mediawiki/...')
#7 /srv/mediawiki/multiversion/MWScript.php(101): require_once('/srv/mediawiki/...')
#8 {main}

eswikibooks-20201005-cirrussearch-content:

Elastica\Exception\Connection\HttpException from line 190 of /srv/mediawiki/php-1.36.0-wmf.10/vendor/ruflin/elastica/lib/Elastica/Transport/Http.php: Couldn't connect to host, Elasticsearch down?
#0 /srv/mediawiki/php-1.36.0-wmf.10/vendor/ruflin/elastica/lib/Elastica/Request.php(194): Elastica\Transport\Http->exec(Object(Elastica\Request), Array)
#1 /srv/mediawiki/php-1.36.0-wmf.10/vendor/ruflin/elastica/lib/Elastica/Client.php(689): Elastica\Request->send()
#2 /srv/mediawiki/php-1.36.0-wmf.10/vendor/ruflin/elastica/lib/Elastica/Client.php(700): Elastica\Client->request('eswikibooks_con...', 'GET', Array, Array)
#3 /srv/mediawiki/php-1.36.0-wmf.10/vendor/ruflin/elastica/lib/Elastica/Client.php(700): Elastica\Client->request('eswikibooks_con...', 'GET', Array, Array)
#4 /srv/mediawiki/php-1.36.0-wmf.10/vendor/ruflin/elastica/lib/Elastica/Search.php(463): Elastica\Client->request('eswikibooks_con...', 'GET', Array, Array)
#5 /srv/mediawiki/php-1.36.0-wmf.10/vendor/ruflin/elastica/lib/Elastica/Scroll.php(131): Elastica\Search->search()
#6 /srv/mediawiki/php-1.36.0-wmf.10/extensions/CirrusSearch/maintenance/DumpIndex.php(154): Elastica\Scroll->rewind()
#7 /srv/mediawiki/php-1.36.0-wmf.10/maintenance/doMaintenance.php(107): CirrusSearch\Maintenance\DumpIndex->execute()
#8 /srv/mediawiki/php-1.36.0-wmf.10/extensions/CirrusSearch/maintenance/DumpIndex.php(269): require_once('/srv/mediawiki/...')
#9 /srv/mediawiki/multiversion/MWScript.php(101): require_once('/srv/mediawiki/...')
#10 {main}

eswikibooks-20201005-cirrussearch-general:

Elastica\Exception\Connection\HttpException from line 190 of /srv/mediawiki/php-1.36.0-wmf.10/vendor/ruflin/elastica/lib/Elastica/Transport/Http.php: Couldn't connect to host, Elasticsearch down?
#0 /srv/mediawiki/php-1.36.0-wmf.10/vendor/ruflin/elastica/lib/Elastica/Request.php(194): Elastica\Transport\Http->exec(Object(Elastica\Request), Array)
#1 /srv/mediawiki/php-1.36.0-wmf.10/vendor/ruflin/elastica/lib/Elastica/Client.php(689): Elastica\Request->send()
#2 /srv/mediawiki/php-1.36.0-wmf.10/vendor/ruflin/elastica/lib/Elastica/Client.php(700): Elastica\Client->request('eswikibooks_gen...', 'GET', Array, Array)
#3 /srv/mediawiki/php-1.36.0-wmf.10/vendor/ruflin/elastica/lib/Elastica/Client.php(700): Elastica\Client->request('eswikibooks_gen...', 'GET', Array, Array)
#4 /srv/mediawiki/php-1.36.0-wmf.10/vendor/ruflin/elastica/lib/Elastica/Search.php(463): Elastica\Client->request('eswikibooks_gen...', 'GET', Array, Array)
#5 /srv/mediawiki/php-1.36.0-wmf.10/vendor/ruflin/elastica/lib/Elastica/Scroll.php(131): Elastica\Search->search()
#6 /srv/mediawiki/php-1.36.0-wmf.10/extensions/CirrusSearch/maintenance/DumpIndex.php(154): Elastica\Scroll->rewind()
#7 /srv/mediawiki/php-1.36.0-wmf.10/maintenance/doMaintenance.php(107): CirrusSearch\Maintenance\DumpIndex->execute()
#8 /srv/mediawiki/php-1.36.0-wmf.10/extensions/CirrusSearch/maintenance/DumpIndex.php(269): require_once('/srv/mediawiki/...')
#9 /srv/mediawiki/multiversion/MWScript.php(101): require_once('/srv/mediawiki/...')
#10 {main}

(all three log files have an mtime at 2020-10-07T08:09:00:35)
The dump process should probably be a bit more resilient but at least it did not leave corrupted files behind and visible from https://dumps.wikimedia.org/other/cirrussearch/20201005/.

Today's report:

<13>Oct 14 01:47:12 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20201012/enwiki-20201012-cirrussearch-general.json.gz
<13>Oct 15 00:41:35 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20201012/srwiki-20201012-cirrussearch-content.json.gz
<13>Oct 15 00:49:59 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20201012/srwikisource-20201012-cirrussearch-content.json.gz
<13>Oct 15 00:55:24 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20201012/svwiki-20201012-cirrussearch-content.json.gz
TJones renamed this task from CIrrus Search dumps failed for some wikis to Cirrus Search dumps failed for some wikis.Oct 19 2020, 5:20 PM

It would be great to make this process more robust to connection issues but for this I think we should move away from the scroll API to fetch documents.

Last week's report that i added in a comment and forgot to save:

<13>Oct 14 01:47:12 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20201012/enwiki-20201012-cirrussearch-general.json.gz
<13>Oct 15 00:41:35 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20201012/srwiki-20201012-cirrussearch-content.json.gz
<13>Oct 15 00:49:59 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20201012/srwikisource-20201012-cirrussearch-content.json.gz
<13>Oct 15 00:55:24 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20201012/svwiki-20201012-cirrussearch-content.json.gz

Today's report:

<13>Oct 29 01:40:27 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20201026/ptwiki-20201026-cirrussearch-content.json.gz
<13>Oct 29 01:49:28 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20201026/ptwiki-20201026-cirrussearch-general.json.gz

Today's report:

<13>Nov  3 17:45:52 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20201102/commonswiki-20201102-cirrussearch-file.json.gz

Today's report:

<13>Nov 18 16:27:19 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20201116/frwiki-20201116-cirrussearch-content.json.gz

Today's report:

<13>Nov 25 17:06:33 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20201123/ruwiki-20201123-cirrussearch-content.json.gz
<13>Nov 25 17:28:27 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20201123/ruwiki-20201123-cirrussearch-general.json.gz
<13>Nov 26 04:19:40 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20201123/wikidatawiki-20201123-cirrussearch-content.json.gz
<13>Nov 26 04:45:34 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20201123/zhwiki-20201123-cirrussearch-content.json.gz
<13>Nov 26 04:50:00 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20201123/zhwiki-20201123-cirrussearch-general.json.gz
<13>Nov 26 04:54:27 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20201123/zhwikisource-20201123-cirrussearch-content.json.gz

Today's report:

<13>Dec  1 02:03:36 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20201130/commonswiki-20201130-cirrussearch-file.json.gz

Today's report:

<13>Jan 12 17:23:17 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20210111/cswiki-20210111-cirrussearch-content.json.gz
<13>Jan 12 19:41:58 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20210111/diqwiktionary-20210111-cirrussearch-content.json.gz
<13>Jan 12 19:41:59 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20210111/diqwiktionary-20210111-cirrussearch-general.json.gz

Today's report:

<13>Jan 20 18:30:10 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20210118/enwikisource-20210118-cirrussearch-content.json.gz
<13>Jan 20 19:47:58 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20210118/eswiki-20210118-cirrussearch-content.json.gz

Today's report:

<13>Jan 26 14:21:35 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20210125/enwiki-20210125-cirrussearch-general.json.gz

Today's report:

<13>Feb  1 20:36:49 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20210201/cebwiki-20210201-cirrussearch-content.json.gz
<13>Feb  1 22:23:51 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20210201/commonswiki-20210201-cirrussearch-content.json.gz
<13>Feb  3 18:40:13 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20210201/enwiki-20210201-cirrussearch-general.json.gz
<13>Feb  4 10:15:19 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20210201/nlwiki-20210201-cirrussearch-content.json.gz

Today's report:

<13>Feb  9 11:24:47 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20210208/commonswiki-20210208-cirrussearch-file.json.gz
<13>Feb 11 12:20:06 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20210208/wikidatawiki-20210208-cirrussearch-content.json.gz

Today's report:

<13>Feb 24 21:48:03 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20210222/frwikibooks-20210222-cirrussearch-general.json.gz

Today's report:

<13>Mar  4 02:51:29 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20210301/mniwiktionary-20210301-cirrussearch-content.json.gz
<13>Mar  4 02:51:29 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20210301/mniwiktionary-20210301-cirrussearch-general.json.gz

Today's report:

<13>Mar 11 08:02:55 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20210308/mniwiktionary-20210308-cirrussearch-content.json.gz
<13>Mar 11 08:02:56 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20210308/mniwiktionary-20210308-cirrussearch-general.json.gz

Today's report:

<13>Mar 18 08:28:33 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20210315/mniwiktionary-20210315-cirrussearch-content.json.gz
<13>Mar 18 08:28:34 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20210315/mniwiktionary-20210315-cirrussearch-general.json.gz

Today's report:

<13>Mar 24 23:58:07 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20210322/jawiki-20210322-cirrussearch-content.json.gz
<13>Mar 25 00:05:59 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20210322/jawiki-20210322-cirrussearch-general.json.gz
<13>Mar 25 03:15:08 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20210322/mniwiktionary-20210322-cirrussearch-content.json.gz
<13>Mar 25 03:15:09 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20210322/mniwiktionary-20210322-cirrussearch-general.json.gz
<13>Mar 25 16:37:11 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20210322/viwiki-20210322-cirrussearch-content.json.gz

Today's report:

<13>Apr  1 03:15:45 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20210329/mniwiktionary-20210329-cirrussearch-content.json.gz
<13>Apr  1 03:15:45 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20210329/mniwiktionary-20210329-cirrussearch-general.json.gz

Report from yesterday:

<13>Apr  7 10:30:30 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20210405/enwiki-20210405-cirrussearch-general.json.gz
<13>Apr  8 02:12:28 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20210405/mniwiktionary-20210405-cirrussearch-content.json.gz
<13>Apr  8 02:12:28 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20210405/mniwiktionary-20210405-cirrussearch-general.json.gz

Today's report:

<13>Apr 15 03:00:52 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20210412/mniwiktionary-20210412-cirrussearch-content.json.gz
<13>Apr 15 03:00:52 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20210412/mniwiktionary-20210412-cirrussearch-general.json.gz

I know folks are pretty busy, but a quick workaround could be: do retries on failure up to 3 times for a given wiki, or just rerunning the whole loop on failure and skipping over wikis which already have output files, if that makes more sense depending onthe reason for failure. That could go right in the bash script easily enough, if anyone has a few spare cycles.

Today's report:

<13>Apr 22 04:16:02 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20210419/mniwiktionary-20210419-cirrussearch-content.json.gz
<13>Apr 22 04:16:02 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20210419/mniwiktionary-20210419-cirrussearch-general.json.gz

Today's report:

<13>May  4 00:08:29 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20210503/commonswiki-20210503-cirrussearch-file.json.gz
MPhamWMF renamed this task from Cirrus Search dumps failed for some wikis to Make Cirrus Search dump script more resilient to failures (elasticsearch restarts).May 6 2021, 2:30 PM
MPhamWMF updated the task description. (Show Details)

Been awhile since I recorded these but, today's report:

<13>Jun 29 02:43:09 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20210628/commonswiki-20210628-cirrussearch-file.json.gz

Today's report:

<13>Jul 13 22:18:27 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20210712/enwiki-20210712-cirrussearch-general.json.gz

Today's report:

<13>Jul 29 13:55:42 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20210726/viwiki-20210726-cirrussearch-general.json.gz

Today's report:

<13>Aug  4 08:24:47 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20210802/fiwiki-20210802-cirrussearch-content.json.gz

Today's report:

<13>Sep 13 16:26:42 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20210913/arwiki-20210913-cirrussearch-content.json.gz
<13>Sep 13 16:26:53 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20210913/arwiki-20210913-cirrussearch-general.json.gz
<13>Sep 13 16:27:23 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20210913/arwikisource-20210913-cirrussearch-content.json.gz
<13>Sep 13 16:27:23 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20210913/arwikiversity-20210913-cirrussearch-content.json.gz
<13>Sep 13 16:27:23 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20210913/arwikiversity-20210913-cirrussearch-general.json.gz
<13>Sep 13 16:27:48 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20210913/arwiktionary-20210913-cirrussearch-general.json.gz
<13>Sep 13 16:27:49 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20210913/arywiki-20210913-cirrussearch-content.json.gz
<13>Sep 13 16:27:49 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20210913/arywiki-20210913-cirrussearch-general.json.gz
<13>Sep 13 16:28:16 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20210913/arzwiki-20210913-cirrussearch-content.json.gz
<13>Sep 13 16:28:16 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20210913/arzwiki-20210913-cirrussearch-general.json.gz
<13>Sep 13 16:28:17 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20210913/astwiki-20210913-cirrussearch-content.json.gz
<13>Sep 13 16:28:17 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20210913/astwiki-20210913-cirrussearch-general.json.gz
<13>Sep 13 16:28:17 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20210913/astwikibooks-20210913-cirrussearch-content.json.gz
<13>Sep 13 16:28:17 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20210913/astwikibooks-20210913-cirrussearch-general.json.gz
<13>Sep 13 16:28:18 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20210913/astwikiquote-20210913-cirrussearch-content.json.gz
<13>Sep 13 16:28:18 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20210913/astwikiquote-20210913-cirrussearch-general.json.gz
<13>Sep 13 16:28:18 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20210913/astwiktionary-20210913-cirrussearch-content.json.gz
<13>Sep 13 16:28:19 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20210913/astwiktionary-20210913-cirrussearch-general.json.gz
<13>Sep 13 16:28:19 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20210913/aswiki-20210913-cirrussearch-content.json.gz
<13>Sep 13 16:28:19 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20210913/aswiki-20210913-cirrussearch-general.json.gz
<13>Sep 13 16:28:20 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20210913/aswikibooks-20210913-cirrussearch-content.json.gz
<13>Sep 13 16:28:20 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20210913/aswikibooks-20210913-cirrussearch-general.json.gz
<13>Sep 13 16:28:20 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20210913/aswikisource-20210913-cirrussearch-content.json.gz
<13>Sep 13 16:28:20 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20210913/aswikisource-20210913-cirrussearch-general.json.gz
<13>Sep 13 16:28:21 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20210913/aswiktionary-20210913-cirrussearch-content.json.gz
<13>Sep 13 16:28:21 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20210913/aswiktionary-20210913-cirrussearch-general.json.gz
<13>Sep 13 16:28:22 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20210913/atjwiki-20210913-cirrussearch-content.json.gz
<13>Sep 13 16:28:22 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20210913/atjwiki-20210913-cirrussearch-general.json.gz
<13>Sep 13 16:28:23 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20210913/avkwiki-20210913-cirrussearch-content.json.gz
<13>Sep 13 16:28:23 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20210913/avkwiki-20210913-cirrussearch-general.json.gz
<13>Sep 13 16:28:23 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20210913/avwiki-20210913-cirrussearch-content.json.gz
<13>Sep 13 16:28:23 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20210913/avwiki-20210913-cirrussearch-general.json.gz
<13>Sep 13 16:28:24 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20210913/avwiktionary-20210913-cirrussearch-content.json.gz
<13>Sep 13 16:28:24 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20210913/avwiktionary-20210913-cirrussearch-general.json.gz
<13>Sep 13 16:28:24 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20210913/awawiki-20210913-cirrussearch-content.json.gz
<13>Sep 13 16:28:25 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20210913/awawiki-20210913-cirrussearch-general.json.gz
<13>Sep 13 16:28:25 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20210913/aywiki-20210913-cirrussearch-content.json.gz
<13>Sep 13 16:28:25 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20210913/aywiki-20210913-cirrussearch-general.json.gz
<13>Sep 13 16:28:25 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20210913/aywikibooks-20210913-cirrussearch-content.json.gz
<13>Sep 13 16:28:26 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20210913/aywikibooks-20210913-cirrussearch-general.json.gz
<13>Sep 13 16:28:26 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20210913/aywiktionary-20210913-cirrussearch-content.json.gz
<13>Sep 13 16:28:27 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20210913/aywiktionary-20210913-cirrussearch-general.json.gz

Today's report:

<13>Sep 23 21:05:23 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20210920/wikidatawiki-20210920-cirrussearch-general.json.gz
<13>Sep 23 22:06:52 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20210920/zhwiki-20210920-cirrussearch-general.json.gz

Today's report:

<13>Oct 13 14:50:02 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20211011/itwiki-20211011-cirrussearch-content.json.gz
<13>Oct 13 14:50:36 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20211011/itwiki-20211011-cirrussearch-general.json.gz