[Recurring task] CirrusSearch: what is updated during re-indexing
Open, NormalPublic

Description

Since we reindex on a somewhat infrequent basis, we should have a ticket that collects the updates that will happen during the next reindex.

This will be a recurring ticket, meaning that each time we do a reindex, we'll note it here and then use it again to list out new updates that will take effect the next time a reindex is done.

We should make or consider making announcements to the various communities that will be affected by the re-indexing, especially when there is a big gap between the discussion of the upcoming re-index and the actual re-index.

Items not yet done:

Items done:

Related Objects

There are a very large number of changes, so older changes are hidden. Show Older Changes
debt updated the task description. (Show Details)Jul 11 2017, 5:47 PM
debt updated the task description. (Show Details)Jul 11 2017, 5:49 PM

Updated ticket for keeping items in the description, rather than having to scroll down through the ticket to see what needs to be done and what is already done. :)

I think I reindexed hewiki (in-place) on the course of reindexing the wikis for archive. Is that enough or more thorough reindex is needed?

I think I reindexed hewiki (in-place) on the course of reindexing the wikis for archive. Is that enough or more thorough reindex is needed?

It needs to be re-indexed after the new language analyzer is deployed, which hasn't happened yet, so it still needs to be listed here.

Smalyshev updated the task description. (Show Details)Jul 27 2017, 12:53 AM
TJones updated the task description. (Show Details)Aug 16 2017, 10:19 PM
EBernhardson updated the task description. (Show Details)Sep 20 2017, 2:58 PM
dcausse updated the task description. (Show Details)Sep 21 2017, 3:52 PM
EBernhardson updated the task description. (Show Details)Sep 21 2017, 3:55 PM
TJones updated the task description. (Show Details)Sep 25 2017, 5:47 PM

I added a note to consider who should be notified of upcoming re-indexes. Hebrew was delayed for months, so no one was expecting the change when it happened.

A list of places to make any future announcement would be useful, but I'm not sure what all to include.

T147959 should probably be added here?

TJones updated the task description. (Show Details)Oct 10 2017, 7:07 PM

T147959 should probably be added here?

I created a new sub-task ( T177871) which has been added to the task description.

Smalyshev updated the task description. (Show Details)Oct 17 2017, 5:29 PM
TJones updated the task description. (Show Details)Oct 24 2017, 4:10 PM
Smalyshev updated the task description. (Show Details)Oct 24 2017, 5:27 PM
Smalyshev updated the task description. (Show Details)Nov 27 2017, 6:40 PM
debt updated the task description. (Show Details)Dec 19 2017, 6:21 PM
Smalyshev updated the task description. (Show Details)Jan 3 2018, 9:16 PM
Smalyshev updated the task description. (Show Details)Mar 1 2018, 1:15 AM
TJones updated the task description. (Show Details)Mar 5 2018, 5:49 PM
TJones updated the task description. (Show Details)Apr 5 2018, 4:41 PM
Smalyshev updated the task description. (Show Details)May 8 2018, 11:22 PM
Smalyshev updated the task description. (Show Details)May 25 2018, 5:37 AM
Smalyshev updated the task description. (Show Details)
Smalyshev updated the task description. (Show Details)Jun 1 2018, 5:33 PM
EBernhardson updated the task description. (Show Details)Jun 1 2018, 7:46 PM
EBernhardson updated the task description. (Show Details)Jun 1 2018, 7:58 PM
TJones updated the task description. (Show Details)Jun 4 2018, 8:03 PM
TJones updated the task description. (Show Details)Jun 5 2018, 6:46 PM
TJones updated the task description. (Show Details)Jun 7 2018, 5:15 PM
Smalyshev updated the task description. (Show Details)Jun 8 2018, 11:49 PM
TJones updated the task description. (Show Details)Jun 18 2018, 2:26 AM
TJones updated the task description. (Show Details)Jun 19 2018, 6:16 PM
TJones updated the task description. (Show Details)Jun 21 2018, 5:49 PM
TJones updated the task description. (Show Details)Jul 13 2018, 6:54 PM
TJones updated the task description. (Show Details)Jul 19 2018, 6:54 PM
TJones updated the task description. (Show Details)Jul 23 2018, 2:52 PM
Smalyshev updated the task description. (Show Details)Aug 3 2018, 6:31 PM
TJones updated the task description. (Show Details)Aug 14 2018, 12:58 PM
TJones updated the task description. (Show Details)Aug 15 2018, 12:52 PM
Smalyshev updated the task description. (Show Details)Aug 20 2018, 7:12 PM
TJones updated the task description. (Show Details)Tue, Aug 28, 4:24 PM
Smalyshev updated the task description. (Show Details)Wed, Sep 12, 5:28 AM
TJones updated the task description. (Show Details)Thu, Sep 13, 5:08 PM
EBernhardson updated the task description. (Show Details)Thu, Sep 20, 4:13 PM

Mentioned in SAL (#wikimedia-operations) [2018-09-20T18:43:01Z] <SMalyshev> Initiating in-place reindex for wikidatawiki (T147505)

Mentioned in SAL (#wikimedia-operations) [2018-09-20T18:43:01Z] <SMalyshev> Initiating in-place reindex for wikidatawiki (T147505)

https://logstash.wikimedia.org/goto/e93c1017eba4c664a89b0dd90e52395b

50x each (from mwmaint2001)
ErrorException from line 150 of /srv/mediawiki/php-1.32.0-wmf.22/extensions/CirrusSearch/includes/Elastica/ReindexTask.php: PHP Notice: Undefined index: total
ErrorException from line 151 of /srv/mediawiki/php-1.32.0-wmf.22/extensions/CirrusSearch/includes/Elastica/ReindexTask.php: PHP Notice: Undefined index: updated
ErrorException from line 152 of /srv/mediawiki/php-1.32.0-wmf.22/extensions/CirrusSearch/includes/Elastica/ReindexTask.php: PHP Notice: Undefined index: created
ErrorException from line 153 of /srv/mediawiki/php-1.32.0-wmf.22/extensions/CirrusSearch/includes/Elastica/ReindexTask.php: PHP Notice: Undefined index: deleted
ErrorException from line 154 of /srv/mediawiki/php-1.32.0-wmf.22/extensions/CirrusSearch/includes/Elastica/ReindexTask.php: PHP Notice: Undefined index: batches
ErrorException from line 155 of /srv/mediawiki/php-1.32.0-wmf.22/extensions/CirrusSearch/includes/Elastica/ReindexTask.php: PHP Notice: Undefined index: version_conflicts
ErrorException from line 156 of /srv/mediawiki/php-1.32.0-wmf.22/extensions/CirrusSearch/includes/Elastica/ReindexTask.php: PHP Notice: Undefined index: noops
...
...

Also, possibly related, happened in the middle of it

3900x (from web servers)
PHP Warning: Retrying connection to search.svc.codfw.wmnet after 1 attempts. [Called from Closure$ElasticaConnection::getClient

PHP Warning: Retrying connection to search.svc.codfw.wmnet after 2 attempts. [Called from Closure$ElasticaConnection::getClient

Thanks, and what about the spike in connection failures?

Thanks, and what about the spike in connection failures?

No idea, from logstash those do not appear to be related, but maybe they are.

Gehel added a subscriber: Gehel.Thu, Sep 20, 7:36 PM

I correlate that with a spike in pool counter rejections, which is correlated to 3 nodes restarting in the cluster (there is a full cluster restart going on on codfw). This spike did not happen for the last ~50 servers I restarted, so not sure if it is a coincidence or not. Looking...

Gehel added a comment.Thu, Sep 20, 7:38 PM

It looks like it was the master re-election during the cluster restart. Sadly I don't think there is much we can do about this.