Page MenuHomePhabricator

[Recurring task] CirrusSearch: what is updated during re-indexing
Open, NormalPublic

Description

Since we reindex on a somewhat infrequent basis, we should have a ticket that collects the updates that will happen during the next reindex.

This will be a recurring ticket, meaning that each time we do a reindex, we'll note it here and then use it again to list out new updates that will take effect the next time a reindex is done.

We should make or consider making announcements to the various communities that will be affected by the re-indexing, especially when there is a big gap between the discussion of the upcoming re-index and the actual re-index.

Items not yet done:

Items done:

Related Objects

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

T147959 should probably be added here?

TJones updated the task description. (Show Details)Oct 10 2017, 7:07 PM

T147959 should probably be added here?

I created a new sub-task ( T177871) which has been added to the task description.

Smalyshev updated the task description. (Show Details)Oct 17 2017, 5:29 PM
TJones updated the task description. (Show Details)Oct 24 2017, 4:10 PM
Smalyshev updated the task description. (Show Details)Oct 24 2017, 5:27 PM
Smalyshev updated the task description. (Show Details)Nov 27 2017, 6:40 PM
debt updated the task description. (Show Details)Dec 19 2017, 6:21 PM
Smalyshev updated the task description. (Show Details)Jan 3 2018, 9:16 PM
Smalyshev updated the task description. (Show Details)Mar 1 2018, 1:15 AM
TJones updated the task description. (Show Details)Mar 5 2018, 5:49 PM
TJones updated the task description. (Show Details)Apr 5 2018, 4:41 PM
Smalyshev updated the task description. (Show Details)May 8 2018, 11:22 PM
Smalyshev updated the task description. (Show Details)May 25 2018, 5:37 AM
Smalyshev updated the task description. (Show Details)
Smalyshev updated the task description. (Show Details)Jun 1 2018, 5:33 PM
EBernhardson updated the task description. (Show Details)Jun 1 2018, 7:46 PM
EBernhardson updated the task description. (Show Details)Jun 1 2018, 7:58 PM
TJones updated the task description. (Show Details)Jun 4 2018, 8:03 PM
TJones updated the task description. (Show Details)Jun 5 2018, 6:46 PM
TJones updated the task description. (Show Details)Jun 7 2018, 5:15 PM
Smalyshev updated the task description. (Show Details)Jun 8 2018, 11:49 PM
TJones updated the task description. (Show Details)Jun 18 2018, 2:26 AM
TJones updated the task description. (Show Details)Jun 19 2018, 6:16 PM
TJones updated the task description. (Show Details)Jun 21 2018, 5:49 PM
TJones updated the task description. (Show Details)Jul 13 2018, 6:54 PM
TJones updated the task description. (Show Details)Jul 19 2018, 6:54 PM
TJones updated the task description. (Show Details)Jul 23 2018, 2:52 PM
Smalyshev updated the task description. (Show Details)Aug 3 2018, 6:31 PM
TJones updated the task description. (Show Details)Aug 14 2018, 12:58 PM
TJones updated the task description. (Show Details)Aug 15 2018, 12:52 PM
Smalyshev updated the task description. (Show Details)Aug 20 2018, 7:12 PM
TJones updated the task description. (Show Details)Aug 28 2018, 4:24 PM
Smalyshev updated the task description. (Show Details)Sep 12 2018, 5:28 AM
TJones updated the task description. (Show Details)Sep 13 2018, 5:08 PM
EBernhardson updated the task description. (Show Details)Sep 20 2018, 4:13 PM

Mentioned in SAL (#wikimedia-operations) [2018-09-20T18:43:01Z] <SMalyshev> Initiating in-place reindex for wikidatawiki (T147505)

Mentioned in SAL (#wikimedia-operations) [2018-09-20T18:43:01Z] <SMalyshev> Initiating in-place reindex for wikidatawiki (T147505)

https://logstash.wikimedia.org/goto/e93c1017eba4c664a89b0dd90e52395b

50x each (from mwmaint2001)
ErrorException from line 150 of /srv/mediawiki/php-1.32.0-wmf.22/extensions/CirrusSearch/includes/Elastica/ReindexTask.php: PHP Notice: Undefined index: total
ErrorException from line 151 of /srv/mediawiki/php-1.32.0-wmf.22/extensions/CirrusSearch/includes/Elastica/ReindexTask.php: PHP Notice: Undefined index: updated
ErrorException from line 152 of /srv/mediawiki/php-1.32.0-wmf.22/extensions/CirrusSearch/includes/Elastica/ReindexTask.php: PHP Notice: Undefined index: created
ErrorException from line 153 of /srv/mediawiki/php-1.32.0-wmf.22/extensions/CirrusSearch/includes/Elastica/ReindexTask.php: PHP Notice: Undefined index: deleted
ErrorException from line 154 of /srv/mediawiki/php-1.32.0-wmf.22/extensions/CirrusSearch/includes/Elastica/ReindexTask.php: PHP Notice: Undefined index: batches
ErrorException from line 155 of /srv/mediawiki/php-1.32.0-wmf.22/extensions/CirrusSearch/includes/Elastica/ReindexTask.php: PHP Notice: Undefined index: version_conflicts
ErrorException from line 156 of /srv/mediawiki/php-1.32.0-wmf.22/extensions/CirrusSearch/includes/Elastica/ReindexTask.php: PHP Notice: Undefined index: noops
...
...

Also, possibly related, happened in the middle of it

3900x (from web servers)
PHP Warning: Retrying connection to search.svc.codfw.wmnet after 1 attempts. [Called from Closure$ElasticaConnection::getClient

PHP Warning: Retrying connection to search.svc.codfw.wmnet after 2 attempts. [Called from Closure$ElasticaConnection::getClient

Thanks, and what about the spike in connection failures?

Thanks, and what about the spike in connection failures?

No idea, from logstash those do not appear to be related, but maybe they are.

Gehel added a subscriber: Gehel.Sep 20 2018, 7:36 PM

I correlate that with a spike in pool counter rejections, which is correlated to 3 nodes restarting in the cluster (there is a full cluster restart going on on codfw). This spike did not happen for the last ~50 servers I restarted, so not sure if it is a coincidence or not. Looking...

Gehel added a comment.Sep 20 2018, 7:38 PM

It looks like it was the master re-election during the cluster restart. Sadly I don't think there is much we can do about this.

Smalyshev updated the task description. (Show Details)Sep 27 2018, 7:51 PM
TJones updated the task description. (Show Details)Nov 9 2018, 3:08 PM
TJones updated the task description. (Show Details)Nov 17 2018, 4:18 PM
debt updated the task description. (Show Details)Feb 5 2019, 6:38 PM
TJones updated the task description. (Show Details)Feb 21 2019, 4:54 PM
Restricted Application added a subscriber: revi. · View Herald TranscriptFeb 21 2019, 4:54 PM
Smalyshev updated the task description. (Show Details)Feb 28 2019, 11:55 PM
TJones updated the task description. (Show Details)Mar 6 2019, 11:06 PM
TJones updated the task description. (Show Details)Apr 10 2019, 3:11 PM
Smalyshev updated the task description. (Show Details)Apr 23 2019, 7:23 PM
Smalyshev updated the task description. (Show Details)Apr 27 2019, 11:41 PM
Smalyshev updated the task description. (Show Details)
TJones updated the task description. (Show Details)Apr 28 2019, 4:07 AM
Smalyshev updated the task description. (Show Details)Jun 5 2019, 4:41 AM