Page MenuHomePhabricator

[EPIC][Recurring task] CirrusSearch: what is updated during re-indexing
Open, MediumPublic

Description

Since we reindex on a somewhat infrequent basis, we should have a ticket that collects the updates that will happen during the next reindex.

This will be a recurring ticket, meaning that each time we do a reindex, we'll note it here and then use it again to list out new updates that will take effect the next time a reindex is done.

We should make or consider making announcements to the various communities that will be affected by the re-indexing, especially when there is a big gap between the discussion of the upcoming re-index and the actual re-index.

Items not yet done:

Items done:

Related Objects

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Smalyshev updated the task description. (Show Details)May 25 2018, 5:38 AM
Smalyshev updated the task description. (Show Details)Jun 1 2018, 5:33 PM
EBernhardson updated the task description. (Show Details)Jun 1 2018, 7:46 PM
EBernhardson updated the task description. (Show Details)Jun 1 2018, 7:58 PM
TJones updated the task description. (Show Details)Jun 4 2018, 8:03 PM
TJones updated the task description. (Show Details)Jun 5 2018, 6:46 PM
TJones updated the task description. (Show Details)Jun 7 2018, 5:15 PM
Smalyshev updated the task description. (Show Details)Jun 8 2018, 11:49 PM
TJones updated the task description. (Show Details)Jun 18 2018, 2:26 AM
TJones updated the task description. (Show Details)Jun 19 2018, 6:16 PM
TJones updated the task description. (Show Details)Jun 21 2018, 5:49 PM
TJones updated the task description. (Show Details)Jul 13 2018, 6:54 PM
TJones updated the task description. (Show Details)Jul 19 2018, 6:54 PM
TJones updated the task description. (Show Details)Jul 23 2018, 2:52 PM
Smalyshev updated the task description. (Show Details)Aug 3 2018, 6:31 PM
TJones updated the task description. (Show Details)Aug 14 2018, 12:58 PM
TJones updated the task description. (Show Details)Aug 15 2018, 12:52 PM
Smalyshev updated the task description. (Show Details)Aug 20 2018, 7:12 PM
TJones updated the task description. (Show Details)Aug 28 2018, 4:24 PM
Smalyshev updated the task description. (Show Details)Sep 12 2018, 5:28 AM
TJones updated the task description. (Show Details)Sep 13 2018, 5:08 PM
EBernhardson updated the task description. (Show Details)Sep 20 2018, 4:13 PM

Mentioned in SAL (#wikimedia-operations) [2018-09-20T18:43:01Z] <SMalyshev> Initiating in-place reindex for wikidatawiki (T147505)

Mentioned in SAL (#wikimedia-operations) [2018-09-20T18:43:01Z] <SMalyshev> Initiating in-place reindex for wikidatawiki (T147505)

https://logstash.wikimedia.org/goto/e93c1017eba4c664a89b0dd90e52395b

50x each (from mwmaint2001)
ErrorException from line 150 of /srv/mediawiki/php-1.32.0-wmf.22/extensions/CirrusSearch/includes/Elastica/ReindexTask.php: PHP Notice: Undefined index: total
ErrorException from line 151 of /srv/mediawiki/php-1.32.0-wmf.22/extensions/CirrusSearch/includes/Elastica/ReindexTask.php: PHP Notice: Undefined index: updated
ErrorException from line 152 of /srv/mediawiki/php-1.32.0-wmf.22/extensions/CirrusSearch/includes/Elastica/ReindexTask.php: PHP Notice: Undefined index: created
ErrorException from line 153 of /srv/mediawiki/php-1.32.0-wmf.22/extensions/CirrusSearch/includes/Elastica/ReindexTask.php: PHP Notice: Undefined index: deleted
ErrorException from line 154 of /srv/mediawiki/php-1.32.0-wmf.22/extensions/CirrusSearch/includes/Elastica/ReindexTask.php: PHP Notice: Undefined index: batches
ErrorException from line 155 of /srv/mediawiki/php-1.32.0-wmf.22/extensions/CirrusSearch/includes/Elastica/ReindexTask.php: PHP Notice: Undefined index: version_conflicts
ErrorException from line 156 of /srv/mediawiki/php-1.32.0-wmf.22/extensions/CirrusSearch/includes/Elastica/ReindexTask.php: PHP Notice: Undefined index: noops
...
...

Also, possibly related, happened in the middle of it

3900x (from web servers)
PHP Warning: Retrying connection to search.svc.codfw.wmnet after 1 attempts. [Called from Closure$ElasticaConnection::getClient

PHP Warning: Retrying connection to search.svc.codfw.wmnet after 2 attempts. [Called from Closure$ElasticaConnection::getClient

Thanks, and what about the spike in connection failures?

Thanks, and what about the spike in connection failures?

No idea, from logstash those do not appear to be related, but maybe they are.

Gehel added a subscriber: Gehel.Sep 20 2018, 7:36 PM

I correlate that with a spike in pool counter rejections, which is correlated to 3 nodes restarting in the cluster (there is a full cluster restart going on on codfw). This spike did not happen for the last ~50 servers I restarted, so not sure if it is a coincidence or not. Looking...

Gehel added a comment.Sep 20 2018, 7:38 PM

It looks like it was the master re-election during the cluster restart. Sadly I don't think there is much we can do about this.

Smalyshev updated the task description. (Show Details)Sep 27 2018, 7:51 PM
TJones updated the task description. (Show Details)Nov 9 2018, 3:08 PM
TJones updated the task description. (Show Details)Nov 17 2018, 4:18 PM
debt updated the task description. (Show Details)Feb 5 2019, 6:38 PM
TJones updated the task description. (Show Details)Feb 21 2019, 4:54 PM
Restricted Application added a subscriber: revi. · View Herald TranscriptFeb 21 2019, 4:54 PM
Smalyshev updated the task description. (Show Details)Feb 28 2019, 11:55 PM
TJones updated the task description. (Show Details)Mar 6 2019, 11:06 PM
TJones updated the task description. (Show Details)Apr 10 2019, 3:11 PM
Smalyshev updated the task description. (Show Details)Apr 23 2019, 7:23 PM
Smalyshev updated the task description. (Show Details)Apr 27 2019, 11:41 PM
Smalyshev updated the task description. (Show Details)
TJones updated the task description. (Show Details)Apr 28 2019, 4:07 AM
Smalyshev updated the task description. (Show Details)Jun 5 2019, 4:41 AM
TJones updated the task description. (Show Details)Oct 16 2019, 1:54 PM
TJones updated the task description. (Show Details)Oct 29 2019, 8:20 PM
Mstyles updated the task description. (Show Details)Feb 7 2020, 5:58 PM
Mstyles updated the task description. (Show Details)
Krinkle removed a subscriber: Krinkle.Feb 7 2020, 7:19 PM
EBernhardson updated the task description. (Show Details)Feb 10 2020, 6:40 PM
EBernhardson updated the task description. (Show Details)Mar 5 2020, 12:07 AM
TJones renamed this task from [Recurring task] CirrusSearch: what is updated during re-indexing to [EPIC][Recurring task] CirrusSearch: what is updated during re-indexing.Mar 23 2020, 5:36 PM
TJones moved this task from Incoming to Epics on the Discovery-Search (Current work) board.
TJones added a project: Epic.
Mstyles updated the task description. (Show Details)May 5 2020, 4:54 PM
Mstyles updated the task description. (Show Details)
Mstyles updated the task description. (Show Details)Jun 18 2020, 3:46 PM
Aklapper removed debt as the assignee of this task.Jun 19 2020, 4:22 PM

This task has been assigned to the same task owner for more than two years. Resetting task assignee due to inactivity, to decrease task cookie-licking and to get a slightly more realistic overview of plans. Please feel free to assign this task to yourself again if you still realistically work or plan to work on this task - it would be welcome!

For tips how to manage individual work in Phabricator (noisy notifications, lists of task, etc.), see https://phabricator.wikimedia.org/T228575#6237124 for available options.
(For the records, two emails were sent to assignee addresses before resetting assignees. See T228575 for more info and for potential feedback. Thanks!)

@Aklapper thanks for tending the garden! This task doesn't really have an end date because it is the parent Epic for all reindexing tasks (so we can find them, and we know what is currently on the queue). There will always be new reindexing tasks. And, at the time @debt claimed it, she was arguably the person most likely to be responsible for making sure that the tasks eventually got done.

So, what's the right way to do handle this situation? Leave it unclaimed? Are either [EPIC] or [Recurring task] (or both together) enough?

Aklapper assigned this task to debt.Jun 22 2020, 8:33 PM
Aklapper added a project: Tracking-Neverending.

Uh, I apologize for the interruption! I guess I should have somehow excluded this task. Let's add Tracking-Neverending (and I'll update my SQL query).

Thanks, Andre!