Page MenuHomePhabricator

[tracking] CirrusSearch: what is updated during re-indexing
Open, MediumPublic

Description

Since we reindex on a somewhat infrequent basis, we should have a ticket that collects the updates that will happen during the next reindex.

This will be a recurring ticket, meaning that each time we do a reindex, we'll note it here and then use it again to list out new updates that will take effect the next time a reindex is done.

We should make or consider making announcements to the various communities that will be affected by the re-indexing, especially when there is a big gap between the discussion of the upcoming re-index and the actual re-index.

Items not yet done:

Items done:

Related Objects

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Mentioned in SAL (#wikimedia-operations) [2018-09-20T18:43:01Z] <SMalyshev> Initiating in-place reindex for wikidatawiki (T147505)

Mentioned in SAL (#wikimedia-operations) [2018-09-20T18:43:01Z] <SMalyshev> Initiating in-place reindex for wikidatawiki (T147505)

https://logstash.wikimedia.org/goto/e93c1017eba4c664a89b0dd90e52395b

50x each (from mwmaint2001)
ErrorException from line 150 of /srv/mediawiki/php-1.32.0-wmf.22/extensions/CirrusSearch/includes/Elastica/ReindexTask.php: PHP Notice: Undefined index: total
ErrorException from line 151 of /srv/mediawiki/php-1.32.0-wmf.22/extensions/CirrusSearch/includes/Elastica/ReindexTask.php: PHP Notice: Undefined index: updated
ErrorException from line 152 of /srv/mediawiki/php-1.32.0-wmf.22/extensions/CirrusSearch/includes/Elastica/ReindexTask.php: PHP Notice: Undefined index: created
ErrorException from line 153 of /srv/mediawiki/php-1.32.0-wmf.22/extensions/CirrusSearch/includes/Elastica/ReindexTask.php: PHP Notice: Undefined index: deleted
ErrorException from line 154 of /srv/mediawiki/php-1.32.0-wmf.22/extensions/CirrusSearch/includes/Elastica/ReindexTask.php: PHP Notice: Undefined index: batches
ErrorException from line 155 of /srv/mediawiki/php-1.32.0-wmf.22/extensions/CirrusSearch/includes/Elastica/ReindexTask.php: PHP Notice: Undefined index: version_conflicts
ErrorException from line 156 of /srv/mediawiki/php-1.32.0-wmf.22/extensions/CirrusSearch/includes/Elastica/ReindexTask.php: PHP Notice: Undefined index: noops
...
...

Also, possibly related, happened in the middle of it

3900x (from web servers)
PHP Warning: Retrying connection to search.svc.codfw.wmnet after 1 attempts. [Called from Closure$ElasticaConnection::getClient

PHP Warning: Retrying connection to search.svc.codfw.wmnet after 2 attempts. [Called from Closure$ElasticaConnection::getClient

Thanks, and what about the spike in connection failures?

Thanks, and what about the spike in connection failures?

No idea, from logstash those do not appear to be related, but maybe they are.

I correlate that with a spike in pool counter rejections, which is correlated to 3 nodes restarting in the cluster (there is a full cluster restart going on on codfw). This spike did not happen for the last ~50 servers I restarted, so not sure if it is a coincidence or not. Looking...

It looks like it was the master re-election during the cluster restart. Sadly I don't think there is much we can do about this.

Mstyles updated the task description. (Show Details)
TJones renamed this task from [Recurring task] CirrusSearch: what is updated during re-indexing to [EPIC][Recurring task] CirrusSearch: what is updated during re-indexing.Mar 23 2020, 5:36 PM
TJones moved this task from Incoming to Epics on the Discovery-Search (Current work) board.
TJones added a project: Epic.
Mstyles updated the task description. (Show Details)

This task has been assigned to the same task owner for more than two years. Resetting task assignee due to inactivity, to decrease task cookie-licking and to get a slightly more realistic overview of plans. Please feel free to assign this task to yourself again if you still realistically work or plan to work on this task - it would be welcome!

For tips how to manage individual work in Phabricator (noisy notifications, lists of task, etc.), see https://phabricator.wikimedia.org/T228575#6237124 for available options.
(For the records, two emails were sent to assignee addresses before resetting assignees. See T228575 for more info and for potential feedback. Thanks!)

@Aklapper thanks for tending the garden! This task doesn't really have an end date because it is the parent Epic for all reindexing tasks (so we can find them, and we know what is currently on the queue). There will always be new reindexing tasks. And, at the time @debt claimed it, she was arguably the person most likely to be responsible for making sure that the tasks eventually got done.

So, what's the right way to do handle this situation? Leave it unclaimed? Are either [EPIC] or [Recurring task] (or both together) enough?

Aklapper added a project: Tracking-Neverending.

Uh, I apologize for the interruption! I guess I should have somehow excluded this task. Let's add Tracking-Neverending (and I'll update my SQL query).

Gehel renamed this task from [EPIC][Recurring task] CirrusSearch: what is updated during re-indexing to [tracking] CirrusSearch: what is updated during re-indexing.Oct 6 2020, 2:57 PM

There was some question on whether this ticket is needed as a tracking task. Consensus is that we should keep it. There hasn't been much activity lately as there hasn't been much focus on CirrusSearch, but it is still a good way to keep track on reindexes.