Page MenuHomePhabricator

Reindex produces a lot of Undefined index messages
Open, NormalPublic

Description

While reindexing Wikidata:

        Task: 1lpe9N1FQxOGsfOcY969RA:2610140 Search Retries: 0 Bulk Retries: 0 Indexed: 584100 / 50278595                                             
        Task: 1lpe9N1FQxOGsfOcY969RA:2610140 Search Retries: 0 Bulk Retries: 0 Indexed: 623300 / 50278595                                             
[Thu Sep 20 18:59:19 2018] [hphp] [1252:7ffb46e4d3c0:0:000001] []                                                                                     
Notice: Undefined index: total in /srv/mediawiki/php-1.32.0-wmf.22/extensions/CirrusSearch/includes/Elastica/ReindexTask.php on line 150              
[Thu Sep 20 18:59:19 2018] [hphp] [1252:7ffb46e4d3c0:0:000002] []                                                                                     
Notice: Undefined index: updated in /srv/mediawiki/php-1.32.0-wmf.22/extensions/CirrusSearch/includes/Elastica/ReindexTask.php on line 151

etc. lots of messages like this.

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 20 2018, 7:27 PM

Is this from the task reindexing the same cluster that is having a restart (codfw, iirc)? I don't think elasticsearch tasks or scrolls are able to move between hosts and will fail in the face of one of the nodes participating restarting. The error message also means we don't detect that case, there is probably no particular error handling in the Cirrus side of the reindexing code to detect errors.

Yep, according to @Gehel there were restarts going on in the same time (my luck... sigh, should have checked) so maybe it's caused by restarts. Will try again and see. We probably should somehow detect this case anyway.

EBjune triaged this task as Normal priority.Sep 27 2018, 5:05 PM
EBjune moved this task from needs triage to Tech Debt/Misc on the Discovery-Search board.

For clarity: the message does not appear anymore, but I don't close the ticket so that we add some proper handling for this case.