⚓ T157722 Evaluate problems reported by the elasticsearch migration plugin on prod search clusters

Status	Assigned	Task
Resolved	debt	T151324 [epic] System level upgrade for cirrus / elasticsearch
Resolved	• Deskana	T154501 [Epic, Q3 Goal] Upgrade search systems to Elasticsearch 5
Resolved	EBernhardson	T157722 Evaluate problems reported by the elasticsearch migration plugin on prod search clusters

EBernhardson created this task.Feb 9 2017, 7:59 PM

First order of business: the plugin makes no attempt to deduplicate errors, so we get ~3000 problem reports. This bit of javascript will do the deduplication:

(function (undefined) {
  function hashCode(str) {
    var hash = 0, i, chr, len;
    if (str.length === 0) return hash;
    for (i = 0, len = str.length; i < len; i++) {
      chr   = str.charCodeAt(i);
      hash  = ((hash << 5) - hash) + chr;
      hash |= 0; // Convert to 32bit integer
    }
    return hash;
  };


  function merge($container, target_selector) { 
    var errors = {},
        $targets = $container.children(target_selector);
  
    $targets.each(function (idx, el) {
      var $el = $(el),
          name = $el.children('span').text(),
          hash = hashCode($el.children('ul').html());
  
      if (errors[hash] === undefined) {
          errors[hash] = {
              indices: [name],
              content: $el.children('ul').html(),
              status: $el.attr('class')
          }
      } else {
          errors[hash].indices.push(name)
      }
    });
  
 
    $targets.remove();
 
    for (hash in errors) {
      if (!errors.hasOwnProperty(hash)) {
        continue;
      }
      $container.append($('<li>')
          .attr('class', errors[hash].status)
          .html('<span><code>' + errors[hash].indices.join('</code></span><span><code>') + '</code></span><ul>' + errors[hash].content + '</ul>')
      );
    }
  }

  function mergeIndices() {
    // We don't care about warmers, and they vary
    $( 'li.section.warmers.blue' ).remove();
    merge($('span:contains("Indices")').siblings('ul'), 'li.index:not(.status.green)');
  }

  function mergeNodes() {
    merge($('span:contains("Node Settings")').siblings('ul'), 'li.node:not(.status.green)');
  }

  mergeIndices();
  mergeNodes();
})();

EBernhardson updated the task description. (Show Details)Feb 9 2017, 10:54 PM

EBernhardson updated the task description. (Show Details)

EBernhardson moved this task from Incoming to Needs Reporting on the Discovery-Search (Current work) board.Feb 9 2017, 11:31 PM

EBernhardson moved this task from Needs Reporting to not in use - please delete on the Discovery-Search (Current work) board.

EBernhardson updated the task description. (Show Details)Feb 10 2017, 12:33 AM

EBernhardson updated the task description. (Show Details)Feb 14 2017, 10:58 PM

EBernhardson updated the task description. (Show Details)Feb 14 2017, 11:07 PM

EBernhardson updated the task description. (Show Details)

EBernhardson updated the task description. (Show Details)Feb 14 2017, 11:29 PM

EBernhardson updated the task description. (Show Details)Feb 17 2017, 10:19 PM

Everything on the list has in-progress work. For reference the deduplicated list of patches/tickets that need to be resolved is:

EBernhardson moved this task from not in use - please delete to Needs review on the Discovery-Search (Current work) board.Feb 17 2017, 10:24 PM

Even after the update to use BM25 directly and reindexing, i'm still seeing a handful of wikis in codfw (the only cluster currently finished) referencing classic similarity. Will investigate more.

Ignore that, they are supposed to refer to classic similarity, they are the spaceless languages.

EBernhardson updated the task description. (Show Details)Feb 27 2017, 4:38 PM

EBernhardson updated the task description. (Show Details)Feb 27 2017, 5:23 PM

I re-reviewed the output and everything in eqiad and codfw looks good to go now. All related patches have been merged, moving to done.

EBernhardson moved this task from Needs review to Needs Reporting on the Discovery-Search (Current work) board.Feb 27 2017, 5:43 PM

• Deskana closed this task as Resolved.Mar 31 2017, 12:47 PM

Evaluate problems reported by the elasticsearch migration plugin on prod search clusters
Closed, ResolvedPublic
Actions

Description

Problems reported related to node configuration:

Problems reported related to indices

Related Objects
Search...

Event Timeline

Evaluate problems reported by the elasticsearch migration plugin on prod search clustersClosed, ResolvedPublicActions