Page MenuHomePhabricator

Restart elasticsearch clusters to apply readahead changes
Open, HighPublic3 Estimated Story Points

Description

Overview

We performed a round of restarts following https://gerrit.wikimedia.org/r/c/operations/puppet/+/632319, but we later uncovered an issue that prevented the readahead changes from taking effect. We need to restart again in order to get the readahead shrinking change properly applied.

AC

  • Restart performed on all relevant elasticsearch nodes (relforge, cloudforge, eqiad, codfw)
    • relforge
    • cloudelastic
    • eqiad
    • codfw
  • Verified new readahead values are taking effect

DEPLOY NOTES

We'll deploy the production cirrus clusters (eqiad + codfw) first, and then cloudelastic and relforge later.

This is because for cloudelastic100[5,6] the readahead-udev rule doesn't work due to different partition. (See https://phabricator.wikimedia.org/T265699 to track progress on that fix)

Event Timeline

RKemper created this task.Mon, Oct 26, 5:47 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMon, Oct 26, 5:47 PM
CBogen set the point value for this task to 3.Mon, Oct 26, 6:50 PM

Mentioned in SAL (#wikimedia-operations) [2020-10-28T02:58:43Z] <ryankemper> T266492 Beginning rolling restart of codfw cirrus cluster, 3 nodes at a time, on ryankemper@cumin2001 tmux session elasticsearch_restart_codfw

Mentioned in SAL (#wikimedia-operations) [2020-10-28T04:43:45Z] <ryankemper> T266492 Finished rolling restart of codfw cirrus cluster

RKemper updated the task description. (Show Details)Wed, Oct 28, 6:23 AM
RKemper updated the task description. (Show Details)
Gehel triaged this task as High priority.Wed, Oct 28, 1:29 PM

Mentioned in SAL (#wikimedia-operations) [2020-10-29T01:17:24Z] <ryankemper> T266492 Beginning rolling restart of eqiad cirrus cluster, 3 nodes at a time, on ryankemper@cumin1001 tmux session elasticsearch_restart_eqiad

Is this complete now?

RKemper updated the task description. (Show Details)Mon, Nov 16, 4:37 PM