The update to elasticsearch 7.10.2 has seen these clusters issue
circuit breaking exceptions where elasticsearch has determined they
don't have enough memory to perform various actions. These clusters
have been at 8G since creation in 2018, lets increase by 25% to
10G to give them more room to breathe.
Description
Description
Related Objects
Related Objects
Event Timeline
Comment Actions
Creating this ticket retroactively to link the following code changes:
https://gerrit.wikimedia.org/r/c/operations/puppet/+/855673
https://gerrit.wikimedia.org/r/c/operations/puppet/+/859094
This change has already been applied everywhere except eqiad production, which is currently in progress as I write this. Will update and close upon completion.
Comment Actions
Cloudelastic had a few hosts that didn't have the latest heap size applied; most likely the rolling restart was done before sufficient time had passed for puppet to auto-run on each instance.
Once the cloudelastic restart completes, this ticket should be done since eqiad/codfw are already done.
Comment Actions
Checked via:
gehel@cumin1001:~$ sudo cumin 'A:cloudelastic' 'pgrep -c -f Xmx10G' 6 hosts will be targeted: cloudelastic[1001-1006].wikimedia.org Ok to proceed on 6 hosts? Enter the number of affected hosts to confirm or "q" to quit 6 ===== NODE GROUP ===== (6) cloudelastic[1001-1006].wikimedia.org ----- OUTPUT of 'pgrep -c -f Xmx10G' ----- 2 ================ PASS |██████████████████████████████████| 100% (6/6) [00:00<00:00, 9.53hosts/s] FAIL | | 0% (0/6) [00:00<?, ?hosts/s] 100.0% (6/6) success ratio (>= 100.0% threshold) for command: 'pgrep -c -f Xmx10G'. 100.0% (6/6) success ratio (>= 100.0% threshold) of nodes successfully executed all commands. gehel@cumin1001:~$ sudo cumin 'A:elastic' 'pgrep -c -f Xmx10G' 99 hosts will be targeted: elastic[2037-2048,2050-2086].codfw.wmnet,elastic[1053-1102].eqiad.wmnet Ok to proceed on 99 hosts? Enter the number of affected hosts to confirm or "q" to quit 99 ===== NODE GROUP ===== (99) elastic[2037-2048,2050-2086].codfw.wmnet,elastic[1053-1102].eqiad.wmnet ----- OUTPUT of 'pgrep -c -f Xmx10G' ----- 1 ================ PASS |████████████████████████████████| 100% (99/99) [00:01<00:00, 50.80hosts/s] FAIL | | 0% (0/99) [00:01<?, ?hosts/s] 100.0% (99/99) success ratio (>= 100.0% threshold) for command: 'pgrep -c -f Xmx10G'. 100.0% (99/99) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.