| elukey | |
| Feb 13 2017, 2:29 PM |
| F5624598: Screen Shot 2017-02-13 at 3.53.40 PM.png | |
| Feb 13 2017, 2:54 PM |
| F5624455: Screen Shot 2017-02-13 at 3.25.40 PM.png | |
| Feb 13 2017, 2:29 PM |
| F5624466: Screen Shot 2017-02-13 at 3.27.49 PM.png | |
| Feb 13 2017, 2:29 PM |
| F5624456: Screen Shot 2017-02-13 at 3.24.53 PM.png | |
| Feb 13 2017, 2:29 PM |
While Moritz was restarting zookeeper on conf1001 for jvm security upgrades, I noticed a weird pattern in memory usage for the past 90 days:
It is also nice to notice the huge drop on heap usage after Moritz's restart of zookeeper on conf1001:
Last but not the least, I don't see any -Xmx JVM parameter, that should indicate a maximum heap size set by the JVM itself?
Change 337413 had a related patch set uploaded (by Elukey):
Add default JAVA_OPTS to the zookeeper server class
Moritz completed the restarts and the Heap usage pattern changed on all the nodes, so this is probably something to expect with the current settings. I'd like to add the -Xmx option to the conf100[123] JVMs anyway, and maybe think about testing another Garbage Collector? The druid100[123] hosts are running another cluster of ZK, they might be good candidates for the tests.
Change 337413 merged by Ottomata:
Add default JAVA_OPTS to the zookeeper server class
Change 337792 had a related patch set uploaded (by Elukey):
Update the zookeeper module
Change 337797 had a related patch set uploaded (by Elukey):
Set maximum JVM heap size for Zookeeper
Change 340719 had a related patch set uploaded (by Elukey):
[operations/puppet] Add experimental JVM Heap usage alarm to Zookeeper prod instances
Change 340719 merged by Elukey:
[operations/puppet] Add experimental JVM Heap usage alarm to Zookeeper prod instances
Change 340743 had a related patch set uploaded (by Elukey):
[operations/puppet] Fix the Zookeeper JVM Heap usage alarm
Change 340743 merged by Elukey:
[operations/puppet] Fix the Zookeeper JVM Heap usage alarm
re-opening since Icinga has many alerts:
https://icinga.wikimedia.org/cgi-bin/icinga/status.cgi?search_string=Zookeeper+Node+JVM
some CRIT , some UNKNOWN "Zookeeper node JVM Heap usage"
the common issue that graphite checks are either "No valid datapoints found" or "100% of data above the critical threshold" but then end up not being actioable and just ACKed permanently.
They have a comment saying "wrong metric in the alarm".
Mentioned in SAL (#wikimedia-operations) [2017-04-14T09:43:34Z] <elukey> temporarily set sysctl -w net.ipv4.ip_local_port_range="15000 64000" on mw1306 (jobrunner) as test - (rollback: sysctl -w net.ipv4.ip_local_port_range="32768 60999") - T157968
Change 348206 had a related patch set uploaded (by Elukey):
[operations/puppet@production] Fix Zookeeper's alarm for heap usage
Change 348206 merged by Elukey:
[operations/puppet@production] Fix Zookeeper's alarm for heap usage
Multiple PEBKACs from my side:
The main remaining issue now is that the alarms are in CRITICAL state even if theoretically they shouldn't be. Digging a bit more if I can find the last issue (another PEBKAC for sure).
Change 348214 had a related patch set uploaded (by Elukey):
[operations/puppet@production] Fix thresholds for Zookeeper Heap usage alarms
Change 348214 merged by Elukey:
[operations/puppet@production] Fix thresholds for Zookeeper Heap usage alarms
@Dzahn thanks a lot for the heads up, I should have fixed the issues. My ignorance about icinga lead me to false assumptions, I'll be more careful in the future.
I removed the acks from Icinga now, all green. Please re-open if anything is missing in your option!
@elukey thank you for fixing :) They all look green now. I'll comment if i see them again.