This incident coincide in timing with a deployment of wmf.23 (according to @Dzahn, but the change I found seems to have been deployed after the incident).

Some more notes (@EBernhardson has already written most if not all of the facts, let's start the speculation):

• Heap graphs look like the GC is not able to free memory at the same rate that it is allocated. The 1 minute sampling is a bit too long to have clear visibility, but it still seems clear that we have a GC issue. Looking at the logs, the GC did manage to free memory, so the issue is allocation rate vs GC rate, not overall heap size (actually reducing heap size can help in some cases).
• No Icinga alert was raised, probably because the graphite metric on which it is based has been renamed.

Things we should do (I'll create sub tasks):

• check / fix icinga alert on CirrusSearch response time.
• enable GC logs for elasticsearch JVMs

GC issue seems to perfectly coincide with a merge on elastic1026. We could also start to monitor all segment related stats, most of them are bytes stored in RAM :

curl -s localhost:9200/_nodes/elastic1026/stats |jq '.nodes["_Z3TqKpDS-qBDyGky2r0Kg"].indices.segments':

        "segments": {
          "fixed_bit_set_memory_in_bytes": 4513824,
          "version_map_memory_in_bytes": 105604,
          "index_writer_max_memory_in_bytes": 3182454991,
          "index_writer_memory_in_bytes": 52100548,
          "memory_in_bytes": 2608021926,
          "count": 2268
        },

But maybe also all stats that report size stored in RAM: completion.size_in_bytes, field_data, filter_cache.

Another small fact : when GC issue started most of the queries reported as WARN (> 1s) in /var/log/elasticsearch/production-search-eqiad_index_search_slowlog.log.1 are for dewiki_content.
I copied this log file in ~dcausse/T134829/production-search-eqiad_index_search_slowlog.log.1

Are we going to do anything else with this ticket? Should move it to done?

The usage in Discovery is to move tasks to Done on board and let our product owner have a final review and closing them.