Change 429552 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] wdqs: enable UseNUMA on blazegraph and updater

https://gerrit.wikimedia.org/r/429552

Change 429552 merged by Gehel:
[operations/puppet@production] wdqs: enable UseNUMA on blazegraph and updater

https://gerrit.wikimedia.org/r/429552

Mentioned in SAL (#wikimedia-operations) [2018-04-30T17:26:39Z] <gehel> restart blazegraph and updater on wdqs1003 to activate UseNUMA -T193365

@Gehel, @EBernhardson do we have any ways to see what could be improved? It doesn't seem to break anything but I am not sure how to see if it improved anything.

Well, there were some hi level and low level changes visible on the elasticsearch servers:

High level

• per-node p95 latencies for the effected servers is now quite consistent, before it would typically have 50-100ms spikes.

Low level

• pcm-numa.x reported increased IPC under load, and an increased ratio of local dram to remote dram access at all times
• pcm-memory.x reported significantly better balance of memory bandwidth used between channels

I assume pcm-numa.x and pcm-memory.x require root? I've built them but they give me permission errors, and I don't have root on the servers, so maybe @Gehel could check if they produce something interesting? I placed them in my home dir on wdq3.

In T193365#4183855, @Smalyshev wrote:

I assume pcm-numa.x and pcm-memory.x require root? I've built them but they give me permission errors, and I don't have root on the servers, so maybe @Gehel could check if they produce something interesting? I placed them in my home dir on wdq3.

I can't get them to work either.

gehel@wdqs1003:~$ sudo ~smalyshev/pcm-numa.x 

 Processor Counter Monitor: NUMA monitoring utility 

Error: NMI watchdog is enabled. This consumes one hw-PMU counter
 to disable NMI watchdog please run under root: echo 0 > /proc/sys/kernel/nmi_watchdog
 or to disable it permanently: echo 'kernel.nmi_watchdog=0' >> /etc/sysctl.conf 
Access to Processor Counter Monitor has denied (no MSR or PCI CFG space access).

numastat gives some values, that at least seem balanced between the NUMA nodes. I don't have a baseline to compare it to, so no idea if it did improve or not. (Side note, I guess that -XX:+AlwaysPreTouch that is used on elasticsearch did make things worse in term of NUMA).

General stats:

gehel@wdqs1003:~$ numastat
                           node0           node1
numa_hit              3727077299      4176691608
numa_miss              630756819       129810267
numa_foreign           129810267       630756819
interleave_hit         133770282       130828100
local_node            3666203625      4092513487
other_node             691630493       213988388

blazegraph stats:

gehel@wdqs1003:~$ sudo numastat -p 23229

Per-node process memory usage (in MBs) for PID 23229 (java)
                           Node 0          Node 1           Total
                  --------------- --------------- ---------------
Huge                         0.00            0.00            0.00
Heap                         5.46            5.11           10.56
Stack                        0.00            0.04            0.04
Private                  18832.78        20152.16        38984.93
----------------  --------------- --------------- ---------------
Total                    18838.23        20157.30        38995.53

Since we don't see any degradation in performances since we enabled UseNUMA and that it seems to make sense to enable it, I propose to leave it enabled and close this task.