Page MenuHomePhabricator

Evaluate using NUMA for Blazegraph
Closed, ResolvedPublic

Description

Given good result achieved by using NUMA in T191236: Resolve elasticsearch latency alerts, we want to evaluate whether setting UseNUMA option for Blazegraph is going to help.

Details

Related Gerrit Patches:

Event Timeline

Restricted Application added projects: Wikidata, Discovery. · View Herald TranscriptApr 30 2018, 1:40 AM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Change 429552 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] wdqs: enable UseNUMA on blazegraph and updater

https://gerrit.wikimedia.org/r/429552

Change 429552 merged by Gehel:
[operations/puppet@production] wdqs: enable UseNUMA on blazegraph and updater

https://gerrit.wikimedia.org/r/429552

Mentioned in SAL (#wikimedia-operations) [2018-04-30T17:26:39Z] <gehel> restart blazegraph and updater on wdqs1003 to activate UseNUMA -T193365

@Gehel, @EBernhardson do we have any ways to see what could be improved? It doesn't seem to break anything but I am not sure how to see if it improved anything.

Smalyshev triaged this task as Medium priority.May 3 2018, 3:04 AM

Well, there were some hi level and low level changes visible on the elasticsearch servers:

High level

  • per-node p95 latencies for the effected servers is now quite consistent, before it would typically have 50-100ms spikes.

Low level

  • pcm-numa.x reported increased IPC under load, and an increased ratio of local dram to remote dram access at all times
  • pcm-memory.x reported significantly better balance of memory bandwidth used between channels

I assume pcm-numa.x and pcm-memory.x require root? I've built them but they give me permission errors, and I don't have root on the servers, so maybe @Gehel could check if they produce something interesting? I placed them in my home dir on wdq3.

Gehel added a comment.May 18 2018, 1:21 PM

I assume pcm-numa.x and pcm-memory.x require root? I've built them but they give me permission errors, and I don't have root on the servers, so maybe @Gehel could check if they produce something interesting? I placed them in my home dir on wdq3.

I can't get them to work either.

gehel@wdqs1003:~$ sudo ~smalyshev/pcm-numa.x 

 Processor Counter Monitor: NUMA monitoring utility 

Error: NMI watchdog is enabled. This consumes one hw-PMU counter
 to disable NMI watchdog please run under root: echo 0 > /proc/sys/kernel/nmi_watchdog
 or to disable it permanently: echo 'kernel.nmi_watchdog=0' >> /etc/sysctl.conf 
Access to Processor Counter Monitor has denied (no MSR or PCI CFG space access).

numastat gives some values, that at least seem balanced between the NUMA nodes. I don't have a baseline to compare it to, so no idea if it did improve or not. (Side note, I guess that -XX:+AlwaysPreTouch that is used on elasticsearch did make things worse in term of NUMA).

General stats:

gehel@wdqs1003:~$ numastat
                           node0           node1
numa_hit              3727077299      4176691608
numa_miss              630756819       129810267
numa_foreign           129810267       630756819
interleave_hit         133770282       130828100
local_node            3666203625      4092513487
other_node             691630493       213988388

blazegraph stats:

gehel@wdqs1003:~$ sudo numastat -p 23229

Per-node process memory usage (in MBs) for PID 23229 (java)
                           Node 0          Node 1           Total
                  --------------- --------------- ---------------
Huge                         0.00            0.00            0.00
Heap                         5.46            5.11           10.56
Stack                        0.00            0.04            0.04
Private                  18832.78        20152.16        38984.93
----------------  --------------- --------------- ---------------
Total                    18838.23        20157.30        38995.53

Since we don't see any degradation in performances since we enabled UseNUMA and that it seems to make sense to enable it, I propose to leave it enabled and close this task.

Smalyshev closed this task as Resolved.May 31 2018, 12:27 AM
Smalyshev claimed this task.

Looks like nothing more to do here.

Vvjjkkii renamed this task from Evaluate using NUMA for Blazegraph to 2zdaaaaaaa.Jul 1 2018, 1:13 AM
Vvjjkkii reopened this task as Open.
Vvjjkkii removed Smalyshev as the assignee of this task.
Vvjjkkii raised the priority of this task from Medium to High.
Vvjjkkii updated the task description. (Show Details)
Vvjjkkii removed subscribers: gerritbot, Aklapper.
CommunityTechBot renamed this task from 2zdaaaaaaa to Evaluate using NUMA for Blazegraph.Jul 2 2018, 2:59 PM
CommunityTechBot closed this task as Resolved.
CommunityTechBot assigned this task to Smalyshev.
CommunityTechBot lowered the priority of this task from High to Medium.
CommunityTechBot updated the task description. (Show Details)
CommunityTechBot added subscribers: gerritbot, Aklapper.