Page MenuHomePhabricator
Paste P7005

(An Untitled Masterwork)
ActivePublic

Authored by EBernhardson on Apr 17 2018, 11:11 PM.
Test setup:
* Cluster idle except for indexing traffic
* Run query normalization routine from MjoLniR to generate load from 50 hadoop executors
* Routine limited to enwiki_content
* Choose two machines serving replicas of the same shard (elastic1017 and elastic1043) from the two separate groups of servers.
A few (relatively representative) memory usage reports from elastic1043:
```
|---------------------------------------||---------------------------------------|
|-- Socket 0 --||-- Socket 1 --|
|---------------------------------------||---------------------------------------|
|-- Memory Channel Monitoring --||-- Memory Channel Monitoring --|
|---------------------------------------||---------------------------------------|
|-- Mem Ch 0: Reads (MB/s): 1243.39 --||-- Mem Ch 0: Reads (MB/s): 249.55 --|
|-- Writes(MB/s): 1061.45 --||-- Writes(MB/s): 17.09 --|
|-- Mem Ch 1: Reads (MB/s): 1241.51 --||-- Mem Ch 1: Reads (MB/s): 241.42 --|
|-- Writes(MB/s): 1061.21 --||-- Writes(MB/s): 9.76 --|
|-- Mem Ch 2: Reads (MB/s): 1242.28 --||-- Mem Ch 2: Reads (MB/s): 241.36 --|
|-- Writes(MB/s): 1061.07 --||-- Writes(MB/s): 9.79 --|
|-- Mem Ch 3: Reads (MB/s): 1253.14 --||-- Mem Ch 3: Reads (MB/s): 241.45 --|
|-- Writes(MB/s): 1070.09 --||-- Writes(MB/s): 9.81 --|
|-- NODE 0 Mem Read (MB/s) : 4980.32 --||-- NODE 1 Mem Read (MB/s) : 973.77 --|
|-- NODE 0 Mem Write(MB/s) : 4253.82 --||-- NODE 1 Mem Write(MB/s) : 46.45 --|
|-- NODE 0 P. Write (T/s): 131039 --||-- NODE 1 P. Write (T/s): 114837 --|
|-- NODE 0 Memory (MB/s): 9234.14 --||-- NODE 1 Memory (MB/s): 1020.23 --|
|---------------------------------------||---------------------------------------|
|---------------------------------------||---------------------------------------|
|-- System Read Throughput(MB/s): 5954.09 --|
|-- System Write Throughput(MB/s): 4300.27 --|
|-- System Memory Throughput(MB/s): 10254.36 --|
|---------------------------------------||---------------------------------------|
|---------------------------------------||---------------------------------------|
|-- Socket 0 --||-- Socket 1 --|
|---------------------------------------||---------------------------------------|
|-- Memory Channel Monitoring --||-- Memory Channel Monitoring --|
|---------------------------------------||---------------------------------------|
|-- Mem Ch 0: Reads (MB/s): 1561.84 --||-- Mem Ch 0: Reads (MB/s): 282.63 --|
|-- Writes(MB/s): 1253.94 --||-- Writes(MB/s): 18.36 --|
|-- Mem Ch 1: Reads (MB/s): 1555.15 --||-- Mem Ch 1: Reads (MB/s): 273.58 --|
|-- Writes(MB/s): 1252.34 --||-- Writes(MB/s): 10.62 --|
|-- Mem Ch 2: Reads (MB/s): 1556.21 --||-- Mem Ch 2: Reads (MB/s): 273.93 --|
|-- Writes(MB/s): 1252.17 --||-- Writes(MB/s): 10.65 --|
|-- Mem Ch 3: Reads (MB/s): 1567.87 --||-- Mem Ch 3: Reads (MB/s): 274.37 --|
|-- Writes(MB/s): 1260.93 --||-- Writes(MB/s): 10.67 --|
|-- NODE 0 Mem Read (MB/s) : 6241.07 --||-- NODE 1 Mem Read (MB/s) : 1104.51 --|
|-- NODE 0 Mem Write(MB/s) : 5019.39 --||-- NODE 1 Mem Write(MB/s) : 50.30 --|
|-- NODE 0 P. Write (T/s): 135793 --||-- NODE 1 P. Write (T/s): 121339 --|
|-- NODE 0 Memory (MB/s): 11260.46 --||-- NODE 1 Memory (MB/s): 1154.81 --|
|---------------------------------------||---------------------------------------|
|---------------------------------------||---------------------------------------|
|-- System Read Throughput(MB/s): 7345.58 --|
|-- System Write Throughput(MB/s): 5069.69 --|
|-- System Memory Throughput(MB/s): 12415.27 --|
|---------------------------------------||---------------------------------------|
|---------------------------------------||---------------------------------------|
|-- Socket 0 --||-- Socket 1 --|
|---------------------------------------||---------------------------------------|
|-- Memory Channel Monitoring --||-- Memory Channel Monitoring --|
|---------------------------------------||---------------------------------------|
|-- Mem Ch 0: Reads (MB/s): 1451.99 --||-- Mem Ch 0: Reads (MB/s): 206.95 --|
|-- Writes(MB/s): 1257.25 --||-- Writes(MB/s): 16.61 --|
|-- Mem Ch 1: Reads (MB/s): 1450.56 --||-- Mem Ch 1: Reads (MB/s): 198.48 --|
|-- Writes(MB/s): 1257.02 --||-- Writes(MB/s): 9.42 --|
|-- Mem Ch 2: Reads (MB/s): 1451.96 --||-- Mem Ch 2: Reads (MB/s): 198.46 --|
|-- Writes(MB/s): 1256.86 --||-- Writes(MB/s): 9.41 --|
|-- Mem Ch 3: Reads (MB/s): 1462.16 --||-- Mem Ch 3: Reads (MB/s): 198.59 --|
|-- Writes(MB/s): 1265.56 --||-- Writes(MB/s): 9.45 --|
|-- NODE 0 Mem Read (MB/s) : 5816.67 --||-- NODE 1 Mem Read (MB/s) : 802.48 --|
|-- NODE 0 Mem Write(MB/s) : 5036.69 --||-- NODE 1 Mem Write(MB/s) : 44.89 --|
|-- NODE 0 P. Write (T/s): 133682 --||-- NODE 1 P. Write (T/s): 111961 --|
|-- NODE 0 Memory (MB/s): 10853.36 --||-- NODE 1 Memory (MB/s): 847.37 --|
|---------------------------------------||---------------------------------------|
```
Same thing for elastic1017:
```
|---------------------------------------||---------------------------------------|
|-- Socket 0 --||-- Socket 1 --|
|---------------------------------------||---------------------------------------|
|-- Memory Channel Monitoring --||-- Memory Channel Monitoring --|
|---------------------------------------||---------------------------------------|
|-- Mem Ch 1: Reads (MB/s): 3230.91 --||-- Mem Ch 1: Reads (MB/s): 387.15 --|
|-- Writes(MB/s): 3521.80 --||-- Writes(MB/s): 51.89 --|
|-- Mem Ch 2: Reads (MB/s): 713.36 --||-- Mem Ch 2: Reads (MB/s): 183.63 --|
|-- Writes(MB/s): 726.49 --||-- Writes(MB/s): 4.03 --|
|-- Mem Ch 3: Reads (MB/s): 712.75 --||-- Mem Ch 3: Reads (MB/s): 183.74 --|
|-- Writes(MB/s): 726.22 --||-- Writes(MB/s): 4.08 --|
|-- NODE 0 Mem Read (MB/s) : 4657.02 --||-- NODE 1 Mem Read (MB/s) : 754.51 --|
|-- NODE 0 Mem Write(MB/s) : 4974.51 --||-- NODE 1 Mem Write(MB/s) : 60.00 --|
|-- NODE 0 P. Write (T/s): 282753 --||-- NODE 1 P. Write (T/s): 130817 --|
|-- NODE 0 Memory (MB/s): 9631.53 --||-- NODE 1 Memory (MB/s): 814.51 --|
|---------------------------------------||---------------------------------------|
|---------------------------------------||---------------------------------------|
|-- System Read Throughput(MB/s): 5411.53 --|
|-- System Write Throughput(MB/s): 5034.51 --|
|-- System Memory Throughput(MB/s): 10446.04 --|
|---------------------------------------||---------------------------------------|
|---------------------------------------||---------------------------------------|
|-- Socket 0 --||-- Socket 1 --|
|---------------------------------------||---------------------------------------|
|-- Memory Channel Monitoring --||-- Memory Channel Monitoring --|
|---------------------------------------||---------------------------------------|
|-- Mem Ch 1: Reads (MB/s): 4239.13 --||-- Mem Ch 1: Reads (MB/s): 866.91 --|
|-- Writes(MB/s): 4404.82 --||-- Writes(MB/s): 467.04 --|
|-- Mem Ch 2: Reads (MB/s): 175.46 --||-- Mem Ch 2: Reads (MB/s): 458.88 --|
|-- Writes(MB/s): 12.40 --||-- Writes(MB/s): 250.83 --|
|-- Mem Ch 3: Reads (MB/s): 174.79 --||-- Mem Ch 3: Reads (MB/s): 458.59 --|
|-- Writes(MB/s): 12.09 --||-- Writes(MB/s): 250.71 --|
|-- NODE 0 Mem Read (MB/s) : 4589.39 --||-- NODE 1 Mem Read (MB/s) : 1784.39 --|
|-- NODE 0 Mem Write(MB/s) : 4429.31 --||-- NODE 1 Mem Write(MB/s) : 968.58 --|
|-- NODE 0 P. Write (T/s): 336267 --||-- NODE 1 P. Write (T/s): 145569 --|
|-- NODE 0 Memory (MB/s): 9018.70 --||-- NODE 1 Memory (MB/s): 2752.96 --|
|---------------------------------------||---------------------------------------|
|---------------------------------------||---------------------------------------|
|-- System Read Throughput(MB/s): 6373.78 --|
|-- System Write Throughput(MB/s): 5397.89 --|
|-- System Memory Throughput(MB/s): 11771.67 --|
|---------------------------------------||---------------------------------------|
|---------------------------------------||---------------------------------------|
|-- Socket 0 --||-- Socket 1 --|
|---------------------------------------||---------------------------------------|
|-- Memory Channel Monitoring --||-- Memory Channel Monitoring --|
|---------------------------------------||---------------------------------------|
|-- Mem Ch 1: Reads (MB/s): 1065.14 --||-- Mem Ch 1: Reads (MB/s): 2136.34 --|
|-- Writes(MB/s): 867.09 --||-- Writes(MB/s): 2027.39 --|
|-- Mem Ch 2: Reads (MB/s): 644.22 --||-- Mem Ch 2: Reads (MB/s): 667.31 --|
|-- Writes(MB/s): 621.00 --||-- Writes(MB/s): 529.47 --|
|-- Mem Ch 3: Reads (MB/s): 643.64 --||-- Mem Ch 3: Reads (MB/s): 667.26 --|
|-- Writes(MB/s): 620.69 --||-- Writes(MB/s): 529.44 --|
|-- NODE 0 Mem Read (MB/s) : 2353.00 --||-- NODE 1 Mem Read (MB/s) : 3470.91 --|
|-- NODE 0 Mem Write(MB/s) : 2108.78 --||-- NODE 1 Mem Write(MB/s) : 3086.30 --|
|-- NODE 0 P. Write (T/s): 171676 --||-- NODE 1 P. Write (T/s): 196425 --|
|-- NODE 0 Memory (MB/s): 4461.78 --||-- NODE 1 Memory (MB/s): 6557.21 --|
|---------------------------------------||---------------------------------------|
|---------------------------------------||---------------------------------------|
|-- System Read Throughput(MB/s): 5823.91 --|
|-- System Write Throughput(MB/s): 5195.08 --|
|-- System Memory Throughput(MB/s): 11018.99 --|
|---------------------------------------||---------------------------------------|
```
Open questions:
* Why is 1017 so heavily hitting Mem Ch 1, when 1043 is spreading out the reads between channels
* Why does socket 1 have such low memory bandwidth compared to socket 0 on both machines?