Page Menu
Home
Phabricator
Search
Configure Global Search
Log In
Files
F17049904
No One
Actions
View File
Edit File
Delete File
View Transforms
Subscribe
Mute Notifications
Award Token
Flag For Later
Authored By
EBernhardson
Apr 17 2018, 11:11 PM
2018-04-17 23:11:21 (UTC+0)
Size
10 KB
Referenced Files
None
Subscribers
None
View Options
Test setup:
* Cluster idle except for indexing traffic
* Run query normalization routine from MjoLniR to generate load from 50 hadoop executors
* Routine limited to enwiki_content
* Choose two machines serving replicas of the same shard (elastic1017 and elastic1043) from the two separate groups of servers.
A few (relatively representative) memory usage reports from elastic1043:
```
|---------------------------------------||---------------------------------------|
|-- Socket 0 --||-- Socket 1 --|
|---------------------------------------||---------------------------------------|
|-- Memory Channel Monitoring --||-- Memory Channel Monitoring --|
|---------------------------------------||---------------------------------------|
|-- Mem Ch 0: Reads (MB/s): 1243.39 --||-- Mem Ch 0: Reads (MB/s): 249.55 --|
|-- Writes(MB/s): 1061.45 --||-- Writes(MB/s): 17.09 --|
|-- Mem Ch 1: Reads (MB/s): 1241.51 --||-- Mem Ch 1: Reads (MB/s): 241.42 --|
|-- Writes(MB/s): 1061.21 --||-- Writes(MB/s): 9.76 --|
|-- Mem Ch 2: Reads (MB/s): 1242.28 --||-- Mem Ch 2: Reads (MB/s): 241.36 --|
|-- Writes(MB/s): 1061.07 --||-- Writes(MB/s): 9.79 --|
|-- Mem Ch 3: Reads (MB/s): 1253.14 --||-- Mem Ch 3: Reads (MB/s): 241.45 --|
|-- Writes(MB/s): 1070.09 --||-- Writes(MB/s): 9.81 --|
|-- NODE 0 Mem Read (MB/s) : 4980.32 --||-- NODE 1 Mem Read (MB/s) : 973.77 --|
|-- NODE 0 Mem Write(MB/s) : 4253.82 --||-- NODE 1 Mem Write(MB/s) : 46.45 --|
|-- NODE 0 P. Write (T/s): 131039 --||-- NODE 1 P. Write (T/s): 114837 --|
|-- NODE 0 Memory (MB/s): 9234.14 --||-- NODE 1 Memory (MB/s): 1020.23 --|
|---------------------------------------||---------------------------------------|
|---------------------------------------||---------------------------------------|
|-- System Read Throughput(MB/s): 5954.09 --|
|-- System Write Throughput(MB/s): 4300.27 --|
|-- System Memory Throughput(MB/s): 10254.36 --|
|---------------------------------------||---------------------------------------|
|---------------------------------------||---------------------------------------|
|-- Socket 0 --||-- Socket 1 --|
|---------------------------------------||---------------------------------------|
|-- Memory Channel Monitoring --||-- Memory Channel Monitoring --|
|---------------------------------------||---------------------------------------|
|-- Mem Ch 0: Reads (MB/s): 1561.84 --||-- Mem Ch 0: Reads (MB/s): 282.63 --|
|-- Writes(MB/s): 1253.94 --||-- Writes(MB/s): 18.36 --|
|-- Mem Ch 1: Reads (MB/s): 1555.15 --||-- Mem Ch 1: Reads (MB/s): 273.58 --|
|-- Writes(MB/s): 1252.34 --||-- Writes(MB/s): 10.62 --|
|-- Mem Ch 2: Reads (MB/s): 1556.21 --||-- Mem Ch 2: Reads (MB/s): 273.93 --|
|-- Writes(MB/s): 1252.17 --||-- Writes(MB/s): 10.65 --|
|-- Mem Ch 3: Reads (MB/s): 1567.87 --||-- Mem Ch 3: Reads (MB/s): 274.37 --|
|-- Writes(MB/s): 1260.93 --||-- Writes(MB/s): 10.67 --|
|-- NODE 0 Mem Read (MB/s) : 6241.07 --||-- NODE 1 Mem Read (MB/s) : 1104.51 --|
|-- NODE 0 Mem Write(MB/s) : 5019.39 --||-- NODE 1 Mem Write(MB/s) : 50.30 --|
|-- NODE 0 P. Write (T/s): 135793 --||-- NODE 1 P. Write (T/s): 121339 --|
|-- NODE 0 Memory (MB/s): 11260.46 --||-- NODE 1 Memory (MB/s): 1154.81 --|
|---------------------------------------||---------------------------------------|
|---------------------------------------||---------------------------------------|
|-- System Read Throughput(MB/s): 7345.58 --|
|-- System Write Throughput(MB/s): 5069.69 --|
|-- System Memory Throughput(MB/s): 12415.27 --|
|---------------------------------------||---------------------------------------|
|---------------------------------------||---------------------------------------|
|-- Socket 0 --||-- Socket 1 --|
|---------------------------------------||---------------------------------------|
|-- Memory Channel Monitoring --||-- Memory Channel Monitoring --|
|---------------------------------------||---------------------------------------|
|-- Mem Ch 0: Reads (MB/s): 1451.99 --||-- Mem Ch 0: Reads (MB/s): 206.95 --|
|-- Writes(MB/s): 1257.25 --||-- Writes(MB/s): 16.61 --|
|-- Mem Ch 1: Reads (MB/s): 1450.56 --||-- Mem Ch 1: Reads (MB/s): 198.48 --|
|-- Writes(MB/s): 1257.02 --||-- Writes(MB/s): 9.42 --|
|-- Mem Ch 2: Reads (MB/s): 1451.96 --||-- Mem Ch 2: Reads (MB/s): 198.46 --|
|-- Writes(MB/s): 1256.86 --||-- Writes(MB/s): 9.41 --|
|-- Mem Ch 3: Reads (MB/s): 1462.16 --||-- Mem Ch 3: Reads (MB/s): 198.59 --|
|-- Writes(MB/s): 1265.56 --||-- Writes(MB/s): 9.45 --|
|-- NODE 0 Mem Read (MB/s) : 5816.67 --||-- NODE 1 Mem Read (MB/s) : 802.48 --|
|-- NODE 0 Mem Write(MB/s) : 5036.69 --||-- NODE 1 Mem Write(MB/s) : 44.89 --|
|-- NODE 0 P. Write (T/s): 133682 --||-- NODE 1 P. Write (T/s): 111961 --|
|-- NODE 0 Memory (MB/s): 10853.36 --||-- NODE 1 Memory (MB/s): 847.37 --|
|---------------------------------------||---------------------------------------|
```
Same thing for elastic1017:
```
|---------------------------------------||---------------------------------------|
|-- Socket 0 --||-- Socket 1 --|
|---------------------------------------||---------------------------------------|
|-- Memory Channel Monitoring --||-- Memory Channel Monitoring --|
|---------------------------------------||---------------------------------------|
|-- Mem Ch 1: Reads (MB/s): 3230.91 --||-- Mem Ch 1: Reads (MB/s): 387.15 --|
|-- Writes(MB/s): 3521.80 --||-- Writes(MB/s): 51.89 --|
|-- Mem Ch 2: Reads (MB/s): 713.36 --||-- Mem Ch 2: Reads (MB/s): 183.63 --|
|-- Writes(MB/s): 726.49 --||-- Writes(MB/s): 4.03 --|
|-- Mem Ch 3: Reads (MB/s): 712.75 --||-- Mem Ch 3: Reads (MB/s): 183.74 --|
|-- Writes(MB/s): 726.22 --||-- Writes(MB/s): 4.08 --|
|-- NODE 0 Mem Read (MB/s) : 4657.02 --||-- NODE 1 Mem Read (MB/s) : 754.51 --|
|-- NODE 0 Mem Write(MB/s) : 4974.51 --||-- NODE 1 Mem Write(MB/s) : 60.00 --|
|-- NODE 0 P. Write (T/s): 282753 --||-- NODE 1 P. Write (T/s): 130817 --|
|-- NODE 0 Memory (MB/s): 9631.53 --||-- NODE 1 Memory (MB/s): 814.51 --|
|---------------------------------------||---------------------------------------|
|---------------------------------------||---------------------------------------|
|-- System Read Throughput(MB/s): 5411.53 --|
|-- System Write Throughput(MB/s): 5034.51 --|
|-- System Memory Throughput(MB/s): 10446.04 --|
|---------------------------------------||---------------------------------------|
|---------------------------------------||---------------------------------------|
|-- Socket 0 --||-- Socket 1 --|
|---------------------------------------||---------------------------------------|
|-- Memory Channel Monitoring --||-- Memory Channel Monitoring --|
|---------------------------------------||---------------------------------------|
|-- Mem Ch 1: Reads (MB/s): 4239.13 --||-- Mem Ch 1: Reads (MB/s): 866.91 --|
|-- Writes(MB/s): 4404.82 --||-- Writes(MB/s): 467.04 --|
|-- Mem Ch 2: Reads (MB/s): 175.46 --||-- Mem Ch 2: Reads (MB/s): 458.88 --|
|-- Writes(MB/s): 12.40 --||-- Writes(MB/s): 250.83 --|
|-- Mem Ch 3: Reads (MB/s): 174.79 --||-- Mem Ch 3: Reads (MB/s): 458.59 --|
|-- Writes(MB/s): 12.09 --||-- Writes(MB/s): 250.71 --|
|-- NODE 0 Mem Read (MB/s) : 4589.39 --||-- NODE 1 Mem Read (MB/s) : 1784.39 --|
|-- NODE 0 Mem Write(MB/s) : 4429.31 --||-- NODE 1 Mem Write(MB/s) : 968.58 --|
|-- NODE 0 P. Write (T/s): 336267 --||-- NODE 1 P. Write (T/s): 145569 --|
|-- NODE 0 Memory (MB/s): 9018.70 --||-- NODE 1 Memory (MB/s): 2752.96 --|
|---------------------------------------||---------------------------------------|
|---------------------------------------||---------------------------------------|
|-- System Read Throughput(MB/s): 6373.78 --|
|-- System Write Throughput(MB/s): 5397.89 --|
|-- System Memory Throughput(MB/s): 11771.67 --|
|---------------------------------------||---------------------------------------|
|---------------------------------------||---------------------------------------|
|-- Socket 0 --||-- Socket 1 --|
|---------------------------------------||---------------------------------------|
|-- Memory Channel Monitoring --||-- Memory Channel Monitoring --|
|---------------------------------------||---------------------------------------|
|-- Mem Ch 1: Reads (MB/s): 1065.14 --||-- Mem Ch 1: Reads (MB/s): 2136.34 --|
|-- Writes(MB/s): 867.09 --||-- Writes(MB/s): 2027.39 --|
|-- Mem Ch 2: Reads (MB/s): 644.22 --||-- Mem Ch 2: Reads (MB/s): 667.31 --|
|-- Writes(MB/s): 621.00 --||-- Writes(MB/s): 529.47 --|
|-- Mem Ch 3: Reads (MB/s): 643.64 --||-- Mem Ch 3: Reads (MB/s): 667.26 --|
|-- Writes(MB/s): 620.69 --||-- Writes(MB/s): 529.44 --|
|-- NODE 0 Mem Read (MB/s) : 2353.00 --||-- NODE 1 Mem Read (MB/s) : 3470.91 --|
|-- NODE 0 Mem Write(MB/s) : 2108.78 --||-- NODE 1 Mem Write(MB/s) : 3086.30 --|
|-- NODE 0 P. Write (T/s): 171676 --||-- NODE 1 P. Write (T/s): 196425 --|
|-- NODE 0 Memory (MB/s): 4461.78 --||-- NODE 1 Memory (MB/s): 6557.21 --|
|---------------------------------------||---------------------------------------|
|---------------------------------------||---------------------------------------|
|-- System Read Throughput(MB/s): 5823.91 --|
|-- System Write Throughput(MB/s): 5195.08 --|
|-- System Memory Throughput(MB/s): 11018.99 --|
|---------------------------------------||---------------------------------------|
```
Open questions:
* Why is 1017 so heavily hitting Mem Ch 1, when 1043 is spreading out the reads between channels
* Why does socket 1 have such low memory bandwidth compared to socket 0 on both machines?
File Metadata
Details
Attached
Mime Type
text/plain; charset=utf-8
Storage Engine
blob
Storage Format
Raw Data
Storage Handle
5746113
Default Alt Text
(10 KB)
Attached To
Mode
P7005 (An Untitled Masterwork)
Attached
Detach File
Event Timeline
Log In to Comment