Page MenuHomePhabricator
Paste P13677

cloudcephosd1001 test results with_cache_disabled.2.txt
ActivePublic

Authored by dcaro on Jan 8 2021, 10:51 AM.
##### Linear read:
test: (g=0): rw=read, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=32
fio-3.12
Starting 1 process
test: (groupid=0, jobs=1): err= 0: pid=8037: Fri Jan 8 10:26:18 2021
read: IOPS=124, BW=499MiB/s (523MB/s)(29.3GiB/60131msec)
slat (usec): min=308, max=221379, avg=7998.11, stdev=6068.27
clat (msec): min=126, max=459, avg=248.54, stdev=33.72
lat (msec): min=128, max=466, avg=256.54, stdev=34.26
clat percentiles (msec):
| 1.00th=[ 241], 5.00th=[ 241], 10.00th=[ 241], 20.00th=[ 243],
| 30.00th=[ 243], 40.00th=[ 243], 50.00th=[ 243], 60.00th=[ 243],
| 70.00th=[ 243], 80.00th=[ 245], 90.00th=[ 249], 95.00th=[ 257],
| 99.00th=[ 456], 99.50th=[ 460], 99.90th=[ 460], 99.95th=[ 460],
| 99.99th=[ 460]
bw ( KiB/s): min=294912, max=532480, per=99.79%, avg=509520.32, stdev=50803.89, samples=120
iops : min= 72, max= 130, avg=124.38, stdev=12.43, samples=120
lat (msec) : 250=90.76%, 500=9.24%
cpu : usr=0.11%, sys=1.24%, ctx=14982, majf=0, minf=584
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.2%, 32=99.6%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwts: total=7496,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
Run status group 0 (all jobs):
READ: bw=499MiB/s (523MB/s), 499MiB/s-499MiB/s (523MB/s-523MB/s), io=29.3GiB (31.4GB), run=60131-60131msec
Disk stats (read/write):
sdc: ios=119443/0, merge=0/0, ticks=15106949/0, in_queue=1916788, util=99.90%
##### Linear write:
test: (g=0): rw=write, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=32
fio-3.12
Starting 1 process
test: (groupid=0, jobs=1): err= 0: pid=8158: Fri Jan 8 10:27:19 2021
write: IOPS=101, BW=407MiB/s (426MB/s)(23.9GiB/60158msec); 0 zone resets
slat (usec): min=73, max=217805, avg=9806.22, stdev=5963.62
clat (msec): min=152, max=509, avg=304.77, stdev=33.62
lat (msec): min=156, max=518, avg=314.58, stdev=34.18
clat percentiles (msec):
| 1.00th=[ 300], 5.00th=[ 300], 10.00th=[ 300], 20.00th=[ 300],
| 30.00th=[ 300], 40.00th=[ 300], 50.00th=[ 300], 60.00th=[ 300],
| 70.00th=[ 300], 80.00th=[ 300], 90.00th=[ 300], 95.00th=[ 300],
| 99.00th=[ 510], 99.50th=[ 510], 99.90th=[ 510], 99.95th=[ 510],
| 99.99th=[ 510]
bw ( KiB/s): min=270336, max=425984, per=99.74%, avg=415259.04, stdev=32796.65, samples=120
iops : min= 66, max= 104, avg=101.38, stdev= 8.01, samples=120
lat (msec) : 250=0.34%, 500=97.12%, 750=2.53%
cpu : usr=1.69%, sys=0.91%, ctx=12232, majf=0, minf=11
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.3%, 32=99.5%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwts: total=0,6115,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
Run status group 0 (all jobs):
WRITE: bw=407MiB/s (426MB/s), 407MiB/s-407MiB/s (426MB/s-426MB/s), io=23.9GiB (25.6GB), run=60158-60158msec
Disk stats (read/write):
sdc: ios=43/97749, merge=0/0, ticks=8/15149217, in_queue=1923596, util=99.86%
#### Peak parallel random read:
test: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=128
fio-3.12
Starting 1 process
test: (groupid=0, jobs=1): err= 0: pid=8254: Fri Jan 8 10:28:20 2021
read: IOPS=56.8k, BW=222MiB/s (233MB/s)(13.0GiB/60003msec)
slat (nsec): min=1371, max=89100, avg=2207.29, stdev=1448.44
clat (usec): min=108, max=244584, avg=2250.32, stdev=2815.87
lat (usec): min=110, max=244586, avg=2252.58, stdev=2815.87
clat percentiles (usec):
| 1.00th=[ 367], 5.00th=[ 562], 10.00th=[ 750], 20.00th=[ 1090],
| 30.00th=[ 1401], 40.00th=[ 1713], 50.00th=[ 2008], 60.00th=[ 2311],
| 70.00th=[ 2606], 80.00th=[ 2966], 90.00th=[ 3982], 95.00th=[ 4948],
| 99.00th=[ 6521], 99.50th=[ 8094], 99.90th=[ 13960], 99.95th=[ 16909],
| 99.99th=[227541]
bw ( KiB/s): min=114760, max=293640, per=99.99%, avg=227204.41, stdev=53058.65, samples=120
iops : min=28690, max=73410, avg=56801.08, stdev=13264.69, samples=120
lat (usec) : 250=0.16%, 500=3.31%, 750=6.62%, 1000=7.30%
lat (msec) : 2=32.18%, 4=40.63%, 10=9.52%, 20=0.26%, 50=0.01%
lat (msec) : 250=0.01%
cpu : usr=7.79%, sys=16.22%, ctx=3325524, majf=0, minf=138
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
issued rwts: total=3408563,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=128
Run status group 0 (all jobs):
READ: bw=222MiB/s (233MB/s), 222MiB/s-222MiB/s (233MB/s-233MB/s), io=13.0GiB (13.0GB), run=60003-60003msec
Disk stats (read/write):
sdc: ios=3403247/0, merge=1/0, ticks=7646076/0, in_queue=1665892, util=98.49%
#### Single-threaded read latency:
test: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.12
Starting 1 process
test: (groupid=0, jobs=1): err= 0: pid=8417: Fri Jan 8 10:29:20 2021
read: IOPS=9718, BW=37.0MiB/s (39.8MB/s)(2278MiB/60001msec)
slat (nsec): min=2055, max=64442, avg=3877.36, stdev=2956.37
clat (usec): min=28, max=1097, avg=98.28, stdev=24.82
lat (usec): min=49, max=1100, avg=102.24, stdev=25.23
clat percentiles (usec):
| 1.00th=[ 49], 5.00th=[ 50], 10.00th=[ 78], 20.00th=[ 79],
| 30.00th=[ 81], 40.00th=[ 92], 50.00th=[ 96], 60.00th=[ 97],
| 70.00th=[ 112], 80.00th=[ 128], 90.00th=[ 133], 95.00th=[ 135],
| 99.00th=[ 145], 99.50th=[ 155], 99.90th=[ 190], 99.95th=[ 196],
| 99.99th=[ 223]
bw ( KiB/s): min=35992, max=62512, per=100.00%, avg=38880.92, stdev=5592.55, samples=119
iops : min= 8998, max=15628, avg=9720.23, stdev=1398.14, samples=119
lat (usec) : 50=5.18%, 100=56.92%, 250=37.90%, 500=0.01%, 750=0.01%
lat (usec) : 1000=0.01%
lat (msec) : 2=0.01%
cpu : usr=2.43%, sys=5.49%, ctx=583103, majf=0, minf=10
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=583095,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=37.0MiB/s (39.8MB/s), 37.0MiB/s-37.0MiB/s (39.8MB/s-39.8MB/s), io=2278MiB (2388MB), run=60001-60001msec
Disk stats (read/write):
sdc: ios=581872/0, merge=0/0, ticks=57139/0, in_queue=56380, util=94.05%
#### Peak parallel random write:
test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=128
fio-3.12
Starting 1 process
test: (groupid=0, jobs=1): err= 0: pid=8507: Fri Jan 8 10:30:21 2021
write: IOPS=62.0k, BW=242MiB/s (254MB/s)(14.2GiB/60003msec); 0 zone resets
slat (nsec): min=1454, max=55749, avg=2116.35, stdev=943.07
clat (usec): min=443, max=24312, avg=2061.64, stdev=1086.86
lat (usec): min=448, max=24314, avg=2063.80, stdev=1086.85
clat percentiles (usec):
| 1.00th=[ 570], 5.00th=[ 685], 10.00th=[ 832], 20.00th=[ 1123],
| 30.00th=[ 1418], 40.00th=[ 1713], 50.00th=[ 1991], 60.00th=[ 2278],
| 70.00th=[ 2573], 80.00th=[ 2868], 90.00th=[ 3195], 95.00th=[ 3392],
| 99.00th=[ 5407], 99.50th=[ 7504], 99.90th=[11600], 99.95th=[13042],
| 99.99th=[16188]
bw ( KiB/s): min=245532, max=249512, per=99.99%, avg=247994.95, stdev=808.67, samples=120
iops : min=61383, max=62378, avg=61998.68, stdev=202.10, samples=120
lat (usec) : 500=0.01%, 750=7.31%, 1000=8.54%
lat (msec) : 2=34.24%, 4=48.39%, 10=1.32%, 20=0.20%, 50=0.01%
cpu : usr=8.49%, sys=17.55%, ctx=3702072, majf=0, minf=9
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
issued rwts: total=0,3720362,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=128
Run status group 0 (all jobs):
WRITE: bw=242MiB/s (254MB/s), 242MiB/s-242MiB/s (254MB/s-254MB/s), io=14.2GiB (15.2GB), run=60003-60003msec
Disk stats (read/write):
sdc: ios=99/3712488, merge=0/1, ticks=31/7646928, in_queue=1653896, util=95.70%
#### Journal write latency (with sync):
test: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.12
Starting 1 process
test: (groupid=0, jobs=1): err= 0: pid=8589: Fri Jan 8 10:31:21 2021
write: IOPS=27.1k, BW=106MiB/s (111MB/s)(6348MiB/60001msec); 0 zone resets
slat (nsec): min=2099, max=87217, avg=2762.14, stdev=1708.36
clat (nsec): min=595, max=741814, avg=33636.93, stdev=9675.35
lat (usec): min=31, max=744, avg=36.45, stdev= 9.86
clat percentiles (usec):
| 1.00th=[ 31], 5.00th=[ 31], 10.00th=[ 31], 20.00th=[ 31],
| 30.00th=[ 31], 40.00th=[ 31], 50.00th=[ 31], 60.00th=[ 31],
| 70.00th=[ 32], 80.00th=[ 33], 90.00th=[ 39], 95.00th=[ 49],
| 99.00th=[ 79], 99.50th=[ 96], 99.90th=[ 120], 99.95th=[ 129],
| 99.99th=[ 149]
bw ( KiB/s): min=106968, max=109320, per=100.00%, avg=108326.77, stdev=451.21, samples=119
iops : min=26742, max=27330, avg=27081.70, stdev=112.76, samples=119
lat (nsec) : 750=0.01%
lat (usec) : 4=0.01%, 10=0.01%, 20=0.12%, 50=95.18%, 100=4.33%
lat (usec) : 250=0.38%, 500=0.01%, 750=0.01%
cpu : usr=5.03%, sys=10.55%, ctx=1625002, majf=0, minf=11
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,1624986,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=106MiB/s (111MB/s), 106MiB/s-106MiB/s (111MB/s-111MB/s), io=6348MiB (6656MB), run=60001-60001msec
Disk stats (read/write):
sdc: ios=99/1621712, merge=0/0, ticks=35/54340, in_queue=52864, util=88.17%
#### Journal write latency (with fsync):
test: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.12
Starting 1 process
test: (groupid=0, jobs=1): err= 0: pid=8737: Fri Jan 8 10:32:22 2021
write: IOPS=26.4k, BW=103MiB/s (108MB/s)(6178MiB/60001msec); 0 zone resets
slat (nsec): min=2053, max=55855, avg=2729.75, stdev=1773.05
clat (nsec): min=602, max=935254, avg=33779.32, stdev=10795.93
lat (usec): min=31, max=937, avg=36.55, stdev=10.96
clat percentiles (usec):
| 1.00th=[ 31], 5.00th=[ 31], 10.00th=[ 31], 20.00th=[ 31],
| 30.00th=[ 31], 40.00th=[ 31], 50.00th=[ 31], 60.00th=[ 31],
| 70.00th=[ 32], 80.00th=[ 33], 90.00th=[ 39], 95.00th=[ 50],
| 99.00th=[ 82], 99.50th=[ 105], 99.90th=[ 133], 99.95th=[ 141],
| 99.99th=[ 235]
bw ( KiB/s): min=99728, max=107264, per=99.99%, avg=105420.75, stdev=1947.94, samples=119
iops : min=24932, max=26816, avg=26355.17, stdev=486.98, samples=119
lat (nsec) : 750=0.01%, 1000=0.01%
lat (usec) : 10=0.01%, 20=0.14%, 50=94.89%, 100=4.40%, 250=0.56%
lat (usec) : 500=0.01%, 750=0.01%, 1000=0.01%
fsync/fdatasync/sync_file_range:
sync (nsec): min=12, max=21216, avg=26.86, stdev=87.86
sync percentiles (nsec):
| 1.00th=[ 16], 5.00th=[ 17], 10.00th=[ 18], 20.00th=[ 19],
| 30.00th=[ 21], 40.00th=[ 23], 50.00th=[ 24], 60.00th=[ 25],
| 70.00th=[ 26], 80.00th=[ 27], 90.00th=[ 31], 95.00th=[ 47],
| 99.00th=[ 88], 99.50th=[ 95], 99.90th=[ 153], 99.95th=[ 163],
| 99.99th=[ 229]
cpu : usr=6.43%, sys=10.79%, ctx=1581479, majf=0, minf=16
IO depths : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,1581471,0,1581470 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=103MiB/s (108MB/s), 103MiB/s-103MiB/s (108MB/s-108MB/s), io=6178MiB (6478MB), run=60001-60001msec
Disk stats (read/write):
sdc: ios=43/1578310, merge=0/0, ticks=16/53279, in_queue=51344, util=85.63%
#### Single-threaded random write latency:
test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.12
Starting 1 process
test: (groupid=0, jobs=1): err= 0: pid=8813: Fri Jan 8 10:33:23 2021
write: IOPS=26.7k, BW=104MiB/s (109MB/s)(6251MiB/60001msec); 0 zone resets
slat (nsec): min=2119, max=63544, avg=2848.54, stdev=1787.22
clat (nsec): min=588, max=1119.1k, avg=34086.19, stdev=10256.43
lat (usec): min=31, max=1131, avg=36.97, stdev=10.46
clat percentiles (usec):
| 1.00th=[ 31], 5.00th=[ 31], 10.00th=[ 31], 20.00th=[ 31],
| 30.00th=[ 31], 40.00th=[ 31], 50.00th=[ 31], 60.00th=[ 32],
| 70.00th=[ 32], 80.00th=[ 34], 90.00th=[ 40], 95.00th=[ 51],
| 99.00th=[ 81], 99.50th=[ 99], 99.90th=[ 124], 99.95th=[ 133],
| 99.99th=[ 157]
bw ( KiB/s): min=99872, max=108192, per=99.99%, avg=106663.83, stdev=2039.97, samples=119
iops : min=24968, max=27048, avg=26665.97, stdev=510.00, samples=119
lat (nsec) : 750=0.01%, 1000=0.01%
lat (usec) : 10=0.01%, 20=0.13%, 50=94.78%, 100=4.63%, 250=0.46%
lat (usec) : 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%
cpu : usr=5.13%, sys=10.49%, ctx=1600168, majf=0, minf=11
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,1600164,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=104MiB/s (109MB/s), 104MiB/s-104MiB/s (109MB/s-109MB/s), io=6251MiB (6554MB), run=60001-60001msec
Disk stats (read/write):
sdc: ios=43/1596920, merge=0/0, ticks=7/54356, in_queue=52736, util=87.92%