Page MenuHomePhabricator
Paste P13685

cloudcephosd1001 test results with_cache_disabled.txt
ActivePublic

Authored by dcaro on Jan 8 2021, 10:52 AM.
##### Linear read:
test: (g=0): rw=read, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=32
fio-3.12
Starting 1 process
test: (groupid=0, jobs=1): err= 0: pid=4954: Fri Jan 8 10:15:01 2021
read: IOPS=124, BW=500MiB/s (524MB/s)(29.3GiB/60132msec)
slat (usec): min=299, max=216583, avg=7980.52, stdev=5377.29
clat (msec): min=124, max=468, avg=247.98, stdev=30.38
lat (msec): min=125, max=477, avg=255.96, stdev=30.88
clat percentiles (msec):
| 1.00th=[ 241], 5.00th=[ 243], 10.00th=[ 243], 20.00th=[ 243],
| 30.00th=[ 243], 40.00th=[ 243], 50.00th=[ 243], 60.00th=[ 243],
| 70.00th=[ 243], 80.00th=[ 245], 90.00th=[ 251], 95.00th=[ 257],
| 99.00th=[ 447], 99.50th=[ 451], 99.90th=[ 468], 99.95th=[ 468],
| 99.99th=[ 468]
bw ( KiB/s): min=294912, max=532480, per=99.79%, avg=510685.62, stdev=44607.45, samples=120
iops : min= 72, max= 130, avg=124.67, stdev=10.89, samples=120
lat (msec) : 250=88.35%, 500=11.65%
cpu : usr=0.11%, sys=1.15%, ctx=15018, majf=0, minf=585
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.2%, 32=99.6%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwts: total=7513,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
Run status group 0 (all jobs):
READ: bw=500MiB/s (524MB/s), 500MiB/s-500MiB/s (524MB/s-524MB/s), io=29.3GiB (31.5GB), run=60132-60132msec
Disk stats (read/write):
sdc: ios=120206/0, merge=0/0, ticks=15163099/0, in_queue=1924080, util=99.91%
##### Linear write:
test: (g=0): rw=write, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=32
fio-3.12
Starting 1 process
test: (groupid=0, jobs=1): err= 0: pid=5031: Fri Jan 8 10:16:02 2021
write: IOPS=101, BW=404MiB/s (424MB/s)(23.8GiB/60357msec); 0 zone resets
slat (usec): min=91, max=211458, avg=9832.72, stdev=6335.07
clat (msec): min=298, max=643, avg=306.30, stdev=36.22
lat (msec): min=308, max=652, avg=316.13, stdev=36.65
clat percentiles (msec):
| 1.00th=[ 300], 5.00th=[ 300], 10.00th=[ 300], 20.00th=[ 300],
| 30.00th=[ 300], 40.00th=[ 300], 50.00th=[ 300], 60.00th=[ 300],
| 70.00th=[ 300], 80.00th=[ 300], 90.00th=[ 300], 95.00th=[ 300],
| 99.00th=[ 502], 99.50th=[ 502], 99.90th=[ 584], 99.95th=[ 617],
| 99.99th=[ 642]
bw ( KiB/s): min=122880, max=425984, per=100.00%, avg=414084.32, stdev=43754.76, samples=120
iops : min= 30, max= 104, avg=101.08, stdev=10.68, samples=120
lat (msec) : 500=97.11%, 750=2.89%
cpu : usr=1.86%, sys=0.73%, ctx=12192, majf=0, minf=10
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.3%, 32=99.5%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwts: total=0,6098,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
Run status group 0 (all jobs):
WRITE: bw=404MiB/s (424MB/s), 404MiB/s-404MiB/s (424MB/s-424MB/s), io=23.8GiB (25.6GB), run=60357-60357msec
Disk stats (read/write):
sdc: ios=43/97561, merge=0/0, ticks=12/15196965, in_queue=1931368, util=99.87%
#### Peak parallel random read:
test: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=128
fio-3.12
Starting 1 process
test: (groupid=0, jobs=1): err= 0: pid=5325: Fri Jan 8 10:17:02 2021
read: IOPS=55.1k, BW=215MiB/s (226MB/s)(12.6GiB/60003msec)
slat (nsec): min=1395, max=73180, avg=2194.08, stdev=1389.08
clat (usec): min=104, max=234906, avg=2321.06, stdev=2442.96
lat (usec): min=106, max=234908, avg=2323.32, stdev=2442.96
clat percentiles (usec):
| 1.00th=[ 367], 5.00th=[ 570], 10.00th=[ 766], 20.00th=[ 1123],
| 30.00th=[ 1450], 40.00th=[ 1778], 50.00th=[ 2073], 60.00th=[ 2376],
| 70.00th=[ 2704], 80.00th=[ 3130], 90.00th=[ 4146], 95.00th=[ 5080],
| 99.00th=[ 6521], 99.50th=[ 8291], 99.90th=[14091], 99.95th=[16712],
| 99.99th=[24511]
bw ( KiB/s): min=124496, max=292200, per=100.00%, avg=220307.37, stdev=48683.11, samples=120
iops : min=31124, max=73050, avg=55076.87, stdev=12170.77, samples=120
lat (usec) : 250=0.17%, 500=3.16%, 750=6.23%, 1000=6.93%
lat (msec) : 2=30.90%, 4=41.46%, 10=10.83%, 20=0.29%, 50=0.01%
lat (msec) : 250=0.01%
cpu : usr=7.55%, sys=15.38%, ctx=3232576, majf=0, minf=137
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
issued rwts: total=3304891,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=128
Run status group 0 (all jobs):
READ: bw=215MiB/s (226MB/s), 215MiB/s-215MiB/s (226MB/s-226MB/s), io=12.6GiB (13.5GB), run=60003-60003msec
Disk stats (read/write):
sdc: ios=3298122/0, merge=0/0, ticks=7647186/0, in_queue=1802140, util=98.87%
#### Single-threaded read latency:
test: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.12
Starting 1 process
test: (groupid=0, jobs=1): err= 0: pid=5465: Fri Jan 8 10:18:03 2021
read: IOPS=9443, BW=36.9MiB/s (38.7MB/s)(2213MiB/60001msec)
slat (nsec): min=2050, max=41189, avg=3883.80, stdev=2939.43
clat (usec): min=37, max=221787, avg=101.24, stdev=295.26
lat (usec): min=51, max=221789, avg=105.20, stdev=295.28
clat percentiles (usec):
| 1.00th=[ 75], 5.00th=[ 78], 10.00th=[ 78], 20.00th=[ 79],
| 30.00th=[ 84], 40.00th=[ 95], 50.00th=[ 95], 60.00th=[ 100],
| 70.00th=[ 126], 80.00th=[ 127], 90.00th=[ 133], 95.00th=[ 133],
| 99.00th=[ 141], 99.50th=[ 145], 99.90th=[ 176], 99.95th=[ 192],
| 99.99th=[ 235]
bw ( KiB/s): min=22248, max=49208, per=100.00%, avg=37777.96, stdev=2700.14, samples=119
iops : min= 5562, max=12302, avg=9444.49, stdev=675.04, samples=119
lat (usec) : 50=0.01%, 100=60.27%, 250=39.72%, 500=0.01%, 750=0.01%
lat (usec) : 1000=0.01%
lat (msec) : 250=0.01%
cpu : usr=2.55%, sys=5.24%, ctx=566651, majf=0, minf=12
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=566646,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=36.9MiB/s (38.7MB/s), 36.9MiB/s-36.9MiB/s (38.7MB/s-38.7MB/s), io=2213MiB (2321MB), run=60001-60001msec
Disk stats (read/write):
sdc: ios=565411/0, merge=0/0, ticks=57134/0, in_queue=56212, util=93.79%
#### Peak parallel random write:
test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=128
fio-3.12
Starting 1 process
test: (groupid=0, jobs=1): err= 0: pid=5630: Fri Jan 8 10:19:03 2021
write: IOPS=62.1k, BW=243MiB/s (254MB/s)(14.2GiB/60003msec); 0 zone resets
slat (nsec): min=1452, max=134384, avg=2108.72, stdev=932.43
clat (usec): min=407, max=22944, avg=2058.34, stdev=1085.03
lat (usec): min=415, max=22946, avg=2060.48, stdev=1085.03
clat percentiles (usec):
| 1.00th=[ 570], 5.00th=[ 685], 10.00th=[ 832], 20.00th=[ 1123],
| 30.00th=[ 1418], 40.00th=[ 1696], 50.00th=[ 1991], 60.00th=[ 2278],
| 70.00th=[ 2573], 80.00th=[ 2868], 90.00th=[ 3195], 95.00th=[ 3392],
| 99.00th=[ 5407], 99.50th=[ 7504], 99.90th=[11600], 99.95th=[13173],
| 99.99th=[16188]
bw ( KiB/s): min=244632, max=249912, per=100.00%, avg=248403.63, stdev=928.31, samples=120
iops : min=61158, max=62478, avg=62100.92, stdev=232.11, samples=120
lat (usec) : 500=0.01%, 750=7.35%, 1000=8.55%
lat (msec) : 2=34.29%, 4=48.28%, 10=1.32%, 20=0.20%, 50=0.01%
cpu : usr=8.89%, sys=17.20%, ctx=3708327, majf=0, minf=10
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
issued rwts: total=0,3726370,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=128
Run status group 0 (all jobs):
WRITE: bw=243MiB/s (254MB/s), 243MiB/s-243MiB/s (254MB/s-254MB/s), io=14.2GiB (15.3GB), run=60003-60003msec
Disk stats (read/write):
sdc: ios=128/3718437, merge=0/1, ticks=50/7646905, in_queue=1793160, util=97.74%
#### Journal write latency (with sync):
test: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.12
Starting 1 process
test: (groupid=0, jobs=1): err= 0: pid=5715: Fri Jan 8 10:20:04 2021
write: IOPS=26.0k, BW=105MiB/s (111MB/s)(6328MiB/60001msec); 0 zone resets
slat (nsec): min=2095, max=49788, avg=2761.91, stdev=1707.49
clat (nsec): min=552, max=222865k, avg=33840.52, stdev=175371.71
lat (usec): min=31, max=222867, avg=36.65, stdev=175.38
clat percentiles (usec):
| 1.00th=[ 31], 5.00th=[ 31], 10.00th=[ 31], 20.00th=[ 31],
| 30.00th=[ 31], 40.00th=[ 31], 50.00th=[ 31], 60.00th=[ 31],
| 70.00th=[ 32], 80.00th=[ 33], 90.00th=[ 39], 95.00th=[ 49],
| 99.00th=[ 80], 99.50th=[ 99], 99.90th=[ 127], 99.95th=[ 137],
| 99.99th=[ 178]
bw ( KiB/s): min=66904, max=110680, per=100.00%, avg=108037.59, stdev=4074.32, samples=119
iops : min=16726, max=27670, avg=27009.39, stdev=1018.58, samples=119
lat (nsec) : 750=0.01%
lat (usec) : 2=0.01%, 10=0.01%, 20=0.11%, 50=95.12%, 100=4.30%
lat (usec) : 250=0.47%, 500=0.01%, 750=0.01%
lat (msec) : 250=0.01%
cpu : usr=4.44%, sys=10.73%, ctx=1619991, majf=0, minf=12
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,1619990,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=105MiB/s (111MB/s), 105MiB/s-105MiB/s (111MB/s-111MB/s), io=6328MiB (6635MB), run=60001-60001msec
Disk stats (read/write):
sdc: ios=128/1616904, merge=0/0, ticks=56/54592, in_queue=53120, util=88.53%
#### Journal write latency (with fsync):
test: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.12
Starting 1 process
test: (groupid=0, jobs=1): err= 0: pid=5840: Fri Jan 8 10:21:05 2021
write: IOPS=26.1k, BW=102MiB/s (107MB/s)(6129MiB/60001msec); 0 zone resets
slat (nsec): min=2029, max=61928, avg=2880.57, stdev=1738.58
clat (nsec): min=538, max=223671k, avg=33914.70, stdev=178905.16
lat (usec): min=31, max=223674, avg=36.84, stdev=178.92
clat percentiles (usec):
| 1.00th=[ 31], 5.00th=[ 31], 10.00th=[ 31], 20.00th=[ 31],
| 30.00th=[ 31], 40.00th=[ 31], 50.00th=[ 31], 60.00th=[ 31],
| 70.00th=[ 32], 80.00th=[ 33], 90.00th=[ 39], 95.00th=[ 50],
| 99.00th=[ 83], 99.50th=[ 109], 99.90th=[ 133], 99.95th=[ 141],
| 99.99th=[ 239]
bw ( KiB/s): min=54128, max=108376, per=99.99%, avg=104585.50, stdev=5328.88, samples=119
iops : min=13532, max=27094, avg=26146.38, stdev=1332.20, samples=119
lat (nsec) : 750=0.01%
lat (usec) : 10=0.01%, 20=0.12%, 50=94.90%, 100=4.40%, 250=0.58%
lat (usec) : 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 250=0.01%
fsync/fdatasync/sync_file_range:
sync (nsec): min=13, max=17569, avg=26.49, stdev=88.77
sync percentiles (nsec):
| 1.00th=[ 15], 5.00th=[ 16], 10.00th=[ 17], 20.00th=[ 18],
| 30.00th=[ 21], 40.00th=[ 22], 50.00th=[ 23], 60.00th=[ 24],
| 70.00th=[ 25], 80.00th=[ 27], 90.00th=[ 41], 95.00th=[ 47],
| 99.00th=[ 81], 99.50th=[ 94], 99.90th=[ 157], 99.95th=[ 161],
| 99.99th=[ 241]
cpu : usr=5.87%, sys=11.62%, ctx=1568962, majf=0, minf=15
IO depths : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,1568953,0,1568952 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=102MiB/s (107MB/s), 102MiB/s-102MiB/s (107MB/s-107MB/s), io=6129MiB (6426MB), run=60001-60001msec
Disk stats (read/write):
sdc: ios=166/1565817, merge=0/0, ticks=84/53152, in_queue=51704, util=86.12%
#### Single-threaded random write latency:
test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.12
Starting 1 process
test: (groupid=0, jobs=1): err= 0: pid=6835: Fri Jan 8 10:22:05 2021
write: IOPS=26.3k, BW=103MiB/s (108MB/s)(6158MiB/60001msec); 0 zone resets
slat (nsec): min=2034, max=68615, avg=2821.14, stdev=1664.94
clat (nsec): min=550, max=942234, avg=34655.93, stdev=12022.53
lat (usec): min=32, max=944, avg=37.52, stdev=12.18
clat percentiles (usec):
| 1.00th=[ 31], 5.00th=[ 31], 10.00th=[ 31], 20.00th=[ 31],
| 30.00th=[ 31], 40.00th=[ 31], 50.00th=[ 32], 60.00th=[ 32],
| 70.00th=[ 32], 80.00th=[ 34], 90.00th=[ 41], 95.00th=[ 56],
| 99.00th=[ 96], 99.50th=[ 114], 99.90th=[ 137], 99.95th=[ 147],
| 99.99th=[ 241]
bw ( KiB/s): min=100856, max=108592, per=100.00%, avg=105087.28, stdev=2208.93, samples=119
iops : min=25214, max=27148, avg=26271.78, stdev=552.24, samples=119
lat (nsec) : 750=0.01%
lat (usec) : 20=0.08%, 50=94.23%, 100=4.87%, 250=0.81%, 500=0.01%
lat (usec) : 750=0.01%, 1000=0.01%
cpu : usr=5.06%, sys=10.33%, ctx=1576330, majf=0, minf=11
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,1576327,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=103MiB/s (108MB/s), 103MiB/s-103MiB/s (108MB/s-108MB/s), io=6158MiB (6457MB), run=60001-60001msec
Disk stats (read/write):
sdc: ios=43/1573119, merge=0/0, ticks=9/54384, in_queue=52568, util=87.68%