Page MenuHomePhabricator
Paste P13681

cloudcephosd1001 test results with_cache_enabled.2.txt
ActivePublic

Authored by dcaro on Jan 8 2021, 10:51 AM.
##### Linear read:
test: (g=0): rw=read, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=32
fio-3.12
Starting 1 process
test: (groupid=0, jobs=1): err= 0: pid=3857: Fri Jan 8 10:05:10 2021
read: IOPS=125, BW=501MiB/s (525MB/s)(29.4GiB/60127msec)
slat (usec): min=303, max=204274, avg=7958.31, stdev=5094.90
clat (msec): min=126, max=445, avg=247.30, stdev=28.61
lat (msec): min=128, max=453, avg=255.26, stdev=29.08
clat percentiles (msec):
| 1.00th=[ 241], 5.00th=[ 241], 10.00th=[ 243], 20.00th=[ 243],
| 30.00th=[ 243], 40.00th=[ 243], 50.00th=[ 243], 60.00th=[ 243],
| 70.00th=[ 243], 80.00th=[ 245], 90.00th=[ 249], 95.00th=[ 253],
| 99.00th=[ 439], 99.50th=[ 439], 99.90th=[ 447], 99.95th=[ 447],
| 99.99th=[ 447]
bw ( KiB/s): min=385024, max=532480, per=99.79%, avg=512066.88, stdev=32153.84, samples=120
iops : min= 94, max= 130, avg=124.97, stdev= 7.84, samples=120
lat (msec) : 250=91.60%, 500=8.40%
cpu : usr=0.13%, sys=1.10%, ctx=15060, majf=0, minf=583
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.2%, 32=99.6%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwts: total=7533,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
Run status group 0 (all jobs):
READ: bw=501MiB/s (525MB/s), 501MiB/s-501MiB/s (525MB/s-525MB/s), io=29.4GiB (31.6GB), run=60127-60127msec
Disk stats (read/write):
sdc: ios=120038/0, merge=0/0, ticks=15094172/0, in_queue=1916476, util=99.90%
##### Linear write:
test: (g=0): rw=write, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=32
fio-3.12
Starting 1 process
test: (groupid=0, jobs=1): err= 0: pid=4053: Fri Jan 8 10:06:11 2021
write: IOPS=101, BW=406MiB/s (426MB/s)(23.8GiB/60156msec); 0 zone resets
slat (usec): min=73, max=202302, avg=9823.69, stdev=6052.43
clat (msec): min=152, max=492, avg=305.31, stdev=33.95
lat (msec): min=157, max=502, avg=315.13, stdev=34.51
clat percentiles (msec):
| 1.00th=[ 300], 5.00th=[ 300], 10.00th=[ 300], 20.00th=[ 300],
| 30.00th=[ 300], 40.00th=[ 300], 50.00th=[ 300], 60.00th=[ 300],
| 70.00th=[ 300], 80.00th=[ 300], 90.00th=[ 300], 95.00th=[ 300],
| 99.00th=[ 493], 99.50th=[ 493], 99.90th=[ 493], 99.95th=[ 493],
| 99.99th=[ 493]
bw ( KiB/s): min=262144, max=425984, per=99.73%, avg=414479.88, stdev=37182.78, samples=120
iops : min= 64, max= 104, avg=101.16, stdev= 9.07, samples=120
lat (msec) : 250=0.34%, 500=99.66%
cpu : usr=1.66%, sys=0.90%, ctx=12203, majf=0, minf=10
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.3%, 32=99.5%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwts: total=0,6104,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
Run status group 0 (all jobs):
WRITE: bw=406MiB/s (426MB/s), 406MiB/s-406MiB/s (426MB/s-426MB/s), io=23.8GiB (25.6GB), run=60156-60156msec
Disk stats (read/write):
sdc: ios=43/97574, merge=0/0, ticks=11/15149663, in_queue=1923584, util=99.87%
#### Peak parallel random read:
test: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=128
fio-3.12
Starting 1 process
test: (groupid=0, jobs=1): err= 0: pid=4206: Fri Jan 8 10:07:11 2021
read: IOPS=54.1k, BW=211MiB/s (221MB/s)(12.4GiB/60002msec)
slat (nsec): min=1364, max=82392, avg=2168.27, stdev=1426.35
clat (usec): min=97, max=228303, avg=2364.89, stdev=2768.74
lat (usec): min=99, max=228309, avg=2367.10, stdev=2768.73
clat percentiles (usec):
| 1.00th=[ 363], 5.00th=[ 578], 10.00th=[ 775], 20.00th=[ 1139],
| 30.00th=[ 1467], 40.00th=[ 1795], 50.00th=[ 2114], 60.00th=[ 2409],
| 70.00th=[ 2737], 80.00th=[ 3195], 90.00th=[ 4228], 95.00th=[ 5145],
| 99.00th=[ 6587], 99.50th=[ 8455], 99.90th=[ 14615], 99.95th=[ 17433],
| 99.99th=[212861]
bw ( KiB/s): min=118528, max=292336, per=99.99%, avg=216230.15, stdev=48763.86, samples=120
iops : min=29632, max=73084, avg=54057.59, stdev=12191.03, samples=120
lat (usec) : 100=0.01%, 250=0.19%, 500=3.11%, 750=6.09%, 1000=6.77%
lat (msec) : 2=30.46%, 4=41.48%, 10=11.57%, 20=0.30%, 50=0.02%
lat (msec) : 250=0.01%
cpu : usr=7.08%, sys=15.30%, ctx=3168002, majf=0, minf=138
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
issued rwts: total=3243746,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=128
Run status group 0 (all jobs):
READ: bw=211MiB/s (221MB/s), 211MiB/s-211MiB/s (221MB/s-221MB/s), io=12.4GiB (13.3GB), run=60002-60002msec
Disk stats (read/write):
sdc: ios=3236441/0, merge=1/0, ticks=7647585/0, in_queue=1672128, util=98.79%
#### Single-threaded read latency:
test: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.12
Starting 1 process
test: (groupid=0, jobs=1): err= 0: pid=4287: Fri Jan 8 10:08:12 2021
read: IOPS=9435, BW=36.9MiB/s (38.6MB/s)(2211MiB/60001msec)
slat (usec): min=2, max=263, avg= 4.03, stdev= 3.10
clat (nsec): min=1060, max=212522k, avg=101135.78, stdev=283141.56
lat (usec): min=51, max=212525, avg=105.26, stdev=283.17
clat percentiles (usec):
| 1.00th=[ 74], 5.00th=[ 78], 10.00th=[ 78], 20.00th=[ 79],
| 30.00th=[ 84], 40.00th=[ 94], 50.00th=[ 95], 60.00th=[ 100],
| 70.00th=[ 126], 80.00th=[ 127], 90.00th=[ 131], 95.00th=[ 133],
| 99.00th=[ 141], 99.50th=[ 145], 99.90th=[ 186], 99.95th=[ 200],
| 99.99th=[ 318]
bw ( KiB/s): min=21120, max=49536, per=100.00%, avg=37739.26, stdev=2690.94, samples=120
iops : min= 5280, max=12384, avg=9434.79, stdev=672.73, samples=120
lat (usec) : 2=0.01%, 50=0.01%, 100=60.83%, 250=39.14%, 500=0.02%
lat (usec) : 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 250=0.01%
cpu : usr=2.62%, sys=5.34%, ctx=566146, majf=0, minf=13
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=566139,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=36.9MiB/s (38.6MB/s), 36.9MiB/s-36.9MiB/s (38.6MB/s-38.6MB/s), io=2211MiB (2319MB), run=60001-60001msec
Disk stats (read/write):
sdc: ios=564938/0, merge=0/0, ticks=57110/0, in_queue=56220, util=93.77%
#### Peak parallel random write:
test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=128
fio-3.12
Starting 1 process
test: (groupid=0, jobs=1): err= 0: pid=4411: Fri Jan 8 10:09:13 2021
write: IOPS=61.6k, BW=241MiB/s (252MB/s)(14.1GiB/60003msec); 0 zone resets
slat (nsec): min=1472, max=70965, avg=2144.58, stdev=958.17
clat (usec): min=405, max=24440, avg=2073.72, stdev=1091.09
lat (usec): min=428, max=24441, avg=2075.90, stdev=1091.09
clat percentiles (usec):
| 1.00th=[ 570], 5.00th=[ 685], 10.00th=[ 832], 20.00th=[ 1123],
| 30.00th=[ 1418], 40.00th=[ 1713], 50.00th=[ 2008], 60.00th=[ 2311],
| 70.00th=[ 2606], 80.00th=[ 2900], 90.00th=[ 3228], 95.00th=[ 3425],
| 99.00th=[ 5407], 99.50th=[ 7504], 99.90th=[11600], 99.95th=[13042],
| 99.99th=[16188]
bw ( KiB/s): min=240552, max=249576, per=99.99%, avg=246554.45, stdev=1674.86, samples=120
iops : min=60138, max=62394, avg=61638.62, stdev=418.72, samples=120
lat (usec) : 500=0.01%, 750=7.18%, 1000=8.48%
lat (msec) : 2=34.04%, 4=48.73%, 10=1.37%, 20=0.20%, 50=0.01%
cpu : usr=9.13%, sys=17.29%, ctx=3678872, majf=0, minf=10
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
issued rwts: total=0,3698694,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=128
Run status group 0 (all jobs):
WRITE: bw=241MiB/s (252MB/s), 241MiB/s-241MiB/s (252MB/s-252MB/s), io=14.1GiB (15.1GB), run=60003-60003msec
Disk stats (read/write):
sdc: ios=99/3690768, merge=0/1, ticks=34/7647046, in_queue=1653072, util=95.86%
#### Journal write latency (with sync):
test: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.12
Starting 1 process
test: (groupid=0, jobs=1): err= 0: pid=4511: Fri Jan 8 10:10:13 2021
write: IOPS=21.3k, BW=83.3MiB/s (87.3MB/s)(4996MiB/60001msec); 0 zone resets
slat (nsec): min=3175, max=60319, avg=4751.80, stdev=3530.06
clat (nsec): min=560, max=213298k, avg=41597.24, stdev=188908.38
lat (usec): min=40, max=213302, avg=46.44, stdev=188.93
clat percentiles (usec):
| 1.00th=[ 34], 5.00th=[ 38], 10.00th=[ 38], 20.00th=[ 38],
| 30.00th=[ 38], 40.00th=[ 39], 50.00th=[ 39], 60.00th=[ 39],
| 70.00th=[ 39], 80.00th=[ 43], 90.00th=[ 49], 95.00th=[ 58],
| 99.00th=[ 91], 99.50th=[ 119], 99.90th=[ 145], 99.95th=[ 155],
| 99.99th=[ 255]
bw ( KiB/s): min=49768, max=87024, per=99.99%, avg=85250.71, stdev=3575.35, samples=119
iops : min=12442, max=21756, avg=21312.71, stdev=893.85, samples=119
lat (nsec) : 750=0.01%, 1000=0.01%
lat (usec) : 4=0.01%, 10=0.01%, 20=0.01%, 50=91.25%, 100=7.94%
lat (usec) : 250=0.80%, 500=0.01%, 750=0.01%
lat (msec) : 250=0.01%
cpu : usr=4.42%, sys=8.52%, ctx=2557800, majf=0, minf=12
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,1278900,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=83.3MiB/s (87.3MB/s), 83.3MiB/s-83.3MiB/s (87.3MB/s-87.3MB/s), io=4996MiB (5238MB), run=60001-60001msec
Disk stats (read/write):
sdc: ios=99/1276329, merge=0/0, ticks=35/55090, in_queue=53184, util=88.70%
#### Journal write latency (with fsync):
test: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.12
Starting 1 process
test: (groupid=0, jobs=1): err= 0: pid=4633: Fri Jan 8 10:11:14 2021
write: IOPS=7983, BW=31.2MiB/s (32.7MB/s)(1871MiB/60001msec); 0 zone resets
slat (nsec): min=2229, max=44534, avg=3142.52, stdev=2179.77
clat (usec): min=3, max=370, avg=32.41, stdev= 3.78
lat (usec): min=32, max=386, avg=35.60, stdev= 4.59
clat percentiles (usec):
| 1.00th=[ 31], 5.00th=[ 31], 10.00th=[ 31], 20.00th=[ 32],
| 30.00th=[ 32], 40.00th=[ 32], 50.00th=[ 32], 60.00th=[ 32],
| 70.00th=[ 32], 80.00th=[ 33], 90.00th=[ 37], 95.00th=[ 39],
| 99.00th=[ 48], 99.50th=[ 50], 99.90th=[ 61], 99.95th=[ 82],
| 99.99th=[ 133]
bw ( KiB/s): min=30496, max=32496, per=100.00%, avg=31933.42, stdev=354.20, samples=119
iops : min= 7624, max= 8124, avg=7983.34, stdev=88.54, samples=119
lat (usec) : 4=0.01%, 10=0.01%, 20=0.19%, 50=99.37%, 100=0.41%
lat (usec) : 250=0.03%, 500=0.01%
fsync/fdatasync/sync_file_range:
sync (nsec): min=17, max=17430, avg=87.69, stdev=182.92
sync percentiles (nsec):
| 1.00th=[ 41], 5.00th=[ 42], 10.00th=[ 44], 20.00th=[ 46],
| 30.00th=[ 52], 40.00th=[ 83], 50.00th=[ 88], 60.00th=[ 89],
| 70.00th=[ 91], 80.00th=[ 92], 90.00th=[ 100], 95.00th=[ 155],
| 99.00th=[ 310], 99.50th=[ 310], 99.90th=[ 370], 99.95th=[ 426],
| 99.99th=[12864]
cpu : usr=2.54%, sys=5.63%, ctx=1437106, majf=0, minf=11
IO depths : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,479033,0,479033 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=31.2MiB/s (32.7MB/s), 31.2MiB/s-31.2MiB/s (32.7MB/s-32.7MB/s), io=1871MiB (1962MB), run=60001-60001msec
Disk stats (read/write):
sdc: ios=43/956141, merge=0/0, ticks=14/56785, in_queue=15364, util=25.63%
#### Single-threaded random write latency:
test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.12
Starting 1 process
test: (groupid=0, jobs=1): err= 0: pid=4783: Fri Jan 8 10:12:14 2021
write: IOPS=20.9k, BW=81.7MiB/s (85.6MB/s)(4901MiB/60001msec); 0 zone resets
slat (nsec): min=3143, max=88319, avg=4847.39, stdev=3683.61
clat (nsec): min=612, max=218324k, avg=42221.09, stdev=337573.02
lat (usec): min=40, max=218327, avg=47.16, stdev=337.59
clat percentiles (usec):
| 1.00th=[ 34], 5.00th=[ 38], 10.00th=[ 38], 20.00th=[ 39],
| 30.00th=[ 39], 40.00th=[ 39], 50.00th=[ 39], 60.00th=[ 39],
| 70.00th=[ 40], 80.00th=[ 43], 90.00th=[ 49], 95.00th=[ 58],
| 99.00th=[ 89], 99.50th=[ 117], 99.90th=[ 143], 99.95th=[ 151],
| 99.99th=[ 225]
bw ( KiB/s): min=64544, max=86536, per=99.99%, avg=83623.85, stdev=4178.13, samples=119
iops : min=16136, max=21634, avg=20905.94, stdev=1044.53, samples=119
lat (nsec) : 750=0.01%
lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=90.94%, 100=8.30%
lat (usec) : 250=0.75%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 250=0.01%
cpu : usr=4.66%, sys=8.69%, ctx=2509075, majf=0, minf=11
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,1254533,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=81.7MiB/s (85.6MB/s), 81.7MiB/s-81.7MiB/s (85.6MB/s-85.6MB/s), io=4901MiB (5139MB), run=60001-60001msec
Disk stats (read/write):
sdc: ios=43/1251877, merge=0/0, ticks=13/54787, in_queue=53408, util=89.08%