Page MenuHomePhabricator
Paste P13687

cloudcephosd1001 test results with_cache_enabled.txt
ActivePublic

Authored by dcaro on Fri, Jan 8, 10:52 AM.
##### Linear read:
test: (g=0): rw=read, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=32
fio-3.12
Starting 1 process
test: (groupid=0, jobs=1): err= 0: pid=2583: Fri Jan 8 09:54:31 2021
read: IOPS=122, BW=491MiB/s (515MB/s)(28.8GiB/60129msec)
slat (usec): min=304, max=195190, avg=8118.89, stdev=5396.63
clat (msec): min=126, max=447, avg=252.28, stdev=29.67
lat (msec): min=133, max=457, avg=260.40, stdev=30.08
clat percentiles (msec):
| 1.00th=[ 243], 5.00th=[ 243], 10.00th=[ 245], 20.00th=[ 245],
| 30.00th=[ 247], 40.00th=[ 247], 50.00th=[ 247], 60.00th=[ 249],
| 70.00th=[ 249], 80.00th=[ 251], 90.00th=[ 255], 95.00th=[ 257],
| 99.00th=[ 435], 99.50th=[ 439], 99.90th=[ 447], 99.95th=[ 447],
| 99.99th=[ 447]
bw ( KiB/s): min=180224, max=524288, per=99.77%, avg=501843.36, stdev=48790.69, samples=120
iops : min= 44, max= 128, avg=122.48, stdev=11.96, samples=120
lat (msec) : 250=75.45%, 500=24.55%
cpu : usr=0.11%, sys=1.23%, ctx=14754, majf=0, minf=584
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.2%, 32=99.6%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwts: total=7384,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
Run status group 0 (all jobs):
READ: bw=491MiB/s (515MB/s), 491MiB/s-491MiB/s (515MB/s-515MB/s), io=28.8GiB (30.0GB), run=60129-60129msec
Disk stats (read/write):
sdc: ios=117655/0, merge=0/0, ticks=15109963/0, in_queue=1917056, util=99.91%
##### Linear write:
test: (g=0): rw=write, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=32
fio-3.12
Starting 1 process
test: (groupid=0, jobs=1): err= 0: pid=2711: Fri Jan 8 09:55:32 2021
write: IOPS=101, BW=406MiB/s (426MB/s)(23.9GiB/60154msec); 0 zone resets
slat (usec): min=113, max=192867, avg=9817.21, stdev=5748.78
clat (msec): min=151, max=485, avg=305.10, stdev=32.37
lat (msec): min=157, max=494, avg=314.92, stdev=32.91
clat percentiles (msec):
| 1.00th=[ 300], 5.00th=[ 300], 10.00th=[ 300], 20.00th=[ 300],
| 30.00th=[ 300], 40.00th=[ 300], 50.00th=[ 300], 60.00th=[ 300],
| 70.00th=[ 300], 80.00th=[ 300], 90.00th=[ 300], 95.00th=[ 300],
| 99.00th=[ 485], 99.50th=[ 485], 99.90th=[ 485], 99.95th=[ 485],
| 99.99th=[ 485]
bw ( KiB/s): min=294912, max=425984, per=99.73%, avg=414781.17, stdev=27175.95, samples=120
iops : min= 72, max= 104, avg=101.26, stdev= 6.63, samples=120
lat (msec) : 250=0.34%, 500=99.66%
cpu : usr=1.86%, sys=0.67%, ctx=12219, majf=0, minf=10
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.3%, 32=99.5%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwts: total=0,6108,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
Run status group 0 (all jobs):
WRITE: bw=406MiB/s (426MB/s), 406MiB/s-406MiB/s (426MB/s-426MB/s), io=23.9GiB (25.6GB), run=60154-60154msec
Disk stats (read/write):
sdc: ios=16/97639, merge=0/0, ticks=1/15146829, in_queue=1923456, util=99.86%
#### Peak parallel random read:
test: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=128
fio-3.12
Starting 1 process
test: (groupid=0, jobs=1): err= 0: pid=2981: Fri Jan 8 09:56:33 2021
read: IOPS=69.6k, BW=272MiB/s (285MB/s)(15.9GiB/60002msec)
slat (nsec): min=1374, max=91177, avg=2257.25, stdev=1525.94
clat (usec): min=166, max=211496, avg=1834.82, stdev=1856.61
lat (usec): min=168, max=211498, avg=1837.13, stdev=1856.61
clat percentiles (usec):
| 1.00th=[ 449], 5.00th=[ 594], 10.00th=[ 725], 20.00th=[ 996],
| 30.00th=[ 1254], 40.00th=[ 1500], 50.00th=[ 1762], 60.00th=[ 2024],
| 70.00th=[ 2278], 80.00th=[ 2540], 90.00th=[ 2835], 95.00th=[ 3032],
| 99.00th=[ 4817], 99.50th=[ 6652], 99.90th=[10421], 99.95th=[11863],
| 99.99th=[15664]
bw ( KiB/s): min=167168, max=282600, per=99.99%, avg=278580.71, stdev=14490.47, samples=120
iops : min=41792, max=70650, avg=69645.12, stdev=3622.61, samples=120
lat (usec) : 250=0.01%, 500=1.89%, 750=8.99%, 1000=9.27%
lat (msec) : 2=39.08%, 4=39.43%, 10=1.21%, 20=0.12%, 50=0.01%
lat (msec) : 250=0.01%
cpu : usr=9.22%, sys=19.66%, ctx=4058357, majf=0, minf=137
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
issued rwts: total=4179119,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=128
Run status group 0 (all jobs):
READ: bw=272MiB/s (285MB/s), 272MiB/s-272MiB/s (285MB/s-285MB/s), io=15.9GiB (17.1GB), run=60002-60002msec
Disk stats (read/write):
sdc: ios=4170185/0, merge=0/0, ticks=7641255/0, in_queue=1712812, util=98.88%
#### Single-threaded read latency:
test: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.12
Starting 1 process
test: (groupid=0, jobs=1): err= 0: pid=3140: Fri Jan 8 09:57:33 2021
read: IOPS=9653, BW=37.7MiB/s (39.5MB/s)(2263MiB/60001msec)
slat (nsec): min=2195, max=52847, avg=3892.06, stdev=2903.50
clat (usec): min=11, max=867, avg=98.90, stdev=24.97
lat (usec): min=29, max=870, avg=102.90, stdev=25.20
clat percentiles (usec):
| 1.00th=[ 29], 5.00th=[ 76], 10.00th=[ 78], 20.00th=[ 79],
| 30.00th=[ 83], 40.00th=[ 95], 50.00th=[ 96], 60.00th=[ 98],
| 70.00th=[ 126], 80.00th=[ 128], 90.00th=[ 129], 95.00th=[ 133],
| 99.00th=[ 141], 99.50th=[ 145], 99.90th=[ 178], 99.95th=[ 192],
| 99.99th=[ 239]
bw ( KiB/s): min=37888, max=39864, per=100.00%, avg=38615.21, stdev=345.27, samples=119
iops : min= 9472, max= 9966, avg=9653.82, stdev=86.31, samples=119
lat (usec) : 20=0.01%, 50=3.83%, 100=57.94%, 250=38.21%, 500=0.01%
lat (usec) : 750=0.01%, 1000=0.01%
cpu : usr=2.48%, sys=5.35%, ctx=579231, majf=0, minf=10
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=579223,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=37.7MiB/s (39.5MB/s), 37.7MiB/s-37.7MiB/s (39.5MB/s-39.5MB/s), io=2263MiB (2372MB), run=60001-60001msec
Disk stats (read/write):
sdc: ios=578001/0, merge=0/0, ticks=57153/0, in_queue=56424, util=94.12%
#### Peak parallel random write:
test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=128
fio-3.12
Starting 1 process
test: (groupid=0, jobs=1): err= 0: pid=3226: Fri Jan 8 09:58:34 2021
write: IOPS=61.7k, BW=241MiB/s (253MB/s)(14.1GiB/60003msec); 0 zone resets
slat (nsec): min=1470, max=114182, avg=2188.00, stdev=986.11
clat (usec): min=476, max=23148, avg=2072.33, stdev=1092.47
lat (usec): min=478, max=23150, avg=2074.55, stdev=1092.47
clat percentiles (usec):
| 1.00th=[ 570], 5.00th=[ 685], 10.00th=[ 832], 20.00th=[ 1123],
| 30.00th=[ 1418], 40.00th=[ 1713], 50.00th=[ 2008], 60.00th=[ 2311],
| 70.00th=[ 2606], 80.00th=[ 2900], 90.00th=[ 3228], 95.00th=[ 3425],
| 99.00th=[ 5407], 99.50th=[ 7504], 99.90th=[11600], 99.95th=[13173],
| 99.99th=[16319]
bw ( KiB/s): min=243880, max=248800, per=99.99%, avg=246708.80, stdev=1023.92, samples=120
iops : min=60970, max=62200, avg=61677.16, stdev=255.95, samples=120
lat (usec) : 500=0.01%, 750=7.17%, 1000=8.50%
lat (msec) : 2=34.07%, 4=48.71%, 10=1.34%, 20=0.20%, 50=0.01%
cpu : usr=8.68%, sys=17.94%, ctx=3679959, majf=0, minf=9
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
issued rwts: total=0,3701068,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=128
Run status group 0 (all jobs):
WRITE: bw=241MiB/s (253MB/s), 241MiB/s-241MiB/s (253MB/s-253MB/s), io=14.1GiB (15.2GB), run=60003-60003msec
Disk stats (read/write):
sdc: ios=99/3693180, merge=0/1, ticks=32/7646934, in_queue=1654220, util=95.60%
#### Journal write latency (with sync):
test: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.12
Starting 1 process
test: (groupid=0, jobs=1): err= 0: pid=3337: Fri Jan 8 09:59:34 2021
write: IOPS=19.8k, BW=77.2MiB/s (80.0MB/s)(4633MiB/60001msec); 0 zone resets
slat (nsec): min=3087, max=62081, avg=4714.77, stdev=3560.04
clat (usec): min=4, max=203545, avg=45.31, stdev=187.70
lat (usec): min=40, max=203557, avg=50.12, stdev=187.74
clat percentiles (usec):
| 1.00th=[ 36], 5.00th=[ 38], 10.00th=[ 38], 20.00th=[ 38],
| 30.00th=[ 38], 40.00th=[ 39], 50.00th=[ 39], 60.00th=[ 40],
| 70.00th=[ 43], 80.00th=[ 47], 90.00th=[ 60], 95.00th=[ 80],
| 99.00th=[ 127], 99.50th=[ 143], 99.90th=[ 188], 99.95th=[ 219],
| 99.99th=[ 297]
bw ( KiB/s): min=63400, max=81920, per=100.00%, avg=79064.08, stdev=2141.77, samples=119
iops : min=15850, max=20480, avg=19766.02, stdev=535.44, samples=119
lat (usec) : 10=0.01%, 20=0.01%, 50=83.85%, 100=13.50%, 250=2.61%
lat (usec) : 500=0.03%, 750=0.01%
lat (msec) : 250=0.01%
cpu : usr=3.85%, sys=8.17%, ctx=2372077, majf=0, minf=11
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,1186033,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=77.2MiB/s (80.0MB/s), 77.2MiB/s-77.2MiB/s (80.0MB/s-80.0MB/s), io=4633MiB (4858MB), run=60001-60001msec
Disk stats (read/write):
sdc: ios=99/1183600, merge=0/0, ticks=30/55404, in_queue=54004, util=90.08%
#### Journal write latency (with fsync):
test: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.12
Starting 1 process
test: (groupid=0, jobs=1): err= 0: pid=3445: Fri Jan 8 10:00:35 2021
write: IOPS=7822, BW=30.6MiB/s (32.0MB/s)(1834MiB/60001msec); 0 zone resets
slat (nsec): min=2275, max=55303, avg=3251.89, stdev=2244.18
clat (nsec): min=641, max=788264, avg=33690.74, stdev=8044.14
lat (usec): min=32, max=791, avg=37.02, stdev= 8.50
clat percentiles (usec):
| 1.00th=[ 31], 5.00th=[ 31], 10.00th=[ 31], 20.00th=[ 32],
| 30.00th=[ 32], 40.00th=[ 32], 50.00th=[ 32], 60.00th=[ 32],
| 70.00th=[ 33], 80.00th=[ 34], 90.00th=[ 38], 95.00th=[ 44],
| 99.00th=[ 71], 99.50th=[ 86], 99.90th=[ 125], 99.95th=[ 139],
| 99.99th=[ 186]
bw ( KiB/s): min=30384, max=31912, per=100.00%, avg=31290.39, stdev=285.57, samples=119
iops : min= 7596, max= 7978, avg=7822.59, stdev=71.40, samples=119
lat (nsec) : 750=0.01%
lat (usec) : 20=0.19%, 50=97.25%, 100=2.27%, 250=0.29%, 500=0.01%
lat (usec) : 1000=0.01%
fsync/fdatasync/sync_file_range:
sync (nsec): min=22, max=20071, avg=92.57, stdev=202.82
sync percentiles (nsec):
| 1.00th=[ 40], 5.00th=[ 42], 10.00th=[ 46], 20.00th=[ 49],
| 30.00th=[ 54], 40.00th=[ 87], 50.00th=[ 90], 60.00th=[ 94],
| 70.00th=[ 96], 80.00th=[ 97], 90.00th=[ 105], 95.00th=[ 169],
| 99.00th=[ 330], 99.50th=[ 330], 99.90th=[ 394], 99.95th=[ 478],
| 99.99th=[12992]
cpu : usr=2.44%, sys=5.83%, ctx=1408157, majf=0, minf=15
IO depths : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,469385,0,469384 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=30.6MiB/s (32.0MB/s), 30.6MiB/s-30.6MiB/s (32.0MB/s-32.0MB/s), io=1834MiB (1923MB), run=60001-60001msec
Disk stats (read/write):
sdc: ios=43/936917, merge=0/0, ticks=14/56741, in_queue=15360, util=25.62%
#### Single-threaded random write latency:
test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.12
Starting 1 process
test: (groupid=0, jobs=1): err= 0: pid=3597: Fri Jan 8 10:01:36 2021
write: IOPS=19.0k, BW=74.2MiB/s (77.8MB/s)(4453MiB/60001msec); 0 zone resets
slat (nsec): min=3137, max=66528, avg=4887.01, stdev=3713.07
clat (nsec): min=615, max=207741k, avg=47005.20, stdev=195752.10
lat (usec): min=40, max=207744, avg=51.98, stdev=195.80
clat percentiles (usec):
| 1.00th=[ 36], 5.00th=[ 38], 10.00th=[ 38], 20.00th=[ 39],
| 30.00th=[ 39], 40.00th=[ 39], 50.00th=[ 39], 60.00th=[ 41],
| 70.00th=[ 44], 80.00th=[ 48], 90.00th=[ 62], 95.00th=[ 87],
| 99.00th=[ 145], 99.50th=[ 174], 99.90th=[ 241], 99.95th=[ 277],
| 99.99th=[ 383]
bw ( KiB/s): min=45792, max=77752, per=99.99%, avg=75996.82, stdev=2844.48, samples=119
iops : min=11448, max=19438, avg=18999.23, stdev=711.12, samples=119
lat (nsec) : 750=0.01%
lat (usec) : 10=0.01%, 20=0.01%, 50=82.25%, 100=14.08%, 250=3.58%
lat (usec) : 500=0.08%, 750=0.01%
lat (msec) : 250=0.01%
cpu : usr=4.29%, sys=7.94%, ctx=2280106, majf=0, minf=12
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,1140051,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=74.2MiB/s (77.8MB/s), 74.2MiB/s-74.2MiB/s (77.8MB/s-77.8MB/s), io=4453MiB (4670MB), run=60001-60001msec
Disk stats (read/write):
sdc: ios=43/1137671, merge=0/0, ticks=11/55285, in_queue=53912, util=89.93%