Page MenuHomePhabricator
Paste P13664

Speedtest cloudceph2001-dev.codfw.wmnet write cache enabled
ActivePublic

Authored by dcaro on Jan 7 2021, 3:17 PM.
##### Linear read:
test: (g=0): rw=read, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=32
fio-3.12
Starting 1 process
test: (groupid=0, jobs=1): err= 0: pid=29315: Thu Jan 7 15:05:36 2021
read: IOPS=130, BW=522MiB/s (547MB/s)(30.6GiB/60125msec)
slat (usec): min=25, max=26588, avg=7645.76, stdev=7540.41
clat (msec): min=112, max=341, avg=237.53, stdev= 9.09
lat (msec): min=127, max=356, avg=245.18, stdev= 7.47
clat percentiles (msec):
| 1.00th=[ 228], 5.00th=[ 228], 10.00th=[ 228], 20.00th=[ 228],
| 30.00th=[ 230], 40.00th=[ 243], 50.00th=[ 243], 60.00th=[ 243],
| 70.00th=[ 243], 80.00th=[ 245], 90.00th=[ 245], 95.00th=[ 245],
| 99.00th=[ 253], 99.50th=[ 253], 99.90th=[ 296], 99.95th=[ 326],
| 99.99th=[ 342]
bw ( KiB/s): min=376832, max=540672, per=99.79%, avg=533136.09, stdev=15810.88, samples=120
iops : min= 92, max= 132, avg=130.14, stdev= 3.86, samples=120
lat (msec) : 250=97.96%, 500=2.04%
cpu : usr=0.10%, sys=1.05%, ctx=3992, majf=0, minf=584
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.2%, 32=99.6%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwts: total=7842,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
Run status group 0 (all jobs):
READ: bw=522MiB/s (547MB/s), 522MiB/s-522MiB/s (547MB/s-547MB/s), io=30.6GiB (32.9GB), run=60125-60125msec
Disk stats (read/write):
sdc: ios=31491/0, merge=0/0, ticks=3615706/0, in_queue=1916672, util=99.89%
##### Linear write:
test: (g=0): rw=write, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=32
fio-3.12
Starting 1 process
test: (groupid=0, jobs=1): err= 0: pid=29370: Thu Jan 7 15:06:37 2021
write: IOPS=128, BW=512MiB/s (537MB/s)(30.1GiB/60115msec); 0 zone resets
slat (usec): min=101, max=29555, avg=7791.10, stdev=7493.21
clat (msec): min=114, max=346, avg=242.04, stdev= 9.97
lat (msec): min=124, max=360, avg=249.84, stdev= 8.59
clat percentiles (msec):
| 1.00th=[ 230], 5.00th=[ 230], 10.00th=[ 232], 20.00th=[ 232],
| 30.00th=[ 234], 40.00th=[ 247], 50.00th=[ 247], 60.00th=[ 247],
| 70.00th=[ 247], 80.00th=[ 249], 90.00th=[ 249], 95.00th=[ 253],
| 99.00th=[ 271], 99.50th=[ 279], 99.90th=[ 300], 99.95th=[ 330],
| 99.99th=[ 347]
bw ( KiB/s): min=376832, max=540672, per=99.78%, avg=523160.83, stdev=16345.85, samples=120
iops : min= 92, max= 132, avg=127.70, stdev= 3.99, samples=120
lat (msec) : 250=94.26%, 500=5.74%
cpu : usr=2.84%, sys=0.83%, ctx=3906, majf=0, minf=9
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.2%, 32=99.6%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwts: total=0,7695,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
Run status group 0 (all jobs):
WRITE: bw=512MiB/s (537MB/s), 512MiB/s-512MiB/s (537MB/s-537MB/s), io=30.1GiB (32.3GB), run=60115-60115msec
Disk stats (read/write):
sdc: ios=47/31020, merge=0/0, ticks=341/3623060, in_queue=1922872, util=99.86%
#### Peak parallel random read:
test: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=128
fio-3.12
Starting 1 process
test: (groupid=0, jobs=1): err= 0: pid=30087: Thu Jan 7 15:07:38 2021
read: IOPS=54.2k, BW=212MiB/s (222MB/s)(12.4GiB/60002msec)
slat (nsec): min=966, max=5051.1k, avg=17719.73, stdev=48814.33
clat (usec): min=818, max=25140, avg=2343.89, stdev=866.19
lat (usec): min=822, max=25142, avg=2361.69, stdev=871.49
clat percentiles (usec):
| 1.00th=[ 947], 5.00th=[ 1074], 10.00th=[ 1254], 20.00th=[ 1500],
| 30.00th=[ 1745], 40.00th=[ 2008], 50.00th=[ 2278], 60.00th=[ 2573],
| 70.00th=[ 2868], 80.00th=[ 3163], 90.00th=[ 3523], 95.00th=[ 3720],
| 99.00th=[ 4228], 99.50th=[ 4555], 99.90th=[ 5604], 99.95th=[ 6063],
| 99.99th=[ 7373]
bw ( KiB/s): min=199328, max=273312, per=99.99%, avg=216727.65, stdev=10249.28, samples=120
iops : min=49832, max=68328, avg=54181.88, stdev=2562.33, samples=120
lat (usec) : 1000=2.37%
lat (msec) : 2=37.63%, 4=58.09%, 10=1.91%, 20=0.01%, 50=0.01%
cpu : usr=3.70%, sys=12.21%, ctx=362327, majf=0, minf=137
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
issued rwts: total=3251416,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=128
Run status group 0 (all jobs):
READ: bw=212MiB/s (222MB/s), 212MiB/s-212MiB/s (222MB/s-222MB/s), io=12.4GiB (13.3GB), run=60002-60002msec
Disk stats (read/write):
sdc: ios=3245618/0, merge=1/0, ticks=3608598/0, in_queue=1915672, util=99.88%
#### Single-threaded read latency:
test: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.12
Starting 1 process
test: (groupid=0, jobs=1): err= 0: pid=31319: Thu Jan 7 15:08:38 2021
read: IOPS=7894, BW=30.8MiB/s (32.3MB/s)(1850MiB/60001msec)
slat (nsec): min=1684, max=197774, avg=6913.25, stdev=1795.98
clat (nsec): min=1609, max=2510.2k, avg=118590.14, stdev=20728.20
lat (usec): min=83, max=2518, avg=125.71, stdev=20.92
clat percentiles (usec):
| 1.00th=[ 92], 5.00th=[ 95], 10.00th=[ 97], 20.00th=[ 99],
| 30.00th=[ 101], 40.00th=[ 113], 50.00th=[ 115], 60.00th=[ 116],
| 70.00th=[ 143], 80.00th=[ 145], 90.00th=[ 147], 95.00th=[ 147],
| 99.00th=[ 149], 99.50th=[ 149], 99.90th=[ 155], 99.95th=[ 161],
| 99.99th=[ 223]
bw ( KiB/s): min=30656, max=32624, per=100.00%, avg=31579.40, stdev=481.58, samples=119
iops : min= 7664, max= 8156, avg=7894.84, stdev=120.41, samples=119
lat (usec) : 2=0.01%, 100=27.35%, 250=72.64%, 500=0.01%, 750=0.01%
lat (usec) : 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%
cpu : usr=2.10%, sys=8.95%, ctx=473663, majf=0, minf=10
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=473659,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=30.8MiB/s (32.3MB/s), 30.8MiB/s-30.8MiB/s (32.3MB/s-32.3MB/s), io=1850MiB (1940MB), run=60001-60001msec
Disk stats (read/write):
sdc: ios=472836/0, merge=0/0, ticks=56172/0, in_queue=52920, util=88.26%
#### Peak parallel random write:
test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=128
fio-3.12
Starting 1 process
test: (groupid=0, jobs=1): err= 0: pid=31388: Thu Jan 7 15:09:39 2021
write: IOPS=83.9k, BW=328MiB/s (344MB/s)(19.2GiB/60001msec); 0 zone resets
slat (nsec): min=1005, max=3836.8k, avg=11174.77, stdev=29161.94
clat (usec): min=671, max=13871, avg=1514.21, stdev=301.87
lat (usec): min=765, max=13872, avg=1525.46, stdev=303.75
clat percentiles (usec):
| 1.00th=[ 1172], 5.00th=[ 1188], 10.00th=[ 1254], 20.00th=[ 1303],
| 30.00th=[ 1385], 40.00th=[ 1418], 50.00th=[ 1516], 60.00th=[ 1549],
| 70.00th=[ 1631], 80.00th=[ 1663], 90.00th=[ 1762], 95.00th=[ 1827],
| 99.00th=[ 2540], 99.50th=[ 2868], 99.90th=[ 4178], 99.95th=[ 5538],
| 99.99th=[ 9110]
bw ( KiB/s): min=288776, max=339048, per=99.99%, avg=335517.15, stdev=5820.56, samples=120
iops : min=72194, max=84762, avg=83879.35, stdev=1454.86, samples=120
lat (usec) : 750=0.01%, 1000=0.01%
lat (msec) : 2=96.94%, 4=2.94%, 10=0.11%, 20=0.01%
cpu : usr=5.88%, sys=18.55%, ctx=590458, majf=0, minf=9
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
issued rwts: total=0,5033311,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=128
Run status group 0 (all jobs):
WRITE: bw=328MiB/s (344MB/s), 328MiB/s-328MiB/s (344MB/s-344MB/s), io=19.2GiB (20.6GB), run=60001-60001msec
Disk stats (read/write):
sdc: ios=43/5023780, merge=0/1, ticks=11/3610148, in_queue=1915300, util=99.89%
#### Journal write latency (with sync):
test: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.12
Starting 1 process
test: (groupid=0, jobs=1): err= 0: pid=31452: Thu Jan 7 15:10:39 2021
write: IOPS=19.8k, BW=77.2MiB/s (80.0MB/s)(4633MiB/60001msec); 0 zone resets
slat (nsec): min=3160, max=60829, avg=3990.93, stdev=856.83
clat (usec): min=2, max=1481, avg=46.17, stdev= 3.78
lat (usec): min=44, max=1485, avg=50.23, stdev= 4.07
clat percentiles (nsec):
| 1.00th=[42240], 5.00th=[42752], 10.00th=[43264], 20.00th=[43776],
| 30.00th=[44288], 40.00th=[44800], 50.00th=[45824], 60.00th=[46336],
| 70.00th=[46848], 80.00th=[47360], 90.00th=[48896], 95.00th=[52480],
| 99.00th=[58624], 99.50th=[61184], 99.90th=[67072], 99.95th=[69120],
| 99.99th=[80384]
bw ( KiB/s): min=75720, max=81576, per=99.97%, avg=79050.44, stdev=2266.51, samples=119
iops : min=18930, max=20394, avg=19762.61, stdev=566.65, samples=119
lat (usec) : 4=0.01%, 20=0.01%, 50=91.51%, 100=8.48%, 250=0.01%
lat (usec) : 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%
cpu : usr=1.98%, sys=6.93%, ctx=2372348, majf=0, minf=13
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,1186169,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=77.2MiB/s (80.0MB/s), 77.2MiB/s-77.2MiB/s (80.0MB/s-80.0MB/s), io=4633MiB (4859MB), run=60001-60001msec
Disk stats (read/write):
sdc: ios=43/1183974, merge=0/0, ticks=12/56512, in_queue=8, util=0.01%
#### Journal write latency (with fsync):
test: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.12
Starting 1 process
test: (groupid=0, jobs=1): err= 0: pid=31562: Thu Jan 7 15:11:40 2021
write: IOPS=18.5k, BW=72.3MiB/s (75.8MB/s)(4336MiB/60001msec); 0 zone resets
slat (nsec): min=1943, max=30207, avg=2638.22, stdev=646.25
clat (usec): min=9, max=1438, avg=23.80, stdev= 2.82
lat (usec): min=23, max=1440, avg=26.55, stdev= 3.00
clat percentiles (nsec):
| 1.00th=[21632], 5.00th=[22144], 10.00th=[22400], 20.00th=[22656],
| 30.00th=[22912], 40.00th=[23168], 50.00th=[23424], 60.00th=[23936],
| 70.00th=[24192], 80.00th=[24192], 90.00th=[24704], 95.00th=[28032],
| 99.00th=[30848], 99.50th=[34560], 99.90th=[39680], 99.95th=[40704],
| 99.99th=[46848]
bw ( KiB/s): min=70712, max=77280, per=100.00%, avg=74001.45, stdev=2166.47, samples=119
iops : min=17678, max=19320, avg=18500.36, stdev=541.64, samples=119
lat (usec) : 10=0.01%, 20=0.01%, 50=99.98%, 100=0.01%, 250=0.01%
lat (usec) : 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%
fsync/fdatasync/sync_file_range:
sync (nsec): min=36, max=16263, avg=68.58, stdev=64.45
sync percentiles (nsec):
| 1.00th=[ 52], 5.00th=[ 61], 10.00th=[ 61], 20.00th=[ 62],
| 30.00th=[ 62], 40.00th=[ 64], 50.00th=[ 66], 60.00th=[ 68],
| 70.00th=[ 74], 80.00th=[ 75], 90.00th=[ 76], 95.00th=[ 78],
| 99.00th=[ 83], 99.50th=[ 89], 99.90th=[ 201], 99.95th=[ 201],
| 99.99th=[ 241]
cpu : usr=3.20%, sys=13.43%, ctx=3329769, majf=0, minf=12
IO depths : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,1109920,0,1109920 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=72.3MiB/s (75.8MB/s), 72.3MiB/s-72.3MiB/s (75.8MB/s-75.8MB/s), io=4336MiB (4546MB), run=60001-60001msec
Disk stats (read/write):
sdc: ios=43/2215944, merge=0/0, ticks=11/53815, in_queue=21664, util=36.10%
#### Single-threaded random write latency:
test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.12
Starting 1 process
test: (groupid=0, jobs=1): err= 0: pid=31610: Thu Jan 7 15:12:40 2021
write: IOPS=19.6k, BW=76.7MiB/s (80.5MB/s)(4604MiB/60001msec); 0 zone resets
slat (nsec): min=3323, max=52485, avg=4136.36, stdev=911.13
clat (usec): min=2, max=9130, avg=46.13, stdev= 9.87
lat (usec): min=44, max=9144, avg=50.34, stdev=10.01
clat percentiles (nsec):
| 1.00th=[42240], 5.00th=[42752], 10.00th=[43264], 20.00th=[43776],
| 30.00th=[44288], 40.00th=[44800], 50.00th=[45312], 60.00th=[45824],
| 70.00th=[46848], 80.00th=[47360], 90.00th=[48896], 95.00th=[52480],
| 99.00th=[59136], 99.50th=[62208], 99.90th=[68096], 99.95th=[70144],
| 99.99th=[86528]
bw ( KiB/s): min=74850, max=80944, per=99.97%, avg=78545.82, stdev=2192.81, samples=119
iops : min=18712, max=20236, avg=19636.44, stdev=548.20, samples=119
lat (usec) : 4=0.01%, 50=91.58%, 100=8.41%, 250=0.01%, 500=0.01%
lat (usec) : 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%
cpu : usr=2.47%, sys=6.98%, ctx=2357045, majf=0, minf=11
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,1178517,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=76.7MiB/s (80.5MB/s), 76.7MiB/s-76.7MiB/s (80.5MB/s-80.5MB/s), io=4604MiB (4827MB), run=60001-60001msec
Disk stats (read/write):
sdc: ios=43/1176272, merge=0/0, ticks=9/56193, in_queue=32, util=0.02%