subでちょっと会話をしていたんだけど自宅のインターネットを10Gbpsに対応したくなったのでそのメモ
目的
自宅サーバーのNASがSSDx1(ログデバイス), HDDx3でraidz1で、さらにメモリもそこそこデカい量を積んでいるので、ランダムライトでもメモリに乗る程度の書き込みであればSSD並の速度が出る
`-$ fio -rw=randwrite -bs=4k -size=10m -numjobs=10 -runtime=20 -direct=1 -invalidate=1 -ioengine=libaio -iodepth=32 -iodepth_batch=32 -group_reporting -name=randwrite
randwrite: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32
…
fio-3.34
Starting 10 processes
Jobs: 10 (f=10)
randwrite: (groupid=0, jobs=10): err= 0: pid=2435518: Thu Jul 27 22:52:11 2023
write: IOPS=20.8k, BW=81.4MiB/s (85.3MB/s)(100MiB/1229msec); 0 zone resets
slat (usec): min=4, max=1027.0k, avg=10028.13, stdev=89176.47
clat (usec): min=2, max=202267, avg=4040.55, stdev=16506.15
lat (usec): min=7, max=1079.7k, avg=14068.68, stdev=97285.65
clat percentiles (usec):
| 1.00th=188, 5.00th=210, 10.00th=225, 20.00th=231,
| 30.00th=251, 40.00th=289, 50.00th=326, 60.00th=355,
| 70.00th=420, 80.00th=474, 90.00th=1029, 95.00th=23987,
| 99.00th=90702, 99.50th=127402, 99.90th=166724, 99.95th=173016,
| 99.99th=187696
bw ( KiB/s): min= 5208, max= 5208, per=6.25%, avg=5208.00, stdev= 0.00, samples=8
iops : min= 1302, max= 1302, avg=1302.00, stdev= 0.00, samples=8
lat (usec) : 4=0.04%, 10=0.03%, 20=0.07%, 50=0.16%, 100=0.26%
lat (usec) : 250=28.16%, 500=52.18%, 750=4.68%, 1000=4.32%
lat (msec) : 2=0.86%, 4=0.41%, 10=1.51%, 20=1.95%, 50=2.51%
lat (msec) : 100=2.04%, 250=0.84%
cpu : usr=0.75%, sys=2.51%, ctx=709, majf=0, minf=97
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwts: total=0,25600,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
Run status group 0 (all jobs):
WRITE: bw=81.4MiB/s (85.3MB/s), 81.4MiB/s-81.4MiB/s (85.3MB/s-85.3MB/s), io=100MiB (105MB), run=1229-1229msec
10MBのデータを20並列でシーケンシャルに書き込みケースだと4112MB/s
`-$ fio -rw=write -bs=4k -size=10m -numjobs=20 -runtime=20 -direct=1 -invalidate=1 -ioengine=libaio -iodepth=32 -iodepth_batch=32 -group_reporting -name=seqwrite
seqwrite: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32
…
fio-3.34
Starting 20 processes
seqwrite: (groupid=0, jobs=20): err= 0: pid=2447032: Thu Jul 27 22:59:09 2023
write: IOPS=1004k, BW=3922MiB/s (4112MB/s)(200MiB/51msec); 0 zone resets
slat (usec): min=3, max=11181, avg=22.50, stdev=187.54
clat (nsec): min=1513, max=13808k, avg=448116.12, stdev=1009115.25
lat (usec): min=5, max=13815, avg=470.61, stdev=1031.29
clat percentiles (usec):
| 1.00th=143, 5.00th=196, 10.00th=277, 20.00th=289,
| 30.00th=293, 40.00th=297, 50.00th=297, 60.00th=302,
| 70.00th=306, 80.00th=314, 90.00th=334, 95.00th=392,
| 99.00th=5538, 99.50th=7439, 99.90th=13829, 99.95th=13829,
| 99.99th=13829
lat (usec) : 2=0.01%, 4=0.03%, 10=0.03%, 20=0.09%, 50=0.17%
lat (usec) : 100=0.36%, 250=5.70%, 500=89.35%, 750=0.43%, 1000=0.23%
lat (msec) : 2=1.25%, 4=0.50%, 10=1.50%, 20=0.34%
cpu : usr=10.24%, sys=50.44%, ctx=323, majf=0, minf=253
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwts: total=0,51200,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
Run status group 0 (all jobs):
WRITE: bw=3922MiB/s (4112MB/s), 3922MiB/s-3922MiB/s (4112MB/s-4112MB/s), io=200MiB (210MB), run=51-51msec
RAW画像ぐらいを想定した50MBのデータを20並列で書き込むケースで2709MB/s
`-$ fio -rw=write -bs=4k -size=50m -numjobs=20 -runtime=10 -direct=1 -invalidate=1 -ioengine=libaio -iodepth=32 -iodepth_batch=32 -group_reporting -name=seqwrite
seqwrite: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32
…
fio-3.34
Starting 20 processes
seqwrite: (groupid=0, jobs=20): err= 0: pid=2485184: Thu Jul 27 23:02:43 2023
write: IOPS=661k, BW=2584MiB/s (2709MB/s)(1000MiB/387msec); 0 zone resets
slat (usec): min=3, max=57384, avg=22.17, stdev=444.21
clat (nsec): min=1637, max=57617k, avg=749160.33, stdev=2743626.14
lat (usec): min=5, max=57622, avg=771.33, stdev=2784.41
clat percentiles (usec):
| 1.00th=198, 5.00th=233, 10.00th=243, 20.00th=265,
| 30.00th=277, 40.00th=289, 50.00th=297, 60.00th=302,
| 70.00th=310, 80.00th=322, 90.00th=379, 95.00th=1909,
| 99.00th=13304, 99.50th=20055, 99.90th=37487, 99.95th=43779,
| 99.99th=57410
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.04%
lat (usec) : 100=0.10%, 250=13.03%, 500=79.29%, 750=0.71%, 1000=0.39%
lat (msec) : 2=1.53%, 4=1.57%, 10=1.82%, 20=0.99%, 50=0.48%
lat (msec) : 100=0.03%
cpu : usr=7.81%, sys=29.54%, ctx=2684, majf=0, minf=249
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwts: total=0,256000,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
Run status group 0 (all jobs):
WRITE: bw=2584MiB/s (2709MB/s), 2584MiB/s-2584MiB/s (2709MB/s-2709MB/s), io=1000MiB (1049MB), run=387-387msec
動画ぐらいを想定したケースだと流石に遅くなって436MB/sぐらい。
`-$ fio -rw=write -bs=4k -size=1000m -numjobs=10 -runtime=10 -direct=1 -invalidate=1 -ioengine=libaio -iodepth=32 -iodepth_batch=32 -group_reporting -name=seqwrite
seqwrite: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32
…
fio-3.34
Starting 10 processes
seqwrite: Laying out IO file (1 file / 1000MiB)
seqwrite: Laying out IO file (1 file / 1000MiB)
seqwrite: Laying out IO file (1 file / 1000MiB)
seqwrite: Laying out IO file (1 file / 1000MiB)
seqwrite: Laying out IO file (1 file / 1000MiB)
Jobs: 10 (f=10): W(10)100.0%w=291MiB/sw=74.5k IOPSeta 00m:00s
seqwrite: (groupid=0, jobs=10): err= 0: pid=2489001: Thu Jul 27 23:03:32 2023
write: IOPS=106k, BW=416MiB/s (436MB/s)(4159MiB/10001msec); 0 zone resets
slat (usec): min=3, max=30757, avg=90.19, stdev=138.93
clat (usec): min=4, max=51680, avg=2913.74, stdev=3073.13
lat (usec): min=51, max=52344, avg=3003.93, stdev=3168.36
clat percentiles (usec):
| 1.00th=176, 5.00th=198, 10.00th=237, 20.00th=277,
| 30.00th=285, 40.00th=322, 50.00th=3687, 60.00th=4228,
| 70.00th=4490, 80.00th=4686, 90.00th=5276, 95.00th=6259,
| 99.00th=16450, 99.50th=18220, 99.90th=20841, 99.95th=21103,
| 99.99th=21627
bw ( KiB/s): min=68192, max=2990128, per=100.00%, avg=431719.16, stdev=65027.97, samples=190
iops : min=17048, max=747532, avg=107929.79, stdev=16256.99, samples=190
lat (usec) : 10=0.01%, 20=0.01%, 100=0.01%, 250=13.00%, 500=31.22%
lat (usec) : 750=0.90%, 1000=0.39%
lat (msec) : 2=0.77%, 4=7.80%, 10=43.18%, 20=2.51%, 50=0.23%
lat (msec) : 100=0.01%
cpu : usr=4.93%, sys=20.68%, ctx=522154, majf=0, minf=105
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwts: total=0,1064679,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
Run status group 0 (all jobs):
WRITE: bw=416MiB/s (436MB/s), 416MiB/s-416MiB/s (436MB/s-436MB/s), io=4159MiB (4361MB), run=10001-10001msec
fio -rw=write -bs=4k -size=1000m -numjobs=10 -runtime=10 -direct=1 5.12s user 20.78s system 249% cpu 10.367 total
特段早いメモリ（DDR4）やストレージ（SSDもHDDもふつうのSATA）を使っているわけではないけど、小さなファイルを書き込む（読み込みも同様なぐらい早い）だとNASのストレージをマウントして常用できるとは思う。しかし、この性能を活かすには帯域が1Gbpsでは遅すぎるし、現状でもNVMeからNASにファイルを転送しても110MB/sぐらいでサチる。10倍、ないし5倍ぐらいは速くしたい。
録画サーバーのストレージとして活用しているけど実態としては16TBぐらいあるので家中のデータをかき集めてここに集約してもさほど問題ない状態にはなりつつある。また以前に組んだ記憶域プール階層化計画での反省もあり、自宅サーバーのNASに集約していきたい。反省についてはまた後日にでも…。
目的をまとめると
- そこそこな性能を持つNASをもっと活用したい
- 転送速度がボトルネックになるのは見えているのでそこの改善をしたい
- 巨大なストレージは人生を豊かにするという信条を貫く
方針
最終的にはフレッツ光クロスで宅内の外の帯域も10Gbpsにしたいが引っ越しもありそうなので宅内、ないし自室だけでも