Nate W
Dabbler
- Joined
- Jul 10, 2014
- Messages
- 38
I set up a SSD ZFS pool and get amazing performance locally (using dd to write/read). When I mount the storage over NFS via VMWare or directly in an Ubuntu VM, the performance is a small percentage of what I get locally.
Here are the server specs:
FreeNAS 9.2.1.5
Xeon L5640
96GB Memory
24x 480gb Intel 320
Chelcio T520-SO-CR (dual 10G fiber)
I have this set up as Striped 4x raidz2 vdevs.
The benchmarks locally (both compressed and compression off) are pretty good, but when I spin up a VM on the same storage over NFS or even mount the storage over NFS directly on an Ubuntu VM, the speeds are a fraction of the local speeds.
I would love to figure out why the speed difference is so great and get this ZFS box rocking.
Thanks in advance!
Here are some of the benchmarks:
Local Benchmarks (dd)
Without Compression:
[root@zfs] /mnt/storage/nocompression# dd if=/dev/zero of=temp.dat bs=4M count=50k
51200+0 records in
51200+0 records out
214748364800 bytes transferred in 341.918976 secs (628067993 bytes/sec)
[root@zfs] /mnt/storage/nocompression# dd if=temp.dat of=/dev/null bs=4M
51200+0 records in
51200+0 records out
214748364800 bytes transferred in 159.694223 secs (1344747235 bytes/sec)
With Compression:
[root@zfs] /mnt/storage# dd if=/dev/zero of=temp.dat bs=4M count=50k
51200+0 records in
51200+0 records out
214748364800 bytes transferred in 85.979721 secs (2497662969 bytes/sec)
[root@zfs] /mnt/storage# dd if=temp.dat of=/dev/null bs=4M
51200+0 records in
51200+0 records out
214748364800 bytes transferred in 47.818332 secs (4490921285 bytes/sec)
Ubuntu VMs that live on the storage pool, with ESXi mounting it via NFS:
One VM:
dd if=/dev/zero of=temp.dat bs=4M count=5k
5120+0 records in
5120+0 records out
21474836480 bytes (21 GB) copied, 225.22 s, 95.4 MB/s
dd if=temp.dat of=/dev/null bs=4M
5120+0 records in
5120+0 records out
21474836480 bytes (21 GB) copied, 85.0397 s, 253 MB/s
Two VMs (simultaneous):
Write:
dd if=/dev/zero of=temp.dat bs=4M count=5k
5120+0 records in
5120+0 records out
21474836480 bytes (21 GB) copied, 326.994 s, 65.7 MB/s
dd if=/dev/zero of=temp.dat bs=4M count=5k
5120+0 records in
5120+0 records out
21474836480 bytes (21 GB) copied, 334.963 s, 64.1 MB/s
Read:
dd if=temp.dat of=/dev/null bs=4M
5120+0 records in
5120+0 records out
21474836480 bytes (21 GB) copied, 158.938 s, 135 MB/s
dd if=temp.dat of=/dev/null bs=4M
5120+0 records in
5120+0 records out
21474836480 bytes (21 GB) copied, 155.774 s, 138 MB/s
Ubuntu VM that has the NFS mounted:
dd if=/dev/zero of=temp.dat bs=4M count=5k
5120+0 records in
5120+0 records out
21474836480 bytes (21 GB) copied, 136.16 s, 158 MB/s
dd if=temp.dat of=/dev/null bs=4M
5120+0 records in
5120+0 records out
21474836480 bytes (21 GB) copied, 68.4528 s, 314 MB/s
Here are the server specs:
FreeNAS 9.2.1.5
Xeon L5640
96GB Memory
24x 480gb Intel 320
Chelcio T520-SO-CR (dual 10G fiber)
I have this set up as Striped 4x raidz2 vdevs.
The benchmarks locally (both compressed and compression off) are pretty good, but when I spin up a VM on the same storage over NFS or even mount the storage over NFS directly on an Ubuntu VM, the speeds are a fraction of the local speeds.
I would love to figure out why the speed difference is so great and get this ZFS box rocking.
Thanks in advance!
Here are some of the benchmarks:
Local Benchmarks (dd)
Without Compression:
[root@zfs] /mnt/storage/nocompression# dd if=/dev/zero of=temp.dat bs=4M count=50k
51200+0 records in
51200+0 records out
214748364800 bytes transferred in 341.918976 secs (628067993 bytes/sec)
[root@zfs] /mnt/storage/nocompression# dd if=temp.dat of=/dev/null bs=4M
51200+0 records in
51200+0 records out
214748364800 bytes transferred in 159.694223 secs (1344747235 bytes/sec)
With Compression:
[root@zfs] /mnt/storage# dd if=/dev/zero of=temp.dat bs=4M count=50k
51200+0 records in
51200+0 records out
214748364800 bytes transferred in 85.979721 secs (2497662969 bytes/sec)
[root@zfs] /mnt/storage# dd if=temp.dat of=/dev/null bs=4M
51200+0 records in
51200+0 records out
214748364800 bytes transferred in 47.818332 secs (4490921285 bytes/sec)
Ubuntu VMs that live on the storage pool, with ESXi mounting it via NFS:
One VM:
dd if=/dev/zero of=temp.dat bs=4M count=5k
5120+0 records in
5120+0 records out
21474836480 bytes (21 GB) copied, 225.22 s, 95.4 MB/s
dd if=temp.dat of=/dev/null bs=4M
5120+0 records in
5120+0 records out
21474836480 bytes (21 GB) copied, 85.0397 s, 253 MB/s
Two VMs (simultaneous):
Write:
dd if=/dev/zero of=temp.dat bs=4M count=5k
5120+0 records in
5120+0 records out
21474836480 bytes (21 GB) copied, 326.994 s, 65.7 MB/s
dd if=/dev/zero of=temp.dat bs=4M count=5k
5120+0 records in
5120+0 records out
21474836480 bytes (21 GB) copied, 334.963 s, 64.1 MB/s
Read:
dd if=temp.dat of=/dev/null bs=4M
5120+0 records in
5120+0 records out
21474836480 bytes (21 GB) copied, 158.938 s, 135 MB/s
dd if=temp.dat of=/dev/null bs=4M
5120+0 records in
5120+0 records out
21474836480 bytes (21 GB) copied, 155.774 s, 138 MB/s
Ubuntu VM that has the NFS mounted:
dd if=/dev/zero of=temp.dat bs=4M count=5k
5120+0 records in
5120+0 records out
21474836480 bytes (21 GB) copied, 136.16 s, 158 MB/s
dd if=temp.dat of=/dev/null bs=4M
5120+0 records in
5120+0 records out
21474836480 bytes (21 GB) copied, 68.4528 s, 314 MB/s