Poor perfomance with ISCSI

[jsawyer77]

I am new to free NAS and have loaded version 11.2 to do some testing. Our iscsi perfomance is much worse then what we see on VM's running on an NFS share.With NFS we will see rights close to 580 - 650 MB on systems running iscsi we see write speeds in the 3 to 9MB.

dd if=/dev/zero of=/tmp/output conv=fdatasync bs=512k count=1k; rm -f /tmp/output
1024+0 records in
1024+0 records out
536870912 bytes (537 MB, 512 MiB) copied, 175.007 s, 3.1 MB/s

Any ideas why we maybe seeing such a drastic diffrence in write speed between the two?

 

[Chris Moore]

Updated Forum Rules 12/5/18
https://forums.freenas.org/index.php?threads/updated-forum-rules-12-5-18.45124/

Hardware Requirements
http://www.freenas.org/hardware-requirements/

Did you read the manual?
http://doc.freenas.org/11/freenas.html

Slideshow explaining VDev, zpool, ZIL and L2ARC
https://forums.freenas.org/index.ph...ning-vdev-zpool-zil-and-l2arc-for-noobs.7775/

Terminology and Abbreviations Primer
https://forums.freenas.org/index.php?threads/terminology-and-abbreviations-primer.28174/

 

[jsawyer77]

Here are the specs on the system.

Mother board: supermicro SYS-1018R-WC0R
CPU is an intell xeon: Intel(R) Xeon(R) CPU E5-2603 v4 @ 1.70GHz
32 GB ECC RAM
8 7200 2 GB drives
one 256GB intel ssd drive I can use for zill or cache
2 64Gb usb drives setup in a mirror to boot

I currently have the pool set up with the all the drives in a raidZ2

Like I said NFS seems to be ok with the read and write but the iscsi is no good

I have looked over these documents and will check out the ones I have not.

 

[Chris Moore]

All of my comments are based on FreeNAS and ZFS, if you were using this hardware with a different operating system, these comments might not apply.

1.70GHz

That is a really, REALLY slow processor. Six core, six thread, it doesn't even have hyper-threading. I wouldn't even try to use that. YMMV.

32 GB ECC RAM

ZFS uses RAM as cache for IO. This is probably not enough. I have 64GB on each of my home system and 256GB in the systems I manage for work. I think this is going to need to be revised.

I currently have the pool set up with the all the drives in a raidZ2

That is going to inhibit performance, no matter how you share the pool. It is not exact, but the performance (IOPS) of a vdev are roughly equal to the slowest single disk that makes the vdev, so with 8 drives in one vdev, you get about the performance of a single drive for the entire vdev and your pool only has ONE vdev. The recommended configuration for iSCSI (or any virtualization) is to have as many vdevs as possible to boost IOPS. If you can only accomodate 8 drives, which is not many, probably not enough, that would indicate using 4 mirror vdevs in the pool which should get you around 4 times as much performance as you have now. Personally, I wouldn't consider doing iSCSI on FreeNAS with less than 24 spinning disks. You might get along fine with 8 if they are all SSDs. I built a 16 drive (8 mirror vdevs) pool last year for iSCSI using spinning disks and was able to get speeds around what you would expect from a SATA SSD. Faster drives would have boosted the performance.

8 7200 2 GB drives

I am guessing that you mean 2 TB drives, 2 GB is pretty small, but if you do a pool of mirror vdevs that will give you a theoretical capacity of 8 TB, but with iSCSI on ZFS, you shouldn't fill the pool above 50% because of the copy on write nature of the file system. That limits you to only 4 TB.
Is that enough capacity?

Like I said NFS seems to be ok with the read and write but the iscsi is no good

With any system, bue FreeNAS especially, iSCSI brings a host of limitations that NFS does not have, largely because it is not a block storage technology were iSCSI is. You might want to use NFS unless there is a compelling reason not to use it.

 

[jsawyer77]

Thank you for the information. This system is to support my lab so I dont need anything out of this world and NFS should be ok. I will set up the mirrors and see how it goes. Should I add the SSD as a cache drive?



Regards,
Josh

 

[Chris Moore]

Should I add the SSD as a cache drive?

If you are doing synchronous (sync) writes to the pool, you will want what is called SLOG (Separate LOG) to accelerate the write operation.
Here are some links with discussion on the topic.

The ZFS ZIL and SLOG Demystified
http://www.freenas.org/blog/zfs-zil-and-slog-demystified/

Testing the benefits of SLOG using a RAM disk!
https://forums.freenas.org/index.ph...s-of-slog-using-a-ram-disk.56561/#post-396630

SLOG benchmarking and finding the best SLOG
https://forums.freenas.org/index.ph...-and-finding-the-best-slog.63521/#post-454773