freenas as shared storage and virtualization storage for KVM and vmware

[chetan003]

Hello

We would like to setup freenas shared storage which will connect via iscsi share within our datacenter. According to our testing with freenas we are going to setup as below. please help to check and provide us suggestion.

1.1TB SSD x 24 (raid z2) (we will going to use passthrough hardware raid controller )
2.one spare 1TB SSD for zil and L2arc for zpool.
3.dedicated zvol for each client that require storage space. As there is no such option to setup iscsi target size it will allocate according to size of zvol.
4.thin size zvol
5.LACP link aggregation between two 10G intel NIC
6.64GB RAM
7.Cpu dual hexa core

Questions
1.Please suggest best practice setting to get fast performance on zvol and stable freenas version
2.cache setting for zil and l2arc
3.how to assign size and resize iscsi target
4.best practice to setup multiple iscsi target
5.network setup for best bandwidth throughput and storage performance
6.email alert and freenas security.

 

[sretalla]

1TB SSD x 24 (raid z2)

This isn't the optimal setup for IO performance, but will an all-SSD array, you may not care too much... depends on how heavy your load will be, but Mirrored VDEVs is the conventional wisdom for iSCSI, not a single RAIDZ2 of 24 drives.

2.one spare 1TB SSD for zil and L2arc for zpool.

If you're using the same SSD for L2ARC/SLOG as you have in your pool, I would question what you're trying to improve? I think you don't need it unless your SLOG can outpace your pool drives. L2ARC is unlikely to help either in the case you're putting forward... much more RAM would be beneficial instead (will possibly change your Motherboard selection to go higher).

 

[HoneyBadger]

Hello Friend,

We would like to setup freenas shared storage which will connect via iscsi share within our datacenter. According to our testing with freenas we are going to setup as below. please help to check and provide us suggestion. sorry for my bad English

I have a few bits of feedback on the hardware build and some questions of my own. Sorry for length of this post. ;)

1. 1TB SSD x 24 (raid z2) (we will going to use passthrough hardware raid controller )
2. one spare 1TB SSD for zil and L2arc for zpool.
3. dedicated zvol for each client that require storage space. As there is no such option to setup iscsi target size it will allocate according to size of zvol.
4. thin size zvol
5. LACP link aggregation between two 10G intel NIC
6. 64GB RAM
7. Cpu dual hexa core

1. Strongly recommend a true HBA, not a passthrough/JBOD RAID controller. 24x SSD will drive a lot of IOPS and you want to maximize them. Something based on SAS3008 most likely. Ensure you have the right cables and backplane.

2. L2ARC will not be as helpful as you think since the rest of your vdevs are just as fast. Using a single SSD for SLOG will bottleneck your 24x SSDs to that write speed, and probably isn't the right model anyways. Keep the spare as a spare. I suggest you invest in a very fast NVMe SLOG device like an Optane P4800X given the scope of system here.

3. If you plan to present this to KVM and VMware, why not use a shared datastore? Both of those OSes handle clustering quite nicely and for VMware, FreeNAS supports VAAI primitives. You can create additional ZVOLs for raw devices or direct presentation to VMs if desired later.

4. Thin/sparse ZVOL is fine, make sure that your hypervisors properly pass SCSI UNMAP to free the space though. VMFS6 does automatically. KVM you may need to play with.

5. Do not use LACP with iSCSI - you will want to use MPIO. There is a section in the FreeNAS manual about setting this up, but the short version is "use two separate, non-overlapping subnets, ideally with two separate switches for hardware redundancy."

6. 64GB is a good start, more is always better. ;)

Questions
1. Please suggest best practice setting to get fast performance on zvol and stable freenas version
2. cache setting for zil and l2arc
3. how to assign size and resize iscsi target
4. best practice to setup multiple iscsi target
5. network setup for best bandwidth and storage performance
6. email alert and freenas security.

1. Mostly covered above, use FreeNAS 11.2 and do not take "beta" or "nightly" updates. Small mirrored boot SSD suggested.

2. Don't use L2ARC, get an NVMe SLOG, set sync=always on your iSCSI ZVOLs.

3. These are assigned sizes during the creation, make sure to check "sparse volume" if you want thin, and they can be resized in the "edit zvol" afterwards.

4. The documentation suggests associating only one extent per target. One portal with multiple IPs for MPIO.

5. As covered above, use MPIO, two separate subnets that don't overlap, two physical switches ideally. Make sure they are capable of full throughput on all ports simultaneously ("non-blocking") and have support for 802.3x Flow Control on the ports.

6. Set up scheduled SMART scans, pool scrubs, configure an email address for your alerts. Set a strong root/web admin password, consider using SSH keys, and don't port-forward anything to it from the Internet. Make the management IP on a secured subnet internally.

Now for my questions.

1. What make and model of SSD are you planning to use for the 24x 1TB vdevs?

2. Can you elaborate more on the planned storage architecture (shared datastores, VM disk locations, etc)?

 

[chetan003]

1. What make and model of SSD are you planning to use for the 24x 1TB vdevs?

2. Can you elaborate more on the planned storage architecture (shared datastores, VM disk locations, etc)?

I have a few bits of feedback on the hardware build and some questions of my own. Sorry for length of this post. ;)



1. Strongly recommend a true HBA, not a passthrough/JBOD RAID controller. 24x SSD will drive a lot of IOPS and you want to maximize them. Something based on SAS3008 most likely. Ensure you have the right cables and backplane.

2. L2ARC will not be as helpful as you think since the rest of your vdevs are just as fast. Using a single SSD for SLOG will bottleneck your 24x SSDs to that write speed, and probably isn't the right model anyways. Keep the spare as a spare. I suggest you invest in a very fast NVMe SLOG device like an Optane P4800X given the scope of system here.

3. If you plan to present this to KVM and VMware, why not use a shared datastore? Both of those OSes handle clustering quite nicely and for VMware, FreeNAS supports VAAI primitives. You can create additional ZVOLs for raw devices or direct presentation to VMs if desired later.

4. Thin/sparse ZVOL is fine, make sure that your hypervisors properly pass SCSI UNMAP to free the space though. VMFS6 does automatically. KVM you may need to play with.

5. Do not use LACP with iSCSI - you will want to use MPIO. There is a section in the FreeNAS manual about setting this up, but the short version is "use two separate, non-overlapping subnets, ideally with two separate switches for hardware redundancy."

6. 64GB is a good start, more is always better. ;)



1. Mostly covered above, use FreeNAS 11.2 and do not take "beta" or "nightly" updates. Small mirrored boot SSD suggested.

2. Don't use L2ARC, get an NVMe SLOG, set sync=always on your iSCSI ZVOLs.

3. These are assigned sizes during the creation, make sure to check "sparse volume" if you want thin, and they can be resized in the "edit zvol" afterwards.

4. The documentation suggests associating only one extent per target. One portal with multiple IPs for MPIO.

5. As covered above, use MPIO, two separate subnets that don't overlap, two physical switches ideally. Make sure they are capable of full throughput on all ports simultaneously ("non-blocking") and have support for 802.3x Flow Control on the ports.

6. Set up scheduled SMART scans, pool scrubs, configure an email address for your alerts. Set a strong root/web admin password, consider using SSH keys, and don't port-forward anything to it from the Internet. Make the management IP on a secured subnet internally.

Now for my questions.

1. What make and model of SSD are you planning to use for the 24x 1TB vdevs?
Ans:We are going to use Crucial/kingston SSD drive please suggest which make and model we can use for our freenas storage
2. Can you elaborate more on the planned storage architecture (shared datastores, VM disk locations, etc)?

Ans:We are going to use iscsi thin SSD and Sata storage on freenas. We would like to use this SSD and SATA storage to our vmware and KVM HA cluster.
In addtion, we would like to create iscsi LUN for clients that, require external SSD/sata storage. The clients server and our freenas storage will be hosted on same DC so connection between iscsi client and freenas target will connected over private 1G link vlan.