iSCSI and ESXi best network config

[H0mez]

I have a freenas box with 5 NICS, and a ESXI 5.1 server with 4 nics. Currently I have the ESXI box setup with two vSwitches, 1 with 2 NICS for iSCSI and 1 with the other 2 for admin/vmotion. My freenas just has 5 nics all with different IPS, I tried setting them up with different subnets, but that doesn't seem to work with ESXI

I have MPIO setup on the ESXI box, but I'm not seeing the performance I'm expecting, about 50mb read and write. If I connect the iSCSI to my Win 7 im seeing 110mb on both.

Also my the TX on the freenas box looks to be coming from only 1 NIC.

 

[jgreco]

Most sane operating systems route outbound traffic based on the destination IP address. You are assuming that it will ALSO take into consideration the source address, and then maybe magically prefer that particular interface. It doesn't. You have what we call "broken UNIX network design." That's why your traffic "looks to be coming from only 1 NIC." Because it is, that's the interface FreeBSD is using for traffic to that subnet.

If you want multiple interfaces on a single network, you use link aggregation (results in a single virtual interface with multiple physical ports). If you want multiple IP addresses on a single network, you configure the interface on the network for multiple IP addresses ("IP aliases").

ESXi absolutely works fsckin' awesome with multiple subnets, but it is not beginner-level configuration. FreeNAS is a little easier to configure for multiple subnets.

You are best off with multiple subnets and MPIO because link aggregation doesn't work the way most people think it does.

 

[H0mez]

I guess I must be setting up the ESXi side incorrectly to get MPIO to work. Do I need a vswitch per subnet or something? I'm trying to find some examples through google, but my googlefu must be off, ill keep digging.

When I had Freenas adapters setup with different subnets, esxi was only seeing the connection on the 192.168.1.* subnet, but not the 192.168.2.*, 192.168.3.* up to 5.

 

[jgreco]

What you want is VMware's iSCSI SAN Configuration Guide.

 

[Setius]

Sounds like you don't have something configured right for iSCSI MPIO. Once you have ESXi and FreeNAS setup correctly, you will see active IO path on each subnet. My home lab is ESXi 5.1 and FreeNAS 8.3 connected via iSCSI MPIO with 4x1Gb links and it works really well for me. Also, look into tuning ESXi MPIO round robin load balancing IO packet setting. This setting will very based on your environment, but most people set the RR IO to 1 instead of default 1000 and see better performance results.

Pardon my links, I'm on a tablet and the linking is not working. Reading FreeNAS wiki on iSCSI and VMware iSCSI MPIO config guide should be all you need to get this working correctly.

http://www.vmware.com/files/pdf/tec...configuration-software-iSCSI-port-binding.pdf

http://doc.freenas.org/index.php/ISCSI

 

[H0mez]

I have the following on the esxi.

But if you look here, it doesnt seem like the traffic is flowing the same through all NICS

 

[survive]

Hi HOmez,

Don't forget to configure the multi-pathing policy on each of your datastores.

Go to the Config tab & select storage to get a list of your datastores. About half-way down the screen on the right of the "Datastore Details" section there is a Properties link in blue. Click that and in the screen that pops up there is a "Manage Paths..." button on the bottom right. Click that and you can change the path selection to "Round Robin". That will allow your iscsi traffic to be spread acroos all the links you have to your storage.

-Will

 

[H0mez]

Yep, I always have that setup. Still not sure why im not getting I/O from the box

 

[Credo]

Sounds like you don't have something configured right for iSCSI MPIO. Once you have ESXi and FreeNAS setup correctly, you will see active IO path on each subnet. My home lab is ESXi 5.1 and FreeNAS 8.3 connected via iSCSI MPIO with 4x1Gb links and it works really well for me. Also, look into tuning ESXi MPIO round robin load balancing IO packet setting. This setting will very based on your environment, but most people set the RR IO to 1 instead of default 1000 and see better performance results.

Pardon my links, I'm on a tablet and the linking is not working. Reading FreeNAS wiki on iSCSI and VMware iSCSI MPIO config guide should be all you need to get this working correctly.

http://www.vmware.com/files/pdf/tec...configuration-software-iSCSI-port-binding.pdf

http://doc.freenas.org/index.php/ISCSI

I know this is an older post, but I'm still confused. The documents above somewhat contradict eachother. Both port groups on the ESXi host are in the same subnet (Above document page 7: iSCSI-PG1=192.168.100.1, iSCSI-PG2=192.168.100.2), but the FreeNAS doc stipulates every interface needs to be on a different subnet, to which you aggregate them all through Portals. would we still not be limited to 1 Gb Ethernet port per subnet? The other 3 ports would each be listening on different subnets, and essentially unused.

If I have 3 ESXi Hosts all connected to the FreeNAS device using a single port, there's going to be substantial bottlenecking. Am I missing something?

 

[jgreco]

Putting multiple interfaces on a single subnet is a broken IP design. Storage vendors in particular like to do it because the people who configure storage often aren't competent network engineers, and it is possible to make specific cases work. Unfortunately, most UNIX variants don't work right in such a busted environment, at least not without some competent network engineering to help it all out, which is beyond the scope of the automatic systems in an appliance.

 

[brady5]

I've run up against this same scenario myself without any solutions so far. Best I can gather from reading the docs is you might need a physical nic dedicated for each network in your esxi hosts. I haven't tried that yet because all my physical nics are being used.

My attempts at creating vmkernel ports inside the same vswitch for the other networks (Other nics in FreeNAS) has not worked.

Anyone else ever get this setup working?

Side Note: I have a netgear nas that has 4 nics (all on the same network). MPIO with esxi works great with that box. I currently have 4 paths per host and throughput is great. Getting FreeNAS to do the same is proving to be a pain in the arse...

 

[jgreco]

FreeNAS isn't going to do that. Get over it, fix your network topology, and move on.

 

[brady5]

10-4 mr. grinch... Here are some other helpful links on the subject to help other happy Whoville citizens:

http://forums.freenas.org/index.php...via-iscsi-having-some-issues.8557/#post-35821
http://www.epubbud.com/read.php?g=8MVD69ZN&tocp=17#chapter17
https://pubs.vmware.com/vsphere-51/index.jsp?topic=/com.vmware.vsphere.storage.doc/GUID-0D31125F-DC9D-475B-BC3D-A3E131251642.html
http://en.wikipedia.org/wiki/Grinch
Merry Christmas!

2 May 2014 UPDATE:
I had to reset the nics in my freenas box from the console. When I did this I noticed both adapters pulled DHCP from the same network. I set them both with static IP's on the same network FROM THE CONSOLE and FreeNAS didn't complain as it did in the web gui. MPIO happily running with ESXi. All on the same network. May not be by the book Mr. Grinch but it is working without causing any issues on the network.

I would be interested to hear from others why this is a bad idea and why I will go to h**l for doing it this way?