MPIO load balancing issue with Citrix Xenserver 5.6 FP1 and FreeNas 8.0.2

[CBNAS]

Hello,

I followed a How to guide created by Diego Morato for connecting iSCSI with MPIO to Citrix Xenserver.
By the way, this was a great document!
Although it was geared towards version 7, I was able to adapt it to 8.0.2.

I have two GigE links cross-connected from the FreeNAS 8.0.2 server to the Citrix Xenserver.
The Citrix Xenserver is using two physical interfaces that are configured as management interfaces.

I added the configuration to the multipath.conf, and enabled multipathing on the server.
It states that I have 2 paths of 2, and 2 active iSCSI connections to the FreeNAS.
I am not using file extents, but device extents.

Here is what Xenserver states when I request multipath -ll:

330000000218d8a68 dm-0 FreeBSD,iSCSI Disk
[size=7.3T][features=0][hwhandler=0][rw]
\_ round-robin 0 [prio=1][active]
\_ 10:0:0:0 sde 8:64 [active][ready]
\_ round-robin 0 [prio=1][enabled]
\_ 9:0:0:0 sdd 8:48 [active][ready]

Here is my istgt.conf: (FreeNAS 8.0.2)

Global section
[Global]
NodeBase "iqn.2011-03.example.org.istgt"
PidFile "/var/run/istgt.pid"
AuthFile "/usr/local/etc/istgt/auth.conf"
MediaDirectory /mnt
Timeout 30
NopInInterval 20
MaxR2T 255
DiscoveryAuthMethod None
MaxSessions 16
MaxConnections 8
FirstBurstLength 262144
MaxBurstLength 2097152
MaxRecvDataSegmentLength 262144
MaxOutstandingR2T 64
DefaultTime2Wait 2
DefaultTime2Retain 60

[UnitControl]

# PortalGroup section
[PortalGroup1]
Portal DA1 10.0.1.2:3260
Portal DA1 10.0.2.2:3260

# InitiatorGroup section
[InitiatorGroup1]
InitiatorName "ALL"
Netmask ALL

# LogicalUnit section
[LogicalUnit1]
TargetName "venust5c-1"
Mapping PortalGroup1 InitiatorGroup1
AuthMethod None
UseDigest Auto
ReadOnly No
UnitType Disk
UnitInquiry "FreeBSD" "iSCSI Disk" "0123" "000423a8f7be00"
UnitOnline yes
BlockLength 512
QueueDepth 32
LUN0 Storage /dev/label/extent_ada0 auto
LUN0 Option Serial 000423a8f7be000

Here is my multipath.conf addition: (Xenserver 5.6 FP1)

}
device {
vendor"FreeBSD"
product"iSCSI DISK"
path_selector "round-robin 0"
path_grouping_policy multibus
rr_min_io4
}


My issue is that I am not getting equal traffic across both network interfaces?
In fact, I am only getting disk traffic across one of the GIGE interfaces.
The second interface does show activity, but very little.

I have attained up to 600 Mbits/sec across em1, but em2 shows occasional traffic between 10 -15k bits/second.

On the Xenserver, the two dedicated iSCSI management NICs are Intel GIGE on the same PCI controller, and on the FreeNAS server, one is a Broadcom GigE on the motherboard, and the other is on a Intel GigE PCI controller.

Has anyone had success configuring true load balanced traffic from a Xenserver Host to FreeNAS using MBIO?
I am not using iSCSI from the guest VM, but from the core Xenserver host to supply storage to the VMs.

If so, can you guide me?

Not sure what other information you would need to verify my config on both the Xenserver and FreeNAS.
Just tell me what commands you need me to run, and I will post them for your review.

Thanks Much in advance!

 

[ProtoSD]

There have been a few other threads posted with similar results. I think this might be a "feature" of FreeBSD. I'm not sure if it's working in 9.0 Release either, but I just thought I'd let you know you're not the only one experiencing this.

 

[CBNAS]

Thanks for the quick response

There have been a few other threads posted with similar results. I think this might be a "feature" of FreeBSD. I'm not sure if it's working in 9.0 Release either, but I just thought I'd let you know you're not the only one experiencing this.

Thanks for the feedback, it just seems weird that I am not seeing the traffic on both interfaces.
I wish this made me feel better.
I have a production VM on the iSCSI.
I do know that failover works on the network, just not load balancing.

Chris

 

[louisk]

Have you tried playing with link aggregation on FreeNAS and the switch to see if that makes a difference? I'm not aware of FreeBSD doing any round-robin networking, and w/o lagg, it's going to prefer one interface for traffic (think about L2 domains and this will make sense).

 

[koskesh007]

I know this has been a while but check the location of the brackets. the end of the file should look something like this:
device {
vendor "FreeBSD"
product "iSCSI Disk"
path_grouping_policy multibus
path_selector "round-robin 0"
rr_min_io 4
}

} (this last bracket closes the devices group)

And it seems path_grouping_policy must be defined before path_selector

With this, XenServer 6.0.2 splits the load over all iSCSI connections, but I noticed a huge performance hit on read speed when it's doing round robin.

See this for more details on my issue:
http://forums.freenas.org/showthrea...ultipathing-and-round-robin&p=32987#post32987

 

[Brad303]

My issue is that I am not getting equal traffic across both network interfaces?
In fact, I am only getting disk traffic across one of the GIGE interfaces.
The second interface does show activity, but very little.

I have attained up to 600 Mbits/sec across em1, but em2 shows occasional traffic between 10 -15k bits/second.

In case you haven't yet solved this issue, or in case someone else stumbles across this old thread, I suspect the issue has to do with your subnets. Make sure that each interface on both machines is on a separate pair of subnets. For example, 10.0.1/24 for one interface on the Xen and one interface on FreeNAS, then 10.0.2/24 for the other interface on Xen and the other interface on FreeNAS.

FreeBSD will send packets out the first interface is comes to on the destination subnet.

However, even if you do get it to properly round-robin, don't be surprised if you're not getting 2GbE bandwidth. I'm still getting 1GbE over my 2 links and haven't yet figured out why.

 

[louisk]

In case you haven't yet solved this issue, or in case someone else stumbles across this old thread, I suspect the issue has to do with your subnets. Make sure that each interface on both machines is on a separate pair of subnets. For example, 10.0.1/24 for one interface on the Xen and one interface on FreeNAS, then 10.0.2/24 for the other interface on Xen and the other interface on FreeNAS.

FreeBSD will send packets out the first interface is comes to on the destination subnet.

However, even if you do get it to properly round-robin, don't be surprised if you're not getting 2GbE bandwidth. I'm still getting 1GbE over my 2 links and haven't yet figured out why.

1) FreeBSD doesn't do round-robin
2) Even if you configure LAGG, its only going to give you a max of (in this case) 1G per host. LACP works by allowing additional paths to additional hosts, it will never give you more than 1 path's worth of bandwidth to a single host.

 

[cyberjock]

Even if you configure LAGG, its only going to give you a max of (in this case) 1G per host. LACP works by allowing additional paths to additional hosts, it will never give you more than 1 path's worth of bandwidth to a single host.

Exactly! You cannot get more than 1Gb per link. This is because of how LAGG is designed. Yes, I learned this lesson the hard way. ;)

 

[Brad303]

1) FreeBSD doesn't do round-robin

I should clarify. By round-robin, I don't mean round-robin LAGG. Rather, that I have two interfaces on separate subnets in a single portal on FreeNAS and ESXI 4.1 configured with two NICs, on the same two subnets, bound to the software iSCSI adapter and multi-pathing enabled using Round Robin.

This configuration is working right now. It's just not giving me more than 1Gbps. It's definitely using both links on the FreeNAS side.

2) Even if you configure LAGG, its only going to give you a max of (in this case) 1G per host. LACP works by allowing additional paths to additional hosts, it will never give you more than 1 path's worth of bandwidth to a single host.

Not entirely true, especially if you are using multiple interfaces on said host, since LAGG uses an IP (or IP+port) hash to determine which link to use.

 

[cyberjock]

Not entirely true, especially if you are using multiple interfaces on said host, since LAGG uses an IP (or IP+port) hash to determine which link to use.

Actually, that statement is both true AND false. Its like Schrödinger's cat, both dead and alive :)

I emailed a Cisco network engineer about 2 years ago when I had spent about $500 on network switches and NICs to setup LAGG at home when it didn't work like I had hoped. I thought it would be just like owning 2Gb NICs in my house. He said that LAGG can use any kind of "decision making" algorithm it wants. Obviously the NIC(and therefore the network port that is used to send data) is controlled by the computer sending the data, but receiving is much fuzzier. I had noticed that when I transferred data from Computer A to Server A one port would be chosen on the Server's end with regards to receiving data. The short answer is that the port that receives the data can be chosen by anything the network switch wants. Some do round robin taking turns, some go by IP, some go by port(although he said this one is REALLY not recommended because if a server is a file server using CIFS you can expect virtually all of your traffic to be on one single port), others do MAC address, and others use a RNG. There is no "preferred" method because each method has its own situations that make it superior/inferior to the others.

Ultimately in my tests with my 2x 24 port switches I was unable to ever send or receive data via FTP and simultaneously use CIFS and achieve more than 1Gb/sec. CIFS might hit 90MB/sec, but as soon as I started an FTP transfer to the same server CIFS would slow down and my FTP transfer and CIFS transfer would immediately begin to share a single 1Gb/sec link. Needless to say I was quite disappointed because I had a machine setup to reencode blu-rays to DVDs for home videos for friends/family and it wasn't quite working out. My performance limitation was always the network, and I really thought I could increase performance with an upgrade to using LAGG.

I did get my setup to work better for me by cheating. I setup 2 different subnets on the server and 1 workstation. Then I went ahead and used both IPs of the workstation network to connect to each applicable subnet. I could now transfer 2Gb/sec to/from the server as long as I was using both subnets. I'm not sure if this would work in FreeNAS though.

 

[Brad303]

I dunno about your switches, but both my Dells and FreeNAS support LAGG method. In the case of ESXi, it only supports etherchannel (IP hash). The switch has logic that determines which link to use based on usage and while that's not going to ever give 100% of both links, it'll still use both relatively fairly.

Yes, you are correct that LAGG can use other methods to determine which link to use, but there's a fair amount of detail out there on how the IP hashing mechanism itself works.

 

[cyberjock]

Please, post us a screenshot of you copying data to/from the server using LAGG with a single client exceeding 1Gb/sec. That goes against EVERYTHING I've read/heard/witnessed.

IP hashing is ONE of the possible candidates for how LAGG works. My previous post explained this.

 

[Brad303]

First of all, you must select which method your LAGG will use on both ends. For example, if you have IP Hash selected on your host and MAC hash on your switch, it won't work. It's not like LAGG decides willy-nilly which method to use.

However, regardless of which method it uses, it uses that method to determine which single link to use for that traffic. That's the bit that you're referring to.

So, given that, if you have a LAGG to your FreeNAS and have two separate client machines, all using 1GbE, what's the theoretical total bandwidth that you would expect to get?

 

[cyberjock]

Theoretically the total COULD be 2Gb, but it COULD be 1Gb.

That is the problem. You can have a 4 port LAGG connection from your server to a switch, and another 4port LAGG connection from your switch to your desktop. What's your theoretical limit from the server to your desktop? 1Gb/sec.

Now add another desktop with a 4port LAGG connection. Whats the theoretical maximum for the second desktop? 1Gb/sec.

What's the theoretical maximum data your server will be able to dish out to your entire network combined? Either 1Gb/sec OR 2Gb/sec.

If you get screwed and both desktops get assigned the same port, they'll compete for that 1Gb of bandwidth. Otherwise, they'll work independent of each other giving you a theoretical 2Gb/sec(1Gb/sec to each desktop).

For my network, up to 4 machines on and 2x1Gb NICs to my LAN I did better than LAGG. I setup the server's dual port NIC as 2 different IPs. The machines I rarely use are on 1, and the machines I use regularly(but never at the same time) are on the other. If I hadn't done it this way it was possible that my HTPC could not stream movies while I was doing heavy reads or writes to the server. See how I fix it?

I remember someone saying that for FreeNAS if you use LAGG you are still limited to 1 port for outgoing data, the static route. I'm not sure if I have that right, or if the post was even I ccorrect. I abandoned LAGG as a useful performance enhancer for my home network 2 years ago as I explained above. I really wanted 2Gb/sec to my desktop. But I learned that I couldn't get it to work right(at least.. as I defined it in my head). I asked friends in IT and eventually emailed Cisco asking about it since I was using a 24 port Cisco switch at the time. That's how I learned that LAGG is great if you have tons of workstations. You are distributing the load from many machines to 4 ports, so it's reasonable to expect somewhat even distribution. Plus most users won't be utilizing massive bandwidth at the same time in a corporate environment. If you have only a few machines things can get messy. There's no guarantee of any kind of distribution at all. Theoretically you have up to a 1/x chance that x number of machines on your network will share the same port. That's not good odds when I REALLY want my 2 main desktops to have fast speeds without killing other people. So by designing my server IPs to prevent any kind of collision with network speed limits I have made my network better.