2 NICs - Force NAS traffic through one NIC. Leave other for management.

Status
Not open for further replies.

tmstone835

Cadet
Joined
Aug 8, 2011
Messages
5
Hello,

I have a 2-CPU system with 16GB RAM and an 11 drive RAID-5 array with a hot spare. The system came with a single 100Mb NIC and two 1Gb NICs. My intention was to use the single 100Mb NIC as a management interface with all of the SAN/NAS traffic passing through the gigabit network cards. The network adapters are all on the same LAN segment with fixed IP addresses and separate DNS names.

My FreeNAS version is FreeNAS-8.0.4-RELEASE-p2-x64 (11367).

I brought the system up with the 100Mb NIC configured first. I then added one of the gigabit interfaces without a problem. Before integrating the second gigabit NIC, I installed it into my production environment and migrated data to the storage. At first, it seemed to work great until I noticed poor performance with many users attached. I configured the nightly system reports and found out that the inbound traffic to the NAS is passing through the gigabit NIC and the outbound traffic is returning via the 100Mb NIC. I did not aggregate the NICs together so I am surprised that this is happening. It looks like some kind of automatic load balancing on the FreeNAS system but there doesn't seem to be any options for modifying this behavior.

Here is the network portion of the nightly report so you can see what my system is doing. fxp0 is 100Mb and bge0 is 1Gb.

Network interface status:
Name Mtu Network Address Ipkts Ierrs Idrop Opkts Oerrs Coll
fxp0 1500 <Link#1> 00:e0:81:43:80:d6 142418612 0 0 883709363 0 0
fxp0 1500 10.0.0.0 bousan01 80740226 - - 879887366 - -
bge0 1500 <Link#2> 00:e0:81:43:80:e4 920816345 0 0 112830 0 0
bge0 1500 10.0.0.0 bousan01a 859070979 - - 0 - -
bge1* 1500 <Link#3> 00:e0:81:43:80:e5 0 0 0 0 0 0
lo0 16384 <Link#4> 42 0 0 42 0 0
lo0 16384 fe80:4::1 fe80:4::1 0 - - 0 - -
lo0 16384 localhost ::1 18 - - 18 - -
lo0 16384 your-net localhost 6 - - 24 - -

Does anyone know how to force all of the data traffic through the gigabit interface? Do I have to get rid of the 100Mb interface?

The simple solution would be to disable the 100Mb interface but I like to make things difficult.

Any help would be appreciated.
 

praecorloth

Contributor
Joined
Jun 2, 2011
Messages
159
I'm going to take a guess here. My guess is that since the 100Mb NIC was the first one to be set up on the 10.0.0.0 network, any traffic sent to 10.0.0.0 is just going to be sent out of that NIC.

I know it's possible to have a machine work properly with multiple connections to the same network. I don't know how. While I applaud your desire to keep the main connection as clean as possible for the users, you may just want to be rid of that 100Mb NIC for the sake of convenience.

Edit:
Yeah, if you do netstat -r I bet you'll see 10.0.0.0 is tied to fxp0.
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
What I did was setup one controller to use DHCP and the other static IP. Then have the clients access the server via the network name and use the IP address for management. The controller with the static IP will NOT resolve for the clients but you can access it yourself. Essentially if you don't give out the IP then nobody will know it's there unless they scan the IP.
 

joernc

Cadet
Joined
Nov 26, 2012
Messages
3
I have a similar setup for testing: One management interface and one data interface, for the sake of simplicity on the same network.

> If you're using iSCSI, you can create a portal as described here: http://doc.freenas.org/index.php/ISCSI#Portals.

That's exactly what I did. And the initiator knows only about one of the interfaces. Yet I see from the report graphs, that both interfaces get identical traffic when I run a benchmark. So either binding a portal to a single interface does not work, or the reporting functionality is broken...
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
for the sake of simplicity on the same network.

Well, there's your problem, right there. Dat ain't simple because it don't work datz way.

Seriously, don't try to make this work. Multiple interfaces on the same network do not work the way you would guess that they do. praecorloth's message above is approximately correct. FreeBSD picks an output interface for traffic based on the destination IP, which gets looked up in the routing table, and traffic is then shot out the interface that the system has associated with that destination route. It does not then do another lookup to see if there are multiple interfaces on that network; multiple interfaces on a network should be handled via link aggregation.
 

joernc

Cadet
Joined
Nov 26, 2012
Messages
3
FreeBSD picks an output interface for traffic based on the destination IP, which gets looked up in the routing table, and traffic is then shot out the interface that the system has associated with that destination route. It does not then do another lookup to see if there are multiple interfaces on that network; multiple interfaces on a network should be handled via link aggregation.

Thanks for the clarification. Indeed, the Portal configuration only asks for a *listening* interface. I have no idea how the iSCSI implementation of FreeBSD actually works, but if it allows binding to a specific interface for incoming and outgoing traffic, it would be great if FreeNAS could allow this configuration via the GUI.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
Has nothing to do with the iSCSI implementation, which is basically a userland application like "fetch" or "ftp". The "listening interface" you're seeing doesn't have any correlation to anything physical, it wants to know if you want to just do a wildcard bind (0.0.0.0) or bind to some specific IP interface, which coincidentally also often happens to be an address assigned to an ethernet adapter, but not always. There's an entire networking stack between the iSCSI target daemon and the ethernet interfaces, and that stack is pretty complicated, very powerful, and designed for massive performance. It's also not something written by the FreeNAS developers - it's part of FreeBSD. It works the way I said, and there's years of development effort that got it there. There are several possible ways to get FreeBSD to do something vaguely like what you want, but it would probably require some redesign of how FreeNAS was structured, so my best guess is that the FreeNAS developers are likely to tell you to fix your broken network.
 

joernc

Cadet
Joined
Nov 26, 2012
Messages
3
Has nothing to do with the iSCSI implementation, which is basically a userland application like "fetch" or "ftp".

Exactly. And as such, the iSCSI implementation has the freedom to bind to a specific interface (i.e. IP address) - or not. It is just another TCP application. If the implementers of iSCSI on FreeBSD did not allow this freedom, it is obvious that FreeNAS can do nothing about it.

Please don't get me wrong: I just wanted to clarify the situation, I am not blaming anyone for anything. Just as much as my network is not broken, I just decided to go with a specific simple setup for testing purposes. In a productive environment I would have used two different networks anyway.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
No, you're *really* not understanding. In the UNIX userland, you cannot "bind to a specific interface" and a specific interface is NOT "i.e. IP address". You can have an application bind to an IP address, which means that the application will receive packets sent to that IP address, and packets sent from that socket will originate from that IP address.

The whole point is that the underlying network stack does not implement applications binding to an IP address by binding it to a physical ethernet card. The underlying network stack has made certain assumptions about how rational people design and deploy networks, and your network does not meet those expectations, so things aren't working the way you expect.

Most UNIX network administrators would consider your network to be broken and the root problem here. I certainly do, but I will also say that I can see why you might have done what you did, and why you thought it would work. You were wrong, but it's because you expect the abstraction layer to map exactly onto the physical interfaces. It doesn't. It often ends up being a close approximation, but there are just some things that you cannot and/or should not do.
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
I think the confusion is because Windows actually DOES do exactly what the OP wants. Windows and its infinite wisdom will let you bind to the iP address, even if it has 2+ NICs sharing the same network subnet,IP, etc.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
Ugh, there's the whole terminology thing, "bind to the IP address" again. I'm assuming what you mean is that when you bind to an IP address that's assigned to a specific card, traffic goes in/out that card.

To expand for the OP: You can certainly "bind to an IP address" in UNIX but in FreeBSD it's an abstraction that makes no guarantees as to the physical handling of the traffic. This is really useful in many environments for lots of stuff. For example, you can get network redundancy for a server using OSPF by putting your service addresses on a loopback interface, and then advertising that into your interior gateway protocol using OSPF or some other IGP. The address that all external traffic uses isn't configured on ANY ethernet interface. Userland applications "bind to the IP address" just like it was an alias (or the primary address) on an ethernet interface... anyways, point is, application level binding is pretty much not closely coupled to physical interfaces. FreeBSD has a rich set of network interfaces that it supports, including point to point links (PPP, SLIP, parallel port cable), ATM, etc., and the networking subsystem presents it all as an abstraction to the userland. So of course since a lot of the IP configuration is driven by what's defined for physical interfaces, this leads to operational and terminology confusion.

Basically, for the issue at hand, there are two key bits:

Input traffic from an Ethernet network to a host is controlled by ARP. The ARP subsystem publishes MAC addresses in response to requests from an ethernet network, and this happens infrequently (meaning far less than once a second). ARP controls packet ingress. The system ARP table maintains a list of learned and published addresses, and when an ARP packet is received, the system compares it to the system's interfaces and responds with the MAC address of the interface. Now this process works pretty much the way the OP would expect, but it can be subverted. For example, if I have a server with two interfaces, em0=192.168.1.1/24 and em1=192.168.1.2/24, and I set a client's ARP table with a manual entry for 192.168.1.1 pointing at the MAC address of server's em1, the traffic for 192.168.1.1 from client to server enters server on em1. And everything works fine. The UNIX networking layer doesn't think this odd or anything, even if you have a userland app that is bound to 192.168.1.1, it all works.

Output traffic to an ethernet network is controlled by the routing table, and the routing table is keyed by destination IP address. Basically when you do a "route get ip.ad.dr.ess" the system does a routing table lookup, similar to what happens when a packet is output. The source IP address isn't considered because IP routing happens based on destination. So as long as the routing code picks AN interface that's on the network in question, packets will be delivered, and that's pretty reasonable.

If you want to have multiple interfaces on a network, you should use link aggregation.

If you want to have multiple IP addresses on a network, you should define ifconfig aliases.

You can do other things, but then you're fighting the way it was designed, and then it may not work as you expect.
 
Status
Not open for further replies.
Top