Register for the iXsystems Community to get an ad-free experience and exclusive discounts in our eBay Store.

Dual Ethernet without Link Aggregation

jgreco

Resident Grinch
Moderator
Joined
May 29, 2011
Messages
12,826
Multiple physical network interfaces on the same subnet may cause routing issues

the key word in this document is ROUTING. What is routing? Routing is needed when you want to access another subnet.
No, it is needed when you want access to ANY subnet, including a directly connected one. Routing is the process of selecting an appropriate interface out which to pass a packet, and, potentially, also a next-hop.

RFC1812, 2.2.3, "Forwarding an IP datagram generally requires the router to choose the address and relevant interface of the next-hop router or (for the final hop) the destination host. This choice, called relaying or forwarding depends upon a route database within the router. The route database is also called a routing table or forwarding table."

See that? "or (for the final hop) the destination host". You need the routing table to get to that too. Want to try a neat trick? Configure up an IPv4 interface, then issue a "route delete" for the directly connected network. You will lose access to the directly connected network.

The rest of your message is similarly off-kilter, derived from your basic initial misunderstanding. There's a massive difference between things kinda-working because you haven't done something sufficiently broken for it to actually break disastrously, and something that's being used the way it is designed to be.

This is a PERFECTLY valid network configuration; but there is a challenge.
It is an invalid network configuration. That it works is a testament to how resilient IP networking is and how well-written the BSD IP stack is.

It's not an "issue" or a "problem", but a challenge.
It is not an issue or a problem OR a challenge. It's just broken. Don't do it. It won't work the way the average person might wish. Modern standards allow for the use of link aggregation to accomplish this. LACP works better than this backwards strategy of trying to get multiple physical interfaces configured as logical interfaces on a single network.

Now, please note, the article you linked does include the following:

Now there will invariably be some people who will say ... "WRONG! I know that multiple interfaces are legal!" At which we come to an impasse about what kind-of works versus what actually happens.

I don't actually care to debate with you why you feel multiple non-LACP interfaces "should" work a certain way. I don't actually care if it worked for you under Windows or on some storage platform. I'm here to document the way it works on FreeBSD, Linux, Apple, and most other UNIX variants. I didn't make these design decisions but I am spending some time here to help you understand them. So please don't belabor the point.
I'm dead serious. This IS the way that it is. What you're promoting is an invalid network configuration. No amount of insistence otherwise, or hacky workarounds, can make it magically valid. This not a "challenge" to overcome. Your way is broken. I'm sorry. But I'm also not going to entertain allowing blatantly incorrect information being posted to the forum. We have too many new users who will latch onto the vaguest wisp of a hope that something they want to be able to do will "work," and then we end up needing to keep cleaning up the results of your misinformed contribution, long after you've stopped participating in the forum, every time another newbie comes along and tries what you suggest.
 

jgreco

Resident Grinch
Moderator
Joined
May 29, 2011
Messages
12,826
I thought the problem with multiple NICs on the same subnet is that the machine which hosts them is unable to choose which one to use for a given connection it initiates itself, at least with the usual network software. On Linux one can use arp to control routes, but I'm not sure how to do so dynamically to balance load.
No, the "problem" is that outbound interface selection is handled by the routing table. All outbound traffic to a given subnet happens via one interface. When people attach multiple logical interfaces to a single IP subnet, it still picks one to handle outbound traffic.

What's supposed to happen if you have multiple interfaces?

Is it supposed to load-balance them? That's what LACP is for; use LACP, several physical interfaces, and a single logical interface, in a standardized manner. Including other beneficial features!

Is it supposed to try to emit traffic from the interface that happens to have a matching source IP? That's what people often expect, but what if you're dealing with routed traffic? You actually expect the system to inspect EACH packet against all of the interfaces on the destination net, taking a significant performance hit? That'd be hilarious and scales REALLY awesomely, not.

The proponents of this sort of idiocy usually cannot see beyond their own desire to make their own little edge case work the way they'd like. The reality is that making things work the way they'd like would impact performance dramatically for other applications. And since there is already a blessed way to support multiple physical interfaces on a single subnet (LACP), I see this as being a total nonissue.
 

rogerh

Neophyte Sage
Joined
Apr 18, 2014
Messages
1,077
Yes sorry, I'm not really thinking about multiple routes to the same subnet, and routing via the same IP to different subnets is a totally different problem, and probably not good for throughput!
 

jgreco

Resident Grinch
Moderator
Joined
May 29, 2011
Messages
12,826
Yes sorry, I'm not really thinking about multiple routes to the same subnet, and routing via the same IP to different subnets is a totally different problem, and probably not good for throughput!
S'alright, it's a complex topic.
 

Black Ninja

Neophyte Sage
Joined
Nov 11, 2014
Messages
1,174
S'alright, it's a complex topic.
In this case what is the proper way to do it ? I have LACP setup with my dual nic, but it's not always balancing when it needed, I tried by ip or mac hash , but some times it's just using 1 link to receive from 2 machines instead of receiving from both. Perhaps with many clients like you said will will balance better , but it's not guaranteed with 2 clients.

So what is the proper way to use multiple interfaces without LACP in order to balance few clients ?

I just want to make sure backup client for example never crosses path with Workstations traffic to the server. How would you set it up, without LACP ?
 

jgreco

Resident Grinch
Moderator
Joined
May 29, 2011
Messages
12,826
So what is the proper way to use multiple interfaces without LACP in order to balance few clients ?
You upgrade to 10GbE.

I just want to make sure backup client for example never crosses path with Workstations traffic to the server. How would you set it up, without LACP ?
You upgrade to 10GbE.

LACP has certain limitations that are based on the way the underlying technologies operate. Trying to do it without LACP is even more of a losing battle.
 

Black Ninja

Neophyte Sage
Joined
Nov 11, 2014
Messages
1,174
I 'll get 10g eventually but still researching on :coper vs fiber, intel vs chelsio, switch, etc.

Will it work if quad gigabit nic on freenas is set with 4 interfaces:
192.168.0.10
192.168.1.10
192.168.2.10
192.168.3.10

and then if I want backup machine to have dedicated link to freenas not use by anybody else I just give the backup machine ip 192.168.3.20 ? No other machine is on 192.168.3.x subnet , so I assuming only backup server will use that link, the rest of the network let's say it's on 192.168.0.x subnet. Is this the proper way to do it ?
 
Top