Register for the iXsystems Community to get an ad-free experience and exclusive discounts in our eBay Store.

Dual Ethernet without Link Aggregation

Black Ninja

FreeNAS Guru
Joined
Nov 11, 2014
Messages
1,174
Just my 2 cents:

I use static IPs on all my devices but I use "virtual" subnets to make the IP/device pair easier to remember. Explanation: for example I use IPs from 192.168.0.1 to 192.168.0.9 for the servers, from 192.168.0.11 to 192.168.0.19 for the desktops PC, 192.168.0.21 to 192.168.0.29 for the VMs, ... I also have the DHCP attribued IPs on one of these ranges, 192.168.0.101 to 192.168.0.109 for example ;)
That is a very good way to organize them. I have similar strategy with set range 192.168.1.2-199 for DHCP specific assigned IP's according to my scheme and the range from 192.168.1.200-255 is the management hardware like IPMI, DRac , etc in which devices have their own static IP without DHCP intervention.
 

Bidule0hm

Server Electronics Sorcerer
Joined
Aug 5, 2013
Messages
3,710
Yeah, and i'ts pretty flexible. If you have more than 9 devices per group you can jump from .1 to .19, then .21 to .39, ... for example ;)

Note that I don't use the round IPs (.10, .20, ...) because I can't use .0 for the first range I do so to match the other ranges to the first (so I know that the first device of a range is .x1, the second is .x2, ...). Yes, I lose 10% of the IPs by doing this but I don't have more than 15 or 20 devices so... :D

You can also avoid the first range entirely and use the .x0 for every range ;)

NB: the DHCP IPs are only used for WiFi devices (actually there's only one device, and it's not mine :p) and every other temporary device (the smartphone of a guest who stay only a few days for example) :)
 
Last edited:

Black Ninja

FreeNAS Guru
Joined
Nov 11, 2014
Messages
1,174
Yeah, and i'ts pretty flexible. If you have more than 9 devices per group you can jump from .1 to .19, then .21 to .39, ... for example ;)

Note that I don't use the round IPs (.10, .20, ...) because I can't use .0 for the first range I do so to match the other ranges to the first (so I know that the first device of a range is .x1, the second is .x2, ...). Yes, I lose 10% of the IPs by doing this but I don't have more than 15 or 20 devices so... :D

You can also avoid the first range entirely and use the .x0 for every range ;)

NB: the DHCP IPs are only used for WiFi devices (actually there's only one device, and it's not mine :p) and every other temporary device (the smartphone of a guest who stay only a few days for example) :)
I got confused now. Why you can't use round IP like 192.168.1.10 you keyboard "0" is not working ?:D

What is wrong with using from 192.168.1.2 to 192.168.1.10 or larger group to 192.168.1.20 for your servers and the then the next group is from 192.168.1.21 to 192.168.1.30 for next group ?
 

Rilo Ravestein

FreeNAS Experienced
Joined
Mar 6, 2014
Messages
685
P.S. If you don't want you nas to get a DNS and gateway, then I see your point there,but No updates in that case too.
You can configure static routes for the updates.
 

Black Ninja

FreeNAS Guru
Joined
Nov 11, 2014
Messages
1,174

Bidule0hm

Server Electronics Sorcerer
Joined
Aug 5, 2013
Messages
3,710
Ah sorry, maybe not uber clear. I meant that for the .1 to .9, .11 to .19, ... ranges, I don't use the 0 by choice. Of course if you use the .1 to .19, .21 to .39, ... ranges you use the .10, .30, ... (but not the .0, .20, .40, ...) IPs.

The reason is because 192.168.0.0 is the address of the subnet itself so I can't use it so the first range starts unavoidably at .1. Now, if I use .10 for the second range the first device of this range would be on .10, same for the third range with .20. Well, this causes a mismatch in the ranges (.1-.9, .10-.19, .20-.29, ...) as you can see the first device of the first range is not on .0 like the logic leaves you to think, but on .1 instead.

This is purely to have a full logical system with no exceptions to the rules (who often leads to errors because you assume something by logic but it's not the case) I setted ;)

However, I don't know if it's more clear or less clear now... :p
 

Black Ninja

FreeNAS Guru
Joined
Nov 11, 2014
Messages
1,174
Ah sorry, maybe not uber clear. I meant that for the .1 to .9, .11 to .19, ... ranges, I don't use the 0 by choice. Of course if you use the .1 to .19, .21 to .39, ... ranges you use the .10, .30, ... (but not the .0, .20, .40, ...) IPs.

The reason is because 192.168.0.0 is the address of the subnet itself so I can't use it so the first range starts unavoidably at .1. Now, if I use .10 for the second range the first device of this range would be on .10, same for the third range with .20. Well, this causes a mismatch in the ranges (.1-.9, .10-.19, .20-.29, ...) as you can see the first device of the first range is not on .0 like the logic leaves you to think, but on .1 instead.

This is purely to have a full logical system with no exceptions to the rules (who often leads to errors because you assume something by logic but it's not the case) I setted ;)

However, I don't know if it's more clear or less clear now... :p
Now I got it. It make perfect sense. You have a very good logic my friend. All you need is a good UPS , I gather from your signature.:):):)
 

Bidule0hm

Server Electronics Sorcerer
Joined
Aug 5, 2013
Messages
3,710
Thanks ;)

Yeah, it's a pseudo sinus (well, this is the commercial term, because on my oscilloscope it's roughly a square wave... :rolleyes:) but I extensivly tested the pair UPS/server and no problem.

However I'll read on the subject of the active PFC because I don't want to damage the PSU. The logic, my electronic background and the few things I know about the active PFCs of the ATX PSUs tell me that the PSU doesn't risk anything but it's just a intuition, and in the other recent thread about the APC UPS someone says he wants a pure sinus UPS because of the PSU's active PFC. In short, I need to search and read some good doc on the subject :p

NB: I modded the UPS (what I' haven't modded? well... not a lot of things :D) to put a car battery so no problem on the autonomy side of things (the stock battery was gone a long time ago and I don't like the tiny piss-ant lead acid batteries used in every UPS I saw...)

I also wonder about runing an ATX PSU on a DC input. It should work (some PSU are labeled with an AC input and a DC input voltages/currents) but if the PSU isn't specially made for this it might damage it. The goal is to simplify the UPS problem of course :)
 

Black Ninja

FreeNAS Guru
Joined
Nov 11, 2014
Messages
1,174
Pure Sin is the way to got, I uses stepped on so many devices( not just PSU , but microwave , circular saw , etc.) and some don't even start on Stepped wave. Since you are electronics guy , you know that switching time on the UPS has to be faster than PSU hold time , otherwise will drop the load before switch to batteries mode, not a common problem but thing to consider.
 

Bidule0hm

Server Electronics Sorcerer
Joined
Aug 5, 2013
Messages
3,710
Yes, for motors it's important, but the first thing an ATX PSU do (after the PFC if present and the line filter) is rectify the current. It's why, in theory, any ATX PSU can run on a DC input current (but because of the PFC it's not a good idea to do this without knowing what you need to know about) so the UPS can produces a square wave it doesn't matter (but, again, it's the PFC who might be the problem if not a sinus, the rest of the PSU isn't) :)

Yep, I mesured the switch time too: min 1/2 cycle, max 3/4 cycle (at 50 Hz). It's in specs for the ATX specs IIRC, and well in specs for my PSU (another reason to not buy a crappy PSU...) ;)
 

Black Ninja

FreeNAS Guru
Joined
Nov 11, 2014
Messages
1,174
I am not that knowledgeable about electronics. I have to read more about the PFC. As fas as the hold up time I know it's measured in ms and The ATX specification sets the minimum hold-up time to 16 ms, but many power supplies are not in that category of 16ms. Example even good brand corsair inexpensive CS650W is 11 ms, while corsair AX760 is 19ms. I don't know how to measure it so I reline on other peoples testings.
 

Bidule0hm

Server Electronics Sorcerer
Joined
Aug 5, 2013
Messages
3,710
Yeah, It's because I don't have the exact number on hand, I recall just the number of quarters of cycle (at 50 Hz a cycle is 20 ms) so 3/4 is 15 ms, just in the ATX spec. But in fact during the switching of my UPS it's not a total flat line at zero, there is some usable power so the effective switch time is virtually less than 15 ms. The switch time I measured is the time it takes to switch from the line normal waveform to the UPS normal waveform so it's the worst case ;)

Plus the ATX spec is at full load and an average FreeNAS server run the PSU at maybe 10 to 30% so you have a longer hold time effectively.

Corsair CS... yeah, not the best PSUs in, the world I think :p No problem on the AX (they use SeaSonic internals for the most, so obviously...).

The ATX spec is defined as (IIRC): the time it takes to the output power rails to go out of spec (+/- 10% for the +/- 12 V rails, +/- 5% for the others rails IIRC) at full load after unpluging the PSU. But yeah, don't do the test yourself on a real server, it's not a good idea for his longevity :)
 
Last edited:

jgreco

Resident Grinch
Moderator
Joined
May 29, 2011
Messages
12,153
Well if you change MB , it's different computer , it don't matter what chassis is in. My point was that the MAC should be the same for the MB's build in NIC no matter where is going to go.
In the server business the assembly is often referred to as a chassis, even though that's arguably incorrect. Separating a mainboard from an enclosure is not done as part of the normal lifecycle of a server in a machine room or data center, unless the board is defective and being replaced, which usually doesn't happen in the machine room or data center anyways. When you have a hardware problem with one server in a rack of 20, the two common choices are to replace the entire server, which is fine if it is interchangeable, like a hypervisor, or to replace the chassis, which implies at a minimum pulling all the drives and quite possibly other bits and jamming them into a spare chassis, and getting back into service quickly. For a storage server, since there's stateful information on the drives, it's necessary to do the latter. In both cases, the MAC address is changing.

Right, and most are 1 port Realtek ,that's why we need to add Quad Intel or 10GB ports on separate nic.;)
No server boards that I know of come with "1 port Realtek". Most come with "2 port Intel", some with "4 port Intel", and others with "2 port Broadcom."

If you're not using recommended hardware, that's an entirely different problem.
 

Black Ninja

FreeNAS Guru
Joined
Nov 11, 2014
Messages
1,174
Jpreco, I agree with you on 100%, except that chassis and server are two different things and should be referred with their proper names. Some people can refer to computers as cpu, the box, the brain , the chassis or anything they like , but is not correct. If you call server manufacturer Supermicro and tell them "Hey guys I am swapping chassis and need to order new chassis" , they will not think you need new server , they will send you a chassis not a whole computer-server.
That's why I got confused when you said: " when you I swap chassis mac will change" , you were referring to the server I was thinking you mean chassis when you said chassis.

I didn't want to put the definition of server and chassis for you , so you don't feel insulted. I know you are very knowledgeable person and I hope you can take constructive criticism.:)
 

FNSeeker

Newbie
Joined
Jan 3, 2014
Messages
36
It's a good feeling for me to see other people such as Biduleohm approaching the use of static IP addressing in a similar fashion to mine. It takes a couple of minutes each time setting up each host but one has more control over traffic destinations (especially in SOHO-like environments).

In my case, the x.x.x.x0 is reserved for each group's own switch if there's one and in the absence of VLANs. I have a different VLAN for Wifi connection though.

________
Back to the original posting re dual Ethernet without LAG.

I tried it in 9.2 but email simply stopped working. Not sure if it was a bug but it led to my alternative approach of 'road traffic management'.

Combining the two 'roads' into one does not make a car go any faster than it's physically capable of. All LAG does is reducing potential traffic congestion by utilizing both roads for traffic flowing in both directions. This requires constant traffic monitoring.

It means the CPU takes on a cop duties directing network traffic. At peak times, this may adversely affect its computing power when required by something such Plex transcoding, if used on NAS.

Solution without using LAG: Mapping out specific needs based on demand of network devices and assigned them to a specific IP. Kinda traffic from certain group travel on nominated route (i.e. IP) in this case.

And it works out fine.
 

jgreco

Resident Grinch
Moderator
Joined
May 29, 2011
Messages
12,153
Jpreco, I agree with you on 100%, except that chassis and server are two different things and should be referred with their proper names. Some people can refer to computers as cpu, the box, the brain , the chassis or anything they like , but is not correct. If you call server manufacturer Supermicro and tell them "Hey guys I am swapping chassis and need to order new chassis" , they will not think you need new server , they will send you a chassis not a whole computer-server.
That's why I got confused when you said: " when you I swap chassis mac will change" , you were referring to the server I was thinking you mean chassis when you said chassis.

I didn't want to put the definition of server and chassis for you , so you don't feel insulted. I know you are very knowledgeable person and I hope you can take constructive criticism.:)
Actually, HP, and Dell, and Supermicro are the three major brands that most of the industry uses, and the origination of the terminology confusion, because two of them don't sell the parts separately, and the third, while it does sell them separately, prefers to sell them preintegrated.

The problem is that "server" typically means a complete machine, so calling a mainboard+enclosure a "server" without slipping further into confusion. Since most manufacturers sell their gear as at least a mainboard+enclosure, that's often referred to as a chassis, as in "let's swap out the chassis and get it back up and running."

Feel free to suggest a different name for the motherboard+enclosure combination, and if and when it becomes commonly used, I'll be happy to support it.
 

Black Ninja

FreeNAS Guru
Joined
Nov 11, 2014
Messages
1,174
That's a good point. I was thinking about Supermicro when talking about the chassis because they are the one who sell the chassis or case by it self without any other components. For Dell it's just a whole system , if you can plug it in wall and power it up , I would call it a server. When somebody refers as chassis - I see server case with power supplies, fans, drive bays, perhaps fan shroud but nothing else. I know if this was NORCO you won't get the power supplies , but still they refer to the same thing.

For motherboard+enclosure combination ( assuming it need memory, hdd , cpu) I've seen this refer to barebones. But that not my idea, I am neither for or against it.:)
 

philious77

Newbie
Joined
Apr 17, 2015
Messages
10
This is a bit of a late reply but I figure I'd throw it out there for the benefit of any future inquiries. First of all, having multiple nics with IP's in the same subnet is perfectly legitimate. Let's take a look at an example -- I have a server with 4 nics and I give them the following IP's:

nic1: 192.168.1.1 / 255.255.255.0
nic2: 192.168.1.2 / 255.255.255.0
nic3: 192.168.1.3 / 255.255.255.0
nic4: 192.168.1.4 / 255.255.255.0

This is a PERFECTLY valid network configuration; but there is a challenge. It's not an "issue" or a "problem", but a challenge. This challenge is discussed in this link, but it is incomplete:

https://forums.freenas.org/index.php?threads/multiple-network-interfaces-on-a-single-subnet.20204/

If you look at that article and then follow the link from that article to the subsequent Apple article (https://support.apple.com/en-us/HT203436), you will see the title of the document:

Multiple physical network interfaces on the same subnet may cause routing issues

the key word in this document is ROUTING. What is routing? Routing is needed when you want to access another subnet. Think of it as a street name....I live on street 192.168.1.0 and my house address is 192.168.1.1. I want to visit my friend who lives on the exact same street and his address is 192.168.1.2. I don't need directions on how to get there -- he lives on my street! But if I want to visit my concubine who happens to live on another street (192.168.77.0, for example), I need a route -- I need directions -- I need a GPS -- how do I get there?

The default gateway is what is required. The default gateway is my guide -- it guides me where I need to go to get to my mistress' address. Each computer can only have one default gateway. If you want to have more "paths" to other networks then you need to add static routes. I'm starting to babble on here because I'm drunk but he are the summarized facts:

-You can have as many nics as you want and you can put them all on the same subnet. This is a perfectly fine configuration.
-ANY other machine that is also on that SAME subnet will be able to access all those IP's, assuming of course that the service it's trying to access (cifs, nfs, apache, mysql, sql, ssh, etc) is configured to listen on those IP's.
-but any other machine which is NOT on the same subnet ------ that machine will only be able to access the interface that has a default gateway (or static route) configured.

So, if my configuration is this:

nic1: 192.168.1.1 / 255.255.255.0 GW: 192.168.1.254
nic2: 192.168.1.2 / 255.255.255.0 GW: -
nic3: 192.168.1.3 / 255.255.255.0 GW: -
nic4: 192.168.1.4 / 255.255.255.0 GW: -

then every machine on the 192.168.1.0 subnet will be able to see those IP's. It's a great way to segregate traffic (perhaps you want to dedicate one nic for backup traffic?). Regardless of whether I type \\192.168.1.1\backup, \\192.168.1.2\backup, \\192.168.1.3\backup, or \\192.168.1.4\backup, I will see the same share (assuming, of course, that samba is listening on all interfaces) but each one of those IP's will give me a full 1Gps without any kind of LACP.

however every machine which is on a different subnet will ONLY be able to see 192.168.1.1 because the others don't have a route to other networks, and you can't add multiple ROUTES to OTHER networks to interfaces on the SAME subnet. Accessing the shares would yield success to 192.168.1.1, but the other 3 would fail. Trying to create routes to those additional IP's would be a massive pain (not even sure if it's possible).


I hope this is somewhat clear.
 

SweetAndLow

Sweet'NASty
Joined
Nov 6, 2013
Messages
5,968
Not
This is a bit of a late reply but I figure I'd throw it out there for the benefit of any future inquiries. First of all, having multiple nics with IP's in the same subnet is perfectly legitimate. Let's take a look at an example -- I have a server with 4 nics and I give them the following IP's:

nic1: 192.168.1.1 / 255.255.255.0
nic2: 192.168.1.2 / 255.255.255.0
nic3: 192.168.1.3 / 255.255.255.0
nic4: 192.168.1.4 / 255.255.255.0

This is a PERFECTLY valid network configuration; but there is a challenge. It's not an "issue" or a "problem", but a challenge. This challenge is discussed in this link, but it is incomplete:

https://forums.freenas.org/index.php?threads/multiple-network-interfaces-on-a-single-subnet.20204/

If you look at that article and then follow the link from that article to the subsequent Apple article (https://support.apple.com/en-us/HT203436), you will see the title of the document:

Multiple physical network interfaces on the same subnet may cause routing issues

the key word in this document is ROUTING. What is routing? Routing is needed when you want to access another subnet. Think of it as a street name....I live on street 192.168.1.0 and my house address is 192.168.1.1. I want to visit my friend who lives on the exact same street and his address is 192.168.1.2. I don't need directions on how to get there -- he lives on my street! But if I want to visit my concubine who happens to live on another street (192.168.77.0, for example), I need a route -- I need directions -- I need a GPS -- how do I get there?

The default gateway is what is required. The default gateway is my guide -- it guides me where I need to go to get to my mistress' address. Each computer can only have one default gateway. If you want to have more "paths" to other networks then you need to add static routes. I'm starting to babble on here because I'm drunk but he are the summarized facts:

-You can have as many nics as you want and you can put them all on the same subnet. This is a perfectly fine configuration.
-ANY other machine that is also on that SAME subnet will be able to access all those IP's, assuming of course that the service it's trying to access (cifs, nfs, apache, mysql, sql, ssh, etc) is configured to listen on those IP's.
-but any other machine which is NOT on the same subnet ------ that machine will only be able to access the interface that has a default gateway (or static route) configured.

So, if my configuration is this:

nic1: 192.168.1.1 / 255.255.255.0 GW: 192.168.1.254
nic2: 192.168.1.2 / 255.255.255.0 GW: -
nic3: 192.168.1.3 / 255.255.255.0 GW: -
nic4: 192.168.1.4 / 255.255.255.0 GW: -

then every machine on the 192.168.1.0 subnet will be able to see those IP's. It's a great way to segregate traffic (perhaps you want to dedicate one nic for backup traffic?). Regardless of whether I type \\192.168.1.1\backup, \\192.168.1.2\backup, \\192.168.1.3\backup, or \\192.168.1.4\backup, I will see the same share (assuming, of course, that samba is listening on all interfaces) but each one of those IP's will give me a full 1Gps without any kind of LACP.

however every machine which is on a different subnet will ONLY be able to see 192.168.1.1 because the others don't have a route to other networks, and you can't add multiple ROUTES to OTHER networks to interfaces on the SAME subnet. Accessing the shares would yield success to 192.168.1.1, but the other 3 would fail. Trying to create routes to those additional IP's would be a massive pain (not even sure if it's possible).


I hope this is somewhat clear.
Not sure what you are taking about but all of those IP would be addressable from a different subnet as long as there was a router that knew about the subnet. And there should be since they got their ips from something.
 

rogerh

FreeNAS Guru
Joined
Apr 18, 2014
Messages
1,069
I thought the problem with multiple NICs on the same subnet is that the machine which hosts them is unable to choose which one to use for a given connection it initiates itself, at least with the usual network software. On Linux one can use arp to control routes, but I'm not sure how to do so dynamically to balance load.
 
Top