Network fails on reboot

[Dennis Apostolos]

I'm new to FreeNAS and this my first build. Everything seems to be running fine, but when I reboot or shutdown/start the system, about 75% of the time the network does not start up correctly. Symptoms are not consistent, sometimes FreeNAS acts like the link is up, but my PCs can't see it; sometimes it will tell me there No Interfaces Configured. I can sometimes go in and restart the interface and everything works fine, other times I have to re-enter the interface.

System configuration: FreeNAS-9.3-STABLE-201511040813; Intel Xeon-1246v3; 32GB ECC ; ASRock C226-WS, Intel E1G44HT I349-T4 PCIe adapter.

Network equipment: Zyxel GS1900-16; ASUS RT-AC3200/Asuswrt-Merlin 378.54_2

Network config: Static IP; OpenDNS nameservers, Link aggregation- lagg0 (lacp: igb0, igb1, igb2, igb3)

I've rebuilt the interface and aggregation many times, reloaded the OS twice, and the problem continues to appear. Not that I think it's related but CIFS shares periodically stop working on the some PCs while working on others (I've been writing that off to vagaries of Windows).

Any thoughts or ideas would be appreciated.

 

[DrKK]

I'm willing to bet $1500 that it's your link aggregation. For S&G, why not just turn it off, reboot the box with a simple, single, interface to the gateway (no LACP/aggregation), and let's see if the problem disappear.

Then, at least, we'll know where to direct the effort.

 

[Dennis Apostolos]

DrKK, I never seem to able to do anything simply. So I changed the configuration to 4 individual links (ie., no link aggregation). Everything in the chain NAS, switch, & router seems to be real touchy toggling link aggregation and they all had to be powered down/up. The links have survived multiple restarts & power cycles, and the client PC's seem not losing their shares. So that seems to work. I thought I had tested this earlier, but I might have had something like DHCP involved which seems to do funky things when multiple links are in place.

Given how flakey the problems are, I wonder if I should rethink the use of aggregation. Any ideas what would cause the problem with aggregation or is it just real touchy by its nature?

 

[solarisguy]

I am getting the same behaviour on a a hardware that was experiencing no problems on 9.2.1.x. Currently FreeNAS-9.3-STABLE-201511040813, but I had experienced similar behaviour with previous releases sin the 29th of September one.

I can see that a DHCP offer is always accepted, but later it is a hit and miss.

I did Enable netwait feature. That has changed nothing. I can still see DHCP assigning the address, but then I just know sooner (when ICMP to the address from the Netwait IP list fails) that I have to reset the network configuration.

My system is now in production. But I think I have had on more than one occasion that I could access the Web GUI, but I had to add the interface there. Weird.

Ultimately, I have no opportunity to troubleshoot that system, so I have adjusted my start-up procedures instead (be by the console, do not restart remotely).

 

[DrKK]

I wonder if I should rethink the use of aggregation

Yeah, I can't help you, because I've never aggregated links in my life.

But, I can tell you this from a software support standpoint. Link aggregation is *DEFINITELY* one of these things that people think they understand, and do not. As a result, the believe they are going to multiply their network throughput by approximately N where N is the number of aggregating links, but the truth is, on the ground floor, most people that aren't enterprise IT experts experience 0% (or nearly 0%) improvement as a result of various link aggregation scenarios that they implement. This is primarily, I am told, due to misconceptions as to the nature of how link aggregation protocols work, and the resulting overexpectation. We have a lot of things like this: L2ARC, ZIL, encryption, link aggregation, etc., where we have tons of users that think these things are "pimp" and will provide impressive performance gains. The truth on the ground is that they usually do not---if you NEED any of these things, you will already know you do, and you will know why. If you're putting it on just to chase some elusive gain for its own sake, our users tend to be disappointed.

So my instinct is to say: "**ck the link aggregation".

 

[Dennis Apostolos]

I wasn't really expecting a 'performance' gain, but more of an efficient use of existing hardware. My admittedly limiting understanding of aggregation is that I would have a 4 lane highway with a 1gb speed limit on each with aggregation dynamically assigning cars to different lanes as the entered the highway. Fortunately my usage is not that demanding, so I can get the performance and most of the efficiency by statically assigning the cars to lanes myself.

 

[DrKK]

I wasn't really expecting a 'performance' gain, but more of an efficient use of existing hardware. My admittedly limiting understanding of aggregation is that I would have a 4 lane highway with a 1gb speed limit on each with aggregation dynamically assigning cars to different lanes as the entered the highway. Fortunately my usage is not that demanding, so I can get the performance and most of the efficiency by statically assigning the cars to lanes myself.

As I said, I'm no expert at all about LACP and other aggregation strategies. I have been present when people that have understood this have been discussing why the typical user like you, or me, won't actually get anything at all for it.

In any case, the hassle-to-reward factor appears to be extraordinarily bad in your case. Understanding what's happening though would be at least intellectually stimulating.

 

[pirateghost]

OP, you should read these.

https://forums.freenas.org/index.php?threads/lacp-friend-or-foe.30541/

https://forums.freenas.org/index.php?threads/multiple-network-interfaces-on-a-single-subnet.20204/

 

[rogerh]

I wasn't really expecting a 'performance' gain, but more of an efficient use of existing hardware. My admittedly limiting understanding of aggregation is that I would have a 4 lane highway with a 1gb speed limit on each with aggregation dynamically assigning cars to different lanes as the entered the highway. Fortunately my usage is not that demanding, so I can get the performance and most of the efficiency by statically assigning the cars to lanes myself.

If you now have 4 NICs on the same subnet I can't see how you can allocate the traffic to "lanes". Possibly the traffic in one direction will go where it's put, but the traffic from FreeNAS will probably go via the default gateway. Or if you know of a way of doing this, I would be very interested.

 

[DrKK]

OP, you should read these.

https://forums.freenas.org/index.php?threads/lacp-friend-or-foe.30541/

https://forums.freenas.org/index.php?threads/multiple-network-interfaces-on-a-single-subnet.20204/

These were excellent reads. Should be required reading for all of us.

 

[solarisguy]

I'm new to FreeNAS and this my first build. Everything seems to be running fine, but when I reboot or shutdown/start the system, about 75% of the time the network does not start up correctly. Symptoms are not consistent, sometimes FreeNAS acts like the link is up, but my PCs can't see it; sometimes it will tell me there No Interfaces Configured. I can sometimes go in and restart the interface and everything works fine, other times I have to re-enter the interface. [...] Link aggregation- lagg0 (lacp: igb0, igb1, igb2, igb3) [...]

I am seeing the same problems as Dennis had described...

My problems do not appear to be related to LACP (or lagg interface), as I have only one interface. OK, I just lied. My Intel motherboard has two interfaces, but the other one is disabled in the BIOS and FreeNAS (any OS) cannot see it.

I did not report the issue, since:
a) I cannot really take my system off-line for extensive troubleshooting,
b) when I became certain that I am seeing an issue that can be replicated on around 50% of reboots, an announcement was made about aggressive schedule for FreeNAS 10...
c) I can work around it (=stay by the console);
d) I had thought that it must be something peculiar in my networking hardware (switch or router), as otherwise everybody would see the same issue.

P.S.
LACP does work on FreeNAS (see the referenced post). So I think it should be enough to ask posters whether they know what to do on the other end. If they do not or they ask what the other end is, then LACP is not for them.

 

[Dennis Apostolos]

If you now have 4 NICs on the same subnet I can't see how you can allocate the traffic to "lanes". Possibly the traffic in one direction will go where it's put, but the traffic from FreeNAS will probably go via the default gateway. Or if you know of a way of doing this, I would be very interested.

Primarily I'm going from PC>NAS backups and directory copies. Basically its a residential operation although I do work on other peoples systems. I figure that I would periodically saturate a single link, maybe two (adding an additional 2 ports isn't all that expensive). I give each ethernet port a static IP, then on the Windows side I setup the share by directly entering the IP (assigning each computer to a given IP). Hadn't really thought about it, but during any given session, say during a back up I would think the NAS would be sending back ( NAS>PC ) on the same physical link (the I sent you a 1000 bytes, OK I received a 1000 bytes send some more type of banter that goes on ). Most of data is moving from the PC, initiated/directed by me.

 

[pirateghost]

Primarily I'm going from PC>NAS backups and directory copies. Basically its a residential operation although I do work on other peoples systems. I figure that I would periodically saturate a single link, maybe two (adding an additional 2 ports isn't all that expensive). I give each ethernet port a static IP, then on the Windows side I setup the share by directly entering the IP (assigning each computer to a given IP). Hadn't really thought about it, but during any given session, say during a back up I would think the NAS would be sending back ( NAS>PC ) on the same physical link (the I sent you a 1000 bytes, OK I received a 1000 bytes send some more type of banter that goes on ). Most of data is moving from the PC, initiated/directed by me.

Please read the links I provided earlier

 

[rogerh]

Primarily I'm going from PC>NAS backups and directory copies. Basically its a residential operation although I do work on other peoples systems. I figure that I would periodically saturate a single link, maybe two (adding an additional 2 ports isn't all that expensive). I give each ethernet port a static IP, then on the Windows side I setup the share by directly entering the IP (assigning each computer to a given IP). Hadn't really thought about it, but during any given session, say during a back up I would think the NAS would be sending back ( NAS>PC ) on the same physical link (the I sent you a 1000 bytes, OK I received a 1000 bytes send some more type of banter that goes on ). Most of data is moving from the PC, initiated/directed by me.

As @pirateghost says, you need to study more. The packets towards FreeNAS will probably go to the NIC you want, but packets in the other direction (including TCP acks and things) probably won't. And whether all this can be explained to the switch (you must presumably have a managed switch to be attempting link aggregation) sufficiently well for it to co-operate is another question. It is entirely possible that you may lose more than you gain.