FreeNAS Box Brings Down the Whole LAN

[FreeNASBob]

I have an Asrock C2550D4i with 16GB RAM running FreeNAS from a USB stick. No zpools attached yet. As I was working in the FreeNAS console looking around the file system with a lot of ls -a commands the IPMI went haywire and lost the connection and wouldn't reconnect. I power cycled the server. Now when the machine is powered off everything on my LAN works fine. I can ping my router for hours on end with 1ms response times. Shortly after the FreeNAS machine is powered up the router is dropping packets like crazy and nobody's iPhones or Androids can access the Internet. The pings from the desktop to the FreeNAS box go through the roof with about 60% packet loss. It's so laggy that I can't access IMPI again because it gives me errors about unexpected connection closing or something.

I unplugged the LAN cable and only left the IPMI cable plugged in and I still see the same issue. If I unplug the IPMI cable the issue goes away. It seems like there's some major issue with the IPMI. I'm not sure how to go about fixing it or resetting IPMI, or how it could trash the whole network.

Any ideas are appreciated.

 

[DJ9]

Posting your hardware specs would be a good start.

 

[FreeNASBob]

Posting your hardware specs would be a good start.

There isn't any hardware attached except the C2550D4i, the RAM, and the Kingston USB stick.

 

[DrKK]

FreeNASBob: The odds that this is a hardware issue with the board are just about zero. We have hundreds of users that use this board without the behavior you describe. The issue is almost certainly some kind of network conflict. Are your IP addresses assigned by DHCP? Have you done some kind of inadvertent mistake in something related to IP address assignment or routing?

 

[FreeNASBob]

IPMI gets it's IP by DHCP. I just look in the router's devices list to see what IP it's been assigned. I've reset the IPMI to factory defaults and the BIOS to factory defaults. I tried different LAN cables for the IPMI. Still same issue. I did flash the 2.10 BIOS for this board a couple of days ago, but I've not had any issues until today.

 

[DrKK]

OK, dumb question here: You know you don't need a second cable for IPMI right? You can get to the IPMI IP through the same LAN cable that you use for the main NIC (this is, I believe, called the "failover" feature in the BIOS). Having a dedicated cable for the IPMI is not necessary, as far as I know. So, while you worry about figuring out this problem, you can sidestep it entirely for the time being, and still use your IPMI, if I'm not mistaken. My experience is only with SuperMicro boards though.

 

[FreeNASBob]

Right now I just have one LAN cable plugged into the dedicated IPMI port. I'm trying to reduce the variables to as few as possible.

 

[FreeNASBob]

I'm upgrading the BMC to 0.19.0 as I write this. We'll see if that helps.

 

[FreeNASBob]

I'm keeping my fingers crossed, but it looks like upgrading the BMC to 0.19.0 did the trick.

 

[enemy85]

OK, dumb question here: You know you don't need a second cable for IPMI right? You can get to the IPMI IP through the same LAN cable that you use for the main NIC (this is, I believe, called the "failover" feature in the BIOS). Having a dedicated cable for the IPMI is not necessary, as far as I know. So, while you worry about figuring out this problem, you can sidestep it entirely for the time being, and still use your IPMI, if I'm not mistaken. My experience is only with SuperMicro boards though.

Really? I didn't know that and always used 2 different cables. I'm with an Asrock 226D2I board btw. I'll try if that work the same!

 

[Ericloewe]

Really? I didn't know that and always used 2 different cables. I'm with an Asrock 226D2I board btw. I'll try if that work the same!

You may have to enable the option in BIOS.

 

[FreeNASBob]

It turns out it was the FreeNAS box, just not how I thought. I guess the router was too close to the server so whenever the server was powered on the router would go nuts, but only sometimes. I increased the separation between the server and the router and all appears stable for the time being. I really need to get up in the attic and run those CAT6 lines to all the rooms like I've been planning for years. A gigabit switch with lines to all the rooms in the house would be much better than a consumer router with most devices on WiFi.

 

[FreeNASBob]

This issue is not going away. Now the IPMI IP address won't respond to pings, but the FreeNAS IP address does. It's all on one cable so it can't be the NIC. The intermittent nature is making me crazy. I keep thinking I've solved it and then two hours later it rears its head again. I had even gone as far as resetting the BIOS using the jumper pads this morning in case I borked something, but no dice.

 

[mjws00]

Somewhere around here there is also a thread with performance issues, due to ipmi being bound to all nics. The fix was to disable the "failover" in the BIOS and ensure it was ONLY bound to the IPMI nic. Unfortunately a quick search didn't find a link to the thread. So it may be on a different board, but it was asrock. Have you tried disabling IPMI entirely? If the problem goes away that would at least narrow the hunt.

 

[Robert Smith]

Put a Wireshark or something on the network to sniff the traffic and see exactly what is going on.

 

[anodos]

While you are at it post the output of ifconfig -a and some details about the hardware you have on your network.

I once had a small network attached print server that had a magical 'kill my network in the right circumstances' feature. I also once had an old low-end tenda router that would crash if I put too much load on it. When you consider how terrible those appliances were, you could say that I found their real 'killer features'. You might want to try removing everything from the network apart from the FreeNAS server and a workstation and try to make your network crash again.

 

[FreeNASBob]

Thanks for the suggestions and tips so far. I'm painstakingly testing one thing at a time. So far the network's been very well behaved since I wiped the router and set it up new.

This reminds me of a crazy LAN problem I had back in the 1990's. We were preparing a classroom for software instruction classes and we hand-built about 15 workstations and put them all on the LAN, but none of them could see each other on the LAN. Weird. We tried a new hub and the problem remained. After hours of pulling our hair out we got the bright idea to turn off all of the workstations and power them up one by one. Sure enough, the LAN worked great until this one particular workstation came online. Then nobody could talk to anybody. We tried that workstation's NIC in other workstations and it had no issue. We tried it in different slots of the affected PC and it always hosed the network. We never did figure out how it was bringing down the network, but it sure was strange that one card when placed in one particular workstation would do it.

 

[cyberjock]

^^^ That's why hubs are bad and switches are good. :)

 

[pirateghost]

Heh. When I first started playing around with virtual networking, I caused many a broadcast storm...it's no fun