SOLVED Network drops out consistently after running for a week or so

[Stingray88]

Hey guys, so I've been running my FreeNAS rig for about 4 or 5 months now and for the most part everything has worked as I've wanted to. However in the last 2-3 months its started to completely disconnect from the network after being up for about a week. All network functionality drops out, and no other machine on my network can see the FreeNAS machine. I've tried a lot of methods to get it up again, and the only way I've gotten a network connection is by rebooting FreeNAS, then rebooting my router, then rebooting FreeNAS once more. After that all is right again... for another week or so.

My router is an ASUS RT-AC87U running the latest firmware from ASUS (currently 3.0.0.4.378_5134-gefca797). I also use my own modem, an ARRIS SURFboard SB6183.

My FreeNAS rig is an ASRock E3C226D2I, i3 4370, 16GB of ECC RAM, and 6 x 4TB HDDs in RAIDz2. The motherboard has dual Intel i210 NICs, and I'm just using one of them. Currently running the latest stable build of FreeNAS (FreeNAS-9.3-STABLE-201602031011).

Thank you for any assistance you guys can lend me here. I'm really hoping I can figure this out once and for all.

 

[dlavigne]

Post your debug (from System -> Advanced -> Save Debug).

 

[Stingray88]

Thanks for the help dlavigne, here is my debug log.

The last time I noticed FreeNAS was not on the network was February 12th at about 9:23pm PST. Thats when I rebooted FreeNAS and my router. The logs show at about 7am PST that day something with the network happening, but I don't understand what it means.

 

[DrKK]

One thing that happens with the Asus routers, that I can't explain: if network connectivity to the router drops for any reason (e.g., network restart, whatever), then, the FreeNAS never seems to reconnect (nor the jails) until you restart things on the FreeNAS side. This happens whenever I, for example, update the firmware in my router.

 

[Stingray88]

One thing that happens with the Asus routers, that I can't explain: if network connectivity to the router drops for any reason (e.g., network restart, whatever), then, the FreeNAS never seems to reconnect (nor the jails) until you restart things on the FreeNAS side. This happens whenever I, for example, update the firmware in my router.

Ugh... Well that's not great to hear. I spent a pretty penny on that router not too long ago. And it's unfortunately not supported by any of the popular 3rd party router softwares as far as I know. Hoping that's not the issue...

 

[jgreco]

If you have a DHCP assignment for the FreeNAS box, try a static instead. That sounds kind of like DHCP retardedness where the DHCP server loses its lease database and then stupidness spirals out of control until things are very sad.

Infrastructure devices like a NAS should always be wired down anyways. There's no good that can ever come of a NAS suddenly being assigned a different IP while running.

 

[Ericloewe]

One thing that happens with the Asus routers, that I can't explain: if network connectivity to the router drops for any reason (e.g., network restart, whatever), then, the FreeNAS never seems to reconnect (nor the jails) until you restart things on the FreeNAS side. This happens whenever I, for example, update the firmware in my router.

Really? I never noticed that.

 

[Yatti420]

I never use DHCP.. Agreed with jgreco.. Seems odd Asus specific routers would do that.. I know I always have problem with DHCP..

 

[Mirfster]

Wow, if that truly is the case with Asus; then they have some "splainin to do"

 

[DrKK]

Really? I never noticed that.

In the cases I'm aware of, it's almost always when the router is set to "access point" mode, rather than full router mode.

 

[Stingray88]

If you have a DHCP assignment for the FreeNAS box, try a static instead. That sounds kind of like DHCP retardedness where the DHCP server loses its lease database and then stupidness spirals out of control until things are very sad.

Infrastructure devices like a NAS should always be wired down anyways. There's no good that can ever come of a NAS suddenly being assigned a different IP while running.

I've been using a static. I generally always assign static IPs for some of the hardwired devices on my network.

 

[Stingray88]

In the cases I'm aware of, it's almost always when the router is set to "access point" mode, rather than full router mode.

That wouldn't apply here then. I'm using it as a full on router and not just an access point.

 

[Ericloewe]

In the cases I'm aware of, it's almost always when the router is set to "access point" mode, rather than full router mode.

So, the topology would be with the Asus router doing switching and AP duties, but no DHCP or anything else? That is even weirder.

I do have a similar setup right now, temporarily. Asus router doing edge device stuff + DHCP + WiFi, Asus router doing switching and WiFi for most devices, with additional switches. I'll see if anything funny happens.

 

[DrKK]

So, the topology would be with the Asus router doing switching and AP duties, but no DHCP or anything else? That is even weirder.

I do have a similar setup right now, temporarily. Asus router doing edge device stuff + DHCP + WiFi, Asus router doing switching and WiFi for most devices, with additional switches. I'll see if anything funny happens.

I don't have a cogent hypothesis on this. There was, however, a recent thread in which someone was commenting that the jails play some kind of dance with MAC addresses; I was going to investigate if some pernicious interaction between that and the ASUS router in AP mode was to blame if I found myself extremely bored one day.

 

[Stingray88]

I don't have a cogent hypothesis on this. There was, however, a recent thread in which someone was commenting that the jails play some kind of dance with MAC addresses; I was going to investigate if some pernicious interaction between that and the ASUS router in AP mode was to blame if I found myself extremely bored one day.

I did find it quite strange that each jail on my FreeNAS rig pulled their own IP addresses using DHCP even though they have the same Mac address as the rig itself. I wasn't sure if that was standard for freenas or not being new to it and the whole jail ideology.

 

[Ericloewe]

I did find it quite strange that each jail on my FreeNAS rig pulled their own IP addresses using DHCP even though they have the same Mac address as the rig itself. I wasn't sure if that was standard for freenas or not being new to it and the whole jail ideology.

Now that you mention this, I did have some trouble accessing the new server/its BMC when it was bridged via my desktop to WiFi (naturally provided by an RT-AC87U in AP mode).

Eventually it worked, but only after I messed around a bit and set static IPs. What I did notice was that the BMC's MAC was being given my desktop's WiFi NIC's DHCP-reserved IP address.

 

[Stingray88]

Almost like clockwork, Friday rolls around and once again my Freenas is not detectable on my network. Attached a new debug log.

Only now my messages are full of logs saying it can't see my UPS... and I've got no explanation for that one, because it can totally see my UPS just fine.

 

[dlavigne]

Weird...

 

[Stingray88]

Couldn't even make it a week this time... :/

Also I can't get my debug logs to upload to the forum anymore. I click upload, select my debug archive, and it shows the icon that its uploading but then nothing happens after that. Won't show under my post.

Can I post an external URL with my debug log? Like from dropbox? Not sure if that's allowed in this forum.

 

[Robert Trevellyan]

Can I post an external URL with my debug log?

Don't see why not.