FreeNAS resets network connections once per minute

[SirHaxalot]

Hello,

I've set up my FreeNAS box again and I think that I'm seriously a magnet for trouble when it comes to FreeNAS...

This time I've encountered a problem where the FreeNAS box randomly resets all network connections every minute, making it unaccessible for a few seconds every time. It didn't start until I enabled CIFS sharing and Active Directory authentication and I first noticed that all connections to FreeNAS would drop if I initiated a file transfer even though browsing worked fine. It's not only related to Samba because any SSH connection also stops working, and I see latency spikes from NFS clients even though the connection remains.

I've tried to turn off both the CIFS and Active Directory service and rebooted the FreeNAS box with no improvement. It still seems to be triggered with about a minute in between and I can't find any trace of the issue in any logs.

Specs:
Intel Core i3 3220
16GB RAM
4x 2TB 7200 rpm drives, confed as 2x mirror vdevs.
Intel i350-T2 NIC for main data and build in Realtek NIC for management traffic.
FreeNAS 8.3.1-BETA3

EDIT:

Okay, it seems I've made a mistake when ruling out the firewall, which probably is the cause here. I do not under stand why though since it should pass all traffic between the hosts.

 

[downingjosh]

Hello,

I've set up my FreeNAS box again and I think that I'm seriously a magnet for trouble when it comes to FreeNAS...

This time I've encountered a problem where the FreeNAS box randomly resets all network connections every minute, making it unaccessible for a few seconds every time. It didn't start until I enabled CIFS sharing and Active Directory authentication and I first noticed that all connections to FreeNAS would drop if I initiated a file transfer even though browsing worked fine. It's not only related to Samba because any SSH connection also stops working, and I see latency spikes from NFS clients even though the connection remains.

I've tried to turn off both the CIFS and Active Directory service and rebooted the FreeNAS box with no improvement. It still seems to be triggered with about a minute in between and I can't find any trace of the issue in any logs.

Specs:
Intel Core i3 3220
16GB RAM
4x 2TB 7200 rpm drives, confed as 2x mirror vdevs.
Intel i350-T2 NIC for main data and build in Realtek NIC for management traffic.
FreeNAS 8.3.1-BETA3

EDIT:

Okay, it seems I've made a mistake when ruling out the firewall, which probably is the cause here. I do not under stand why though since it should pass all traffic between the hosts.

Did you resolve this?

We are having an identical issue with our Dell T620 with an Intel I350DP adapter. This entry hits /var/log/messages about every minute.

Feb 25 16:25:05 freenas notifier: devd already running? (pid=1119).
Feb 25 16:25:05 freenas notifier: Setting hostuuid: 4c4c4544-0051-5610-8030-c3c04f515731.
Feb 25 16:25:05 freenas notifier: Setting hostid: 0xb65d682f.
Feb 25 16:25:06 freenas notifier: Starting Network: lo0 igb0 igb1.
Feb 25 16:25:06 freenas notifier: lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
Feb 25 16:25:06 freenas notifier: options=3<RXCSUM,TXCSUM>
Feb 25 16:25:06 freenas notifier: inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5
Feb 25 16:25:06 freenas notifier: inet6 ::1 prefixlen 128
Feb 25 16:25:06 freenas notifier: inet 127.0.0.1 netmask 0xff000000
Feb 25 16:25:06 freenas notifier: nd6 options=3<PERFORMNUD,ACCEPT_RTADV>
Feb 25 16:25:06 freenas notifier: igb0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
Feb 25 16:25:06 freenas notifier: options=401bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO>
Feb 25 16:25:06 freenas notifier: ether a0:36:9f:06:8b:1e
Feb 25 16:25:06 freenas notifier: inet 10.1.1.96 netmask 0xffffff00 broadcast 10.1.1.255
Feb 25 16:25:06 freenas notifier: media: Ethernet autoselect (1000baseT <full-duplex>)
Feb 25 16:25:06 freenas notifier: status: active
Feb 25 16:25:06 freenas notifier: igb1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
Feb 25 16:25:06 freenas notifier: options=401bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO>
Feb 25 16:25:06 freenas notifier: ether a0:36:9f:06:8b:1f
Feb 25 16:25:06 freenas notifier: inet 10.1.1.92 netmask 0xffffff00 broadcast 10.1.1.255
Feb 25 16:25:06 freenas notifier: media: Ethernet autoselect (1000baseT <full-duplex>)
Feb 25 16:25:06 freenas notifier: status: active
Feb 25 16:25:06 freenas root: /etc/rc.d/dhclient: ERROR: /etc/rc.d/dhclient: no interface specified
Feb 25 16:25:06 freenas notifier: /etc/rc.d/dhclient: ERROR: /etc/rc.d/dhclient: no interface specified
Feb 25 16:25:06 freenas notifier: route: writing to routing socket: File exists
Feb 25 16:25:06 freenas notifier: add net ::ffff:0.0.0.0: gateway ::1: route already in table
Feb 25 16:25:06 freenas notifier: route: writing to routing socket: File exists
Feb 25 16:25:06 freenas notifier: add net ::0.0.0.0: gateway ::1: route already in table
Feb 25 16:26:04 freenas notifier: devd already running? (pid=1119).
Feb 25 16:26:04 freenas notifier: Setting hostuuid: 4c4c4544-0051-5610-8030-c3c04f515731.
Feb 25 16:26:04 freenas notifier: Setting hostid: 0xb65d682f.
Feb 25 16:26:04 freenas notifier: Starting Network: lo0 igb0 igb1.
Feb 25 16:26:04 freenas notifier: lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
Feb 25 16:26:04 freenas notifier: options=3<RXCSUM,TXCSUM>
Feb 25 16:26:04 freenas notifier: inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5
Feb 25 16:26:04 freenas notifier: inet6 ::1 prefixlen 128
Feb 25 16:26:04 freenas notifier: inet 127.0.0.1 netmask 0xff000000
Feb 25 16:26:04 freenas notifier: nd6 options=3<PERFORMNUD,ACCEPT_RTADV>
Feb 25 16:26:04 freenas notifier: igb0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
Feb 25 16:26:04 freenas notifier: options=401bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO>
Feb 25 16:26:04 freenas notifier: ether a0:36:9f:06:8b:1e
Feb 25 16:26:04 freenas notifier: inet 10.1.1.96 netmask 0xffffff00 broadcast 10.1.1.255
Feb 25 16:26:04 freenas notifier: media: Ethernet autoselect (1000baseT <full-duplex>)
Feb 25 16:26:04 freenas notifier: status: active
Feb 25 16:26:04 freenas notifier: igb1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
Feb 25 16:26:04 freenas notifier: options=401bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO>
Feb 25 16:26:04 freenas notifier: ether a0:36:9f:06:8b:1f
Feb 25 16:26:04 freenas notifier: inet 10.1.1.91 netmask 0xffffff00 broadcast 10.1.1.255
Feb 25 16:26:04 freenas notifier: media: Ethernet autoselect (1000baseT <full-duplex>)
Feb 25 16:26:04 freenas notifier: status: active
Feb 25 16:26:05 freenas root: /etc/rc.d/dhclient: ERROR: /etc/rc.d/dhclient: no interface specified
Feb 25 16:26:05 freenas notifier: /etc/rc.d/dhclient: ERROR: /etc/rc.d/dhclient: no interface specified
Feb 25 16:26:05 freenas notifier: route: writing to routing socket: File exists
Feb 25 16:26:05 freenas notifier: add net ::ffff:0.0.0.0: gateway ::1: route already in table
Feb 25 16:26:05 freenas notifier: route: writing to routing socket: File exists
Feb 25 16:26:05 freenas notifier: add net ::0.0.0.0: gateway ::1: route already in table​

We're going to implement the system tunable recommended here and see if that sorts it out:
http://support.freenas.org/ticket/1729

I'll report back my findings in a day or two.

 

[ProtoSD]

It looks like you're using DHCP and your lease is expiring. Why not set a static IP address?

EDIT: Also, please follow the forum rules and post the complete details about your hardware and FreeNAS version.

 

[downingjosh]

It looks like you're using DHCP and your lease is expiring. Why not set a static IP address?

EDIT: Also, please follow the forum rules and post the complete details about your hardware and FreeNAS version.

It's a dual-NIC; one of the adapters has a static IP set, the other (at the time the log was captured) was set for DHCP. We've tried it with one of the interfaces deleted & the other with a static IP, both DHCP (with reservations), and both set with static IPs (the original configuration).

 

[ProtoSD]

Ok, well the request for the details about your hardware & FreeNAS version wasn't a suggestion.

 

[downingjosh]

Ok, well the request for the details about your hardware & FreeNAS version wasn't a suggestion.

Alright, fair enough; my eyes somehow skipped over that part of your reply.

FreeNAS 8.3 p1 x64
Dell T620
1x Xeon E5-2620
32 GB DDR3 1600Mhz
Network - 1x Intel I350 DP
Storage - PERC H310 RAID Controller
4x WD Red 3 TB SATA drives
4x Samsung 840 Pro 256 GB drives

 

[ProtoSD]

Thank You :)

You don't realize how much time is wasted repeating that question for everyone that posts a question without it.

The tunable you mentioned changing was supposed to be the default in 8.3.0 Beta-1.
You can verify what it is from the command line by doing:

Code:

sysctl kern.ipc.nmbclusters

Some sysctl settings do require a reboot.

 

[downingjosh]

Thank You :)

You don't realize how much time is wasted repeating that question for everyone that posts a question without it.

The tunable you mentioned changing was supposed to be the default in 8.3.0 Beta-1.
You can verify what it is from the command line by doing:

Code:

sysctl kern.ipc.nmbclusters

Some sysctl settings do require a reboot.

I'll check that before setting up the tunable; hopefully this is the fix, because this is some very strange behavior. The only other thing I could think to try was a driver update.

 

[ProtoSD]

I'll check that before setting up the tunable; hopefully this is the fix, because this is some very strange behavior. The only other thing I could think to try was a driver update.

Keep in mind that FreeBSD is not Linux, you can't just download and install an updated driver, they're built into the kernel. Support for that card wasn't introduced until FreeBSD 9.x, so the driver has been backported to work with FreeNAS (FreeBSD 8.3). You could download a live distro of FreeBSD 9.1 and see if the problem still exists there. If it doesn't, maybe there been some changes that need backporting. Finding out now before the release of 8.3.1 would be helpful so if that was the case it could be included.

 

[SirHaxalot]

Did you resolve this?

Well, I discovered that the problem only appears when passing my firewall, because it blocks packets it shouldn't block for some reason so I'm afraid I can't really help here. I though I had tried running all traffic trough the management interface on the same network but apparently I did something wrong..

I'll get a new NAS that will replace my FreeNAS box today though, it will be interesting to see if that also has the same problem with the firewall..

 

[paleoN]

I'll get a new NAS that will replace my FreeNAS box today though, it will be interesting to see if that also has the same problem with the firewall..

If the problem is your firewall then why would it be any different?

 

[downingjosh]

Keep in mind that FreeBSD is not Linux, you can't just download and install an updated driver, they're built into the kernel. Support for that card wasn't introduced until FreeBSD 9.x, so the driver has been backported to work with FreeNAS (FreeBSD 8.3). You could download a live distro of FreeBSD 9.1 and see if the problem still exists there. If it doesn't, maybe there been some changes that need backporting. Finding out now before the release of 8.3.1 would be helpful so if that was the case it could be included.

So we tried the onboard NIC instead with perhaps better results, but the main symptom is still there. The previous log message is gone, the network card does not appear to be resetting any more. Now, we're seeing this entry in 'messages':

Feb 26 19:28:50 freenas istgt[2405]: istgt_lu_disk.c:6737:istgt_lu_disk_execute: ***ERROR*** unsupported SCSI OP=0x4d
Feb 26 19:28:50 freenas istgt[2405]: istgt_lu_disk.c:6737:istgt_lu_disk_execute: ***ERROR*** unsupported SCSI OP=0x4d
Feb 26 19:30:14 freenas last message repeated 2 times
Feb 26 22:05:33 freenas istgt[2405]: Login from iqn.1998-01.com.vmware:localhost-393433a0 (10.1.1.93) on iqn.2011-03.example.org.istgt:datadrive LU2 (10.1.1.91:3260,1), ISID=23d000001, TSIH=2, CID=0, HeaderDigest=off, DataDigest=off

The primary reason we started looking into this issue is that our VMware host was losing connection to our 'datadrive' iSCSI target. I should have mentioned earlier that of the 2 iSCSI targets on the server, only 1 of them is experiencing this problem. 1 is 4 SSDs (Samsung 840 Pro) and the other is 4 rotational drives (Western Digital Reds); the problem only occurs on the Western Digital Red LUN (called 'datadrive' above). The SSDs hold the guest OSes and the WD Reds are given to the file server VM to use as a data drive. This makes me think that the issue may not be strictly with the network card, but perhaps with the drives or their configuration.

Please let me know what further info I can give you to help. Thanks for everything so far!

 

[downingjosh]

Well, I discovered that the problem only appears when passing my firewall, because it blocks packets it shouldn't block for some reason so I'm afraid I can't really help here. I though I had tried running all traffic trough the management interface on the same network but apparently I did something wrong..

I'll get a new NAS that will replace my FreeNAS box today though, it will be interesting to see if that also has the same problem with the firewall..

I would be interested to see if you experience this behavior with an alternate NIC; I stopped seeing the 60-second resets as soon as I stopped using the I350. I'm going to try to run a Live CD of FreeBSD 9 as suggested by protosd to see if perhaps the backported driver simply isn't ready yet.

 

[cyberjock]

I may be wrong, but I seem to remember someone having problems with their i350 and they solved it by adding a sysctl. I may be wrong, but I think it was kern.ipc.maxsockbuf = 2097152.

 

[downingjosh]

I may be wrong, but I seem to remember someone having problems with their i350 and they solved it by adding a sysctl. I may be wrong, but I think it was kern.ipc.maxsockbuf = 2097152.

I wonder if you mean this one:
http://support.freenas.org/ticket/1729

kern.ipc.nmbclusters
Value 262144

If so, that's already included in the version of FreeNAS we're running. If you think the kern.ipc.maxsockbuf sysctl would help, please let me know!

 

[cyberjock]

I wonder if you mean this one:
http://support.freenas.org/ticket/1729

kern.ipc.nmbclusters
Value 262144

If so, that's already included in the version of FreeNAS we're running. If you think the kern.ipc.maxsockbuf sysctl would help, please let me know!

That's it! I was pretty sure the number was 262144 but I removed it recently from my system.

 

[downingjosh]

That's it! I was pretty sure the number was 262144 but I removed it recently from my system.

I'm going to create a new thread for the SCSI command errors to avoid thread-jacking this one. I'll still report back to this thread once we get a chance to try a FreeBSD 9.x live CD with the I350 adapter, though it may be next week before we have time for that.