Network Services Drop

[Jason Brunk]

Good afternoon All,
I have been having an issue recently and I thought this last round of update would fix it.

Periodically all networking services on my freenas stop. They start back up after a bit. But it drops EVERYTHING. Pings stop, iscsi from my esxi box drops (crashing my vms in the process), smb shares drop from every device on the network, nfs drops from a secondary esxi box.

But a few minutes later, all is happy again.

This is NOT a supermicro board as obviously recommended, but the nic's I am using are both intel nic's on board. I started with the one, saw this problem happening so I switched to the other nic. Same issue. I am considering putting a pci intel pro nic in and see if that does anything, but I am not hopeful as it seems like its the networking services failing. It is still an option, just kind of bummed I can't use the 2 onboard gig nic's.

Anyone run in to this? any thoughts?


Thanks!

 

[Mirfster]

Can you post the output of command (in

Code:

 tags): dmesg

Also, please provide specs of your system.

 

[Jason Brunk]

dmesg

[root@skooge] ~# dmesg
WARNING: 192.168.2.12 (iqn.1998-01.com.vmware:55974bbb-9ddd-5751-43da-001cc42ea440-4a2086f0): no ping reply (NOP-Out) after 5 seconds; dropping connection
WARNING: 192.168.2.126 (iqn.1998-01.com.vmware:567a59e4-541c-e991-603a-00e081ef6aea-1c6d65d9): no ping reply (NOP-Out) after 5 seconds; dropping connection
WARNING: 192.168.2.14 (iqn.1991-05.com.microsoft:dib): no ping reply (NOP-Out) after 5 seconds; dropping connection
WARNING: 192.168.2.126 (iqn.1998-01.com.vmware:567a59e4-541c-e991-603a-00e081ef6aea-1c6d65d9): no ping reply (NOP-Out) after 5 seconds; dropping connection
WARNING: 192.168.2.126 (iqn.1998-01.com.vmware:567a59e4-541c-e991-603a-00e081ef6aea-1c6d65d9): no ping reply (NOP-Out) after 5 seconds; dropping connection
WARNING: 192.168.2.14 (iqn.1991-05.com.microsoft:dib): no ping reply (NOP-Out) after 5 seconds; dropping connection
WARNING: 192.168.2.12 (iqn.1998-01.com.vmware:55974bbb-9ddd-5751-43da-001cc42ea440-4a2086f0): no ping reply (NOP-Out) after 5 seconds; dropping connection
WARNING: 192.168.2.14 (iqn.1991-05.com.microsoft:dib): no ping reply (NOP-Out) after 5 seconds; dropping connection
WARNING: 192.168.2.126 (iqn.1998-01.com.vmware:567a59e4-541c-e991-603a-00e081ef6aea-1c6d65d9): no ping reply (NOP-Out) after 5 seconds; dropping connection
WARNING: 192.168.2.12 (iqn.1998-01.com.vmware:55974bbb-9ddd-5751-43da-001cc42ea440-4a2086f0): no ping reply (NOP-Out) after 5 seconds; dropping connection
WARNING: 192.168.2.126 (iqn.1998-01.com.vmware:567a59e4-541c-e991-603a-00e081ef6aea-1c6d65d9): no ping reply (NOP-Out) after 5 seconds; dropping connection
WARNING: 192.168.2.12 (iqn.1998-01.com.vmware:55974bbb-9ddd-5751-43da-001cc42ea440-4a2086f0): no ping reply (NOP-Out) after 5 seconds; dropping connection
WARNING: 192.168.2.126 (iqn.1998-01.com.vmware:567a59e4-541c-e991-603a-00e081ef6aea-1c6d65d9): no ping reply (NOP-Out) after 5 seconds; dropping connection
WARNING: 192.168.2.14 (iqn.1991-05.com.microsoft:dib): no ping reply (NOP-Out) after 5 seconds; dropping connection
WARNING: 192.168.2.12 (iqn.1998-01.com.vmware:55974bbb-9ddd-5751-43da-001cc42ea440-4a2086f0): no ping reply (NOP-Out) after 5 seconds; dropping connection
WARNING: 192.168.2.14 (iqn.1991-05.com.microsoft:dib): no ping reply (NOP-Out) after 5 seconds; dropping connection
WARNING: 192.168.2.126 (iqn.1998-01.com.vmware:567a59e4-541c-e991-603a-00e081ef6aea-1c6d65d9): no ping reply (NOP-Out) after 5 seconds; dropping connection
WARNING: 192.168.2.12 (iqn.1998-01.com.vmware:55974bbb-9ddd-5751-43da-001cc42ea440-4a2086f0): no ping reply (NOP-Out) after 5 seconds; dropping connection
WARNING: 192.168.2.126 (iqn.1998-01.com.vmware:567a59e4-541c-e991-603a-00e081ef6aea-1c6d65d9): no ping reply (NOP-Out) after 5 seconds; dropping connection
WARNING: 192.168.2.14 (iqn.1991-05.com.microsoft:dib): no ping reply (NOP-Out) after 5 seconds; dropping connection
WARNING: 192.168.2.12 (iqn.1998-01.com.vmware:55974bbb-9ddd-5751-43da-001cc42ea440-4a2086f0): no ping reply (NOP-Out) after 5 seconds; dropping connection
WARNING: 192.168.2.126 (iqn.1998-01.com.vmware:567a59e4-541c-e991-603a-00e081ef6aea-1c6d65d9): no ping reply (NOP-Out) after 5 seconds; dropping connection
WARNING: 192.168.2.126 (iqn.1998-01.com.vmware:567a59e4-541c-e991-603a-00e081ef6aea-1c6d65d9): no ping reply (NOP-Out) after 5 seconds; dropping connection
WARNING: 192.168.2.14 (iqn.1991-05.com.microsoft:dib): no ping reply (NOP-Out) after 5 seconds; dropping connection
WARNING: 192.168.2.12 (iqn.1998-01.com.vmware:55974bbb-9ddd-5751-43da-001cc42ea440-4a2086f0): no ping reply (NOP-Out) after 5 seconds; dropping connection
WARNING: 192.168.2.126 (iqn.1998-01.com.vmware:567a59e4-541c-e991-603a-00e081ef6aea-1c6d65d9): no ping reply (NOP-Out) after 5 seconds; dropping connection
WARNING: 192.168.2.12 (iqn.1998-01.com.vmware:55974bbb-9ddd-5751-43da-001cc42ea440-4a2086f0): no ping reply (NOP-Out) after 5 seconds; dropping connection
WARNING: 192.168.2.14 (iqn.1991-05.com.microsoft:dib): no ping reply (NOP-Out) after 5 seconds; dropping connection




Specs:
9 drives (3 raidz in 1 zfs pool)
1 120gig ssd cache
ASRock Q87WS-DL ATX Server Motherboard
Intel Celeron G1820 Haswell Dual-Core 2.7 GHz
8 Gig ECC memory

 

[dlavigne]

Please also post a debug (you can create one in System -> Advanced -> Save Debug.

 

[Jason Brunk]

drop box link

https://www.dropbox.com/s/352qexl8yjvlux8/debug-skooge-20160209123702..tgz?dl=0

 

[Robert Trevellyan]

1 120gig ssd cache

If this is an L2ARC, then you should remove it, because you don't have enough RAM. This might even solve your problem.

 

[Jason Brunk]

Yes, its setup as a cache and my system shows 109G l2arc.

I can remove it. Just curious how much memory I need to be able to use that. Any info on why that would cause a problem? Or recommendations on if there is a different better use for that SSD?

 

[Jason Brunk]

oh, and why would that cause network drop? :)

 

[Robert Trevellyan]

Just curious how much memory I need to be able to use that.

There's no fixed number, but 8GB is definitely not enough. The simplest rule of thumb is to max out your RAM before considering an L2ARC.

When you install an L2ARC, memory is consumed to track its contents. In systems with minimal RAM (yours has the minimum), that RAM would be better used for ARC.

oh, and why would that cause network drop? :)

I'm speculating that your system is choking due to lack of RAM - perhaps some kind of thrashing behavior.

 

[Jason Brunk]

Seems easy enough to try. Any thing I can use that SSD for elsewhere in freenas? Or would i be better off putting it in my esxi box for now?

 

[Robert Trevellyan]

Any thing I can use that SSD for elsewhere in freenas?

Boot device, if you have no better use for it, but it sounds like you do.

 

[Jason Brunk]

roger that. I will try removing the cache ssd and see if it continues. Thanks for the help! :)

 

[jgreco]

Yes, its setup as a cache and my system shows 109G l2arc.

I can remove it. Just curious how much memory I need to be able to use that. Any info on why that would cause a problem? Or recommendations on if there is a different better use for that SSD?

You'd probably want to start off with the L2ARC at around 64GB or 128GB of RAM.

Read all of the first post at https://forums.freenas.org/index.ph...res-more-resources-for-the-same-result.28178/ to understand why.

All your setup is doing is draining the ARC for header entries for the L2ARC, which further cripples its already nearly nonexistent ability to identify useful blocks for the L2ARC.

 

[jgreco]

oh, and why would that cause network drop? :)

It's probably not a network drop so much as it is you're making ZFS have mini strokes and go catatonic trying to handle the demands you are placing on it with a hopelessly tiny amount of resources.

 

[Jason Brunk]

Thank you jgreco. I will definitely review that link. Especially since my esxi environment is relying heavily on iscsi.

 

[jgreco]

If you're actually relying heavily on iSCSI, you should probably read most of the posts I've posted on these forums that mention iSCSI.

 

[pirateghost]

ISCSI? On 8gb?

Feeling lucky?

 

[jgreco]

ISCSI? On 8gb?

Feeling lucky?

I don't know, seems like we've had a small parade of these intriguing tiny-memory-plus-iSCSI deployments in recent weeks. I wonder if I should go back and try to find them all.

 

[Jason Brunk]

I have never had problems till lately with the network dropping. I am guessing I need to go find the recommended specs on build for iscsi use. Granted I only have a couple vm's running but still.

jgreco I did see on your profile you have ALOT of input on the iscsi. I have been reading through some of the threads already :)

 

[Mirfster]

jgreco I did see on your profile you have ALOT of input on the iscsi

He has A LOT of input on EVERYTHING. Just look at the message count, 8,000+. :D