Packet Loss exclusive to FreeNas software

[Curtipus]

I've been trying to copy some large files onto my FreeNas which I just recently set up. Every time I try the copy, at some point, the connection breaks. After doing research online on other forums for errors with rsync and cp I've decided that they're not at fault and my network's robustness likely is.

So I ran some flooding pings [ping -f -s 10000 xxx.xxx.xxx.xxx] to and from the FreeNas server only to find that absolutely every time it resulted in at least a 25% packet loss. After rewiring my network with other cables, trying direct links, and just testing the overall robustness of the rest of my network, I determined that it was only my FreeNas server that was experiencing this problem. Every other PC on my network running those same pings to each other results in 0% packet loss.

This is where it got interesting. I put in a live CD for a linux distro on the FreeNas server and did the same ping tests and it resulted in 0% packet loss.

As far as I know, I've not changed any drastic settings within FreeNas to cause this 25-28% packet loss. I followed the FreeNas Setup guide for my static IP and that's all I've done with the server in terms of networking.

Does anyone have any idea why this would happen and what I can do to fix it?

Thanks in advance!

Forgot to mention, this is FreeNAS 9.2.0 RELEASE 64

 

[krikboh]

What is your hardware? Are using a Realtek NIC?


Sent from my iPhone using Tapatalk

 

[Curtipus]

What is your hardware? Are using a Realtek NIC?


Sent from my iPhone using Tapatalk

Just like you, my hardware description is in my signature, it's a dell R210 II with two Intel Gb NICs that didn't have any problems on Linux or ESXi.

Thanks for the curiosity!

 

[krikboh]

I could not see your signature as I was looking from my phone. Even if I had seen your signature it would not have answered my question as the standard network controller in the Dell PowerEdge R210 II is a Broadcom BCM 5716. Are you confident you have Intel NICs?

 

[Curtipus]

My apologies!

The on-board dual port NIC is the Broadcom 5716 chip. Thanks for catching that. It's a new server of mine and I'm just used to Intel chips in some of my other ones. Regardless, these broadcom's haven't any problems on Linux or ESXi, just FreeNas.

Any idea? I appreciate your help, krikboh. Thanks you.

 

[krikboh]

It's all about the drivers. The Broadcom drivers for FreeBSD/FreeNAS just don't get the same attention. That is why Intel NIC's are so strongly recommended. You'll save yourself a lot of headaches if you just pop in an Intel NIC card.

 

[joeschmuck]

If you have any Intel NICs laying around, give one a try to either rule it out or confirm it. Many RealTek NICs (in FreeBSD/NAS 9) do have an issue with continuous high throughput, especially while transferring high amounts of data in both directions at the same time.

So it's likely a driver issue and I'd recommend you see if FreeBSD has issues with the Broadcom 5716 chip.

 

[Curtipus]

If you have any Intel NICs laying around, give one a try to either rule it out or confirm it. Many RealTek NICs (in FreeBSD/NAS 9) do have an issue with continuous high throughput, especially while transferring high amounts of data in both directions at the same time.

So it's likely a driver issue and I'd recommend you see if FreeBSD has issues with the Broadcom 5716 chip.

Thanks, I'll check to see if FreeBSD has the same problem. Here's to hoping it works fine...

 

[c32767a]

Thanks, I'll check to see if FreeBSD has the same problem. Here's to hoping it works fine...

Be aware that certain OS network stacks have rate limits for ICMP traffic. Flood pinging a network interface will provide little to no useful diagnostic information if the network stack rate limits ICMP.
Even if the stack doesn't rate limit ICMP, the only thing you can infer from that test is whether or not the IP stack can process ICMP efficiently enough to avoid dropping any packets.

 

[Curtipus]

Be aware that certain OS network stacks have rate limits for ICMP traffic. Flood pinging a network interface will provide little to no useful diagnostic information if the network stack rate limits ICMP.
Even if the stack doesn't rate limit ICMP, the only thing you can infer from that test is whether or not the IP stack can process ICMP efficiently enough to avoid dropping any packets.

Have you got other suggestions to test FreeNas/FreeBSD more appropriately without flood pinging?

I have also confirmed that it is indeed occurring within FreeBSD as well. On a FreeBSD 9.2 release live cd I received identical packet loss.

 

[cyberjock]

The common test for networking is iperf. It's available in almost every OS out there. If you have packet loss problems those *do* show up in iperf as slower performance (sometimes catastrophic).

As for pinging, if you really want to know the long-term packet loss you can do pinging, just let it run for a very long time (aka hours). Trying to dish out 10000 packets per second is pointless though. The 1 packet per second default is just fine as packet loss will be random and even 1% packet loss will still manifest itself if you do a standard ping for just an hour.

 

[c32767a]

Have you got other suggestions to test FreeNas/FreeBSD more appropriately without flood pinging?

I have also confirmed that it is indeed occurring within FreeBSD as well. On a FreeBSD 9.2 release live cd I received identical packet loss.

As CJ said, iperf is probably the best tool at this point. It's included with FreeNAS, and it's in the ports collection on FreeBSD.

You should be able to get a binary for whatever OS you have on your client.

The most basic test would be to run iperf -s on your FreeNAS and then iperf -c <nas hostname> on your client (if it's a CLI client).

Then you can reverse the server and client to reverse the direction of the bitstream.

Before we get too far down this path, Are you current with Dell's BIOS/BMC and ethernet card firmware? You should be able to get a live CD for free from the Dell support site that will scan your machine and update any firmware that's behind rev. It's not out of the question that this could be caused by an interaction between the NIC firmware and *BSD's driver.

 

[Curtipus]

As CJ said, iperf is probably the best tool at this point. It's included with FreeNAS, and it's in the ports collection on FreeBSD.

You should be able to get a binary for whatever OS you have on your client.

The most basic test would be to run iperf -s on your FreeNAS and then iperf -c <nas hostname> on your client (if it's a CLI client).

Then you can reverse the server and client to reverse the direction of the bitstream.

Before we get too far down this path, Are you current with Dell's BIOS/BMC and ethernet card firmware? You should be able to get a live CD for free from the Dell support site that will scan your machine and update any firmware that's behind rev. It's not out of the question that this could be caused by an interaction between the NIC firmware and *BSD's driver.

I did run a ping at defaults for an hour as CyberJock suggested, and that came back clean with 0% packet loss.

I just quickly tested using iperf and got good results, about 980 Kb/s and also 0% packet loss.

After this I did use a Centos65-OM74-Firmware-LiveDVD.x86_64-1.1.0-Build6.1 iso from Dell to update the Firmware of both NICs from 6.2.12 to 7.8.16

Reran the ping test and iperf to get same results.

If that's the case, I suppose my NICs actually could be fine, or is there further we should look into? Going back to why I posted this to begin with is because a file transfer of about 800GB that has failed repeatedly. I have a CIFS share on the FreeNas which I'm copying over to from a linux pc using

rsync -r --progress /media/mydata /media/freenasshare

The error was rsync error: error in file IO (code 11) and it seems to happen right around 30GB most times.

Any ideas? And thanks for all of your help so far, guys! I'm learning a lot going through all this.

 

[c32767a]

If that's the case, I suppose my NICs actually could be fine, or is there further we should look into? Going back to why I posted this to begin with is because a file transfer of about 800GB that has failed repeatedly. I have a CIFS share on the FreeNas which I'm copying over to from a linux pc using

The error was rsync error: error in file IO (code 11) and it seems to happen right around 30GB most times.

Any ideas? And thanks for all of your help so far, guys! I'm learning a lot going through all this.

Ok. Just to make sure we get the units right, you have a 1G connection to your FreeNAS and you saw 980Kb/s? Or 980Mb/s?

And your rsync: you've mounted a CIFS share from FreeNAS on a Linux box, and you're running rsync on the Linux box, copying files from the CIFS share to a local folder on the Linux box, correct?

I would check a number of things..

First would be to ensure that the user credentials you've used to mount the CIFS share on the Linux box have read access to everything on the CIFS shares and there are no permissions problems.
How big are the files you're copying? I presume it's not 1 800 GB file?

Are there any error messages from nmbd or smbd in the FreeNAS logs?

How did the files get stored on the FreeNAS box?

 

[cyberjock]

I'd say everything is fine. I will tell you that from experience, 1% packet loss is totally crippling when it comes to using anything over a network. By 3% the network is totally and completely unusable.

 

[Curtipus]

Ok. Just to make sure we get the units right, you have a 1G connection to your FreeNAS and you saw 980Kb/s? Or 980Mb/s?

And your rsync: you've mounted a CIFS share from FreeNAS on a Linux box, and you're running rsync on the Linux box, copying files from the CIFS share to a local folder on the Linux box, correct?

I would check a number of things..

First would be to ensure that the user credentials you've used to mount the CIFS share on the Linux box have read access to everything on the CIFS shares and there are no permissions problems.
How big are the files you're copying? I presume it's not 1 800 GB file?

Are there any error messages from nmbd or smbd in the FreeNAS logs?

How did the files get stored on the FreeNAS box?

1. It is a 1 Gb connection, you're right. I am only seeing 980Kb/s - not 980Mb/s using iperf. This is probably bad, maybe I'm using iperf wrong.
2. I did mount a 100% public no authentication CIFS share where I am using rsync on a linux box, copying files from the linux box to the CIFS share (you had it the other way around). From Windows/Linux I'm able to read and write to the CIFS share seemingly without any issue.
My mount parameters look like the following in /etc/fstab:

//192.168.X.XXX/freenasshare /media/freenasshare cifs guest,rw,nounix,iocharset=utf8,file_mode=0777,dir_mode=0777 0 0

3. The files I'm copying are no larger than about 8GB each. About 500GB worth is 700 to 900MB each
4. I do not have any error messages available as I've restarted the server several times since I last tried to copy (and if I recall correctly, those error logs are stored in ram). If you feel I should try another copy to look for error logs I'd be happy to, just will take some time.
5. The files are not stored on the FreeNas, as described in answer 2. It's on a linux RAID0 of two 500GB drives in the linux box I'm trying to copy from.

 

[c32767a]

Ok thanks.. It's important to be clear about everything so that we know we're troubleshooting the right components.

So what I'm hearing is that if you mount the CIFS share from the FreeNAS on a Windows machine, and copy a file to it, you get acceptable performance, but if you do it from the Linux box, you get the failure you posted above, correct?

Did you try iperf from one of the Windows machines to the FreeNAS?

 

[Curtipus]

Ok thanks.. It's important to be clear about everything so that we know we're troubleshooting the right components.

So what I'm hearing is that if you mount the CIFS share from the FreeNAS on a Windows machine, and copy a file to it, you get acceptable performance, but if you do it from the Linux box, you get the failure you posted above, correct?

Did you try iperf from one of the Windows machines to the FreeNAS?

Windows and linux seemingly give decent performance and I can copy small amounts of data at a time. The most I copied over successfully on linux was probably about 6GB and I didn't have any issues.

For fun, I did go ahead and try to run another copy of about 15GB (this time through drag and drop with Nautilus). These are the results in /var/log/syslog on the linuxbox:
Jul 27 11:56:51 CMDX01 kernel: [606337.218152] CIFS VFS: sends on sock ffff8800a706fc00 stuck for 15 seconds
Jul 27 11:56:51 CMDX01 kernel: [606337.218162] CIFS VFS: Error -11 sending data on socket to server
Jul 27 11:57:06 CMDX01 kernel: [606352.225463] CIFS VFS: sends on sock ffff8800a706fc00 stuck for 15 seconds
Jul 27 11:57:06 CMDX01 kernel: [606352.225473] CIFS VFS: Error -11 sending data on socket to server
Jul 27 11:58:39 CMDX01 kernel: [606445.175592] CIFS VFS: sends on sock ffff8800a706fc00 stuck for 15 seconds
Jul 27 11:58:39 CMDX01 kernel: [606445.175602] CIFS VFS: Error -11 sending data on socket to server
Jul 27 11:58:54 CMDX01 kernel: [606460.182898] CIFS VFS: sends on sock ffff8800a706fc00 stuck for 15 seconds
Jul 27 11:58:54 CMDX01 kernel: [606460.182908] CIFS VFS: Error -11 sending data on socket to server

And the GUI gave me "There was an error copying the file into /media/freenasshare: Error Closing file: input/output error"

FreeNas gave this in /var/log/messages:

Jul 27 11:57:22 curtinas smbd[14965]: [2014/07/27 11:57:22.945746, 0] smbd/process.c:244(read_packet_remainder)
Jul 27 11:57:22 curtinas smbd[14965]: read_fd_with_timeout failed for client 0.0.0.0 read error = NT_STATUS_CONNECTION_RESET.
Jul 27 11:59:08 curtinas smbd[15068]: [2014/07/27 11:59:08.709467, 0] smbd/process.c:244(read_packet_remainder)
Jul 27 11:59:08 curtinas smbd[15068]: read_fd_with_timeout failed for client 0.0.0.0 read error = NT_STATUS_CONNECTION_RESET.


I'll keep doing some more troubleshooting and I'll try a large file copy from a windows box.

 

[cyberjock]

You still using the Broadcom? If so you should probably try an Intel before going further. You seem to be running in circles and we recommend Intel in the FreeNAS manual, my noobie guide, and about 1000 places in the forum. They are simply the best you can get for FreeBSD.

 

[Curtipus]

You still using the Broadcom? If so you should probably try an Intel before going further. You seem to be running in circles and we recommend Intel in the FreeNAS manual, my noobie guide, and about 1000 places in the forum. They are simply the best you can get for FreeBSD.

Yes, still trying the broadcom. But I'll try an intel just to double check things as soon as I can. It's just sad that something which desires to be free and open would limit itself to such proprietary limitations (Don't take that the wrong way - I know it's not FreeNas' fault for lack of driver support).