SOLVED Can Someone Interpret This For Me?

[joeschmuck]

Just a gut feeling but I think you will find out the speed fluctuations are due to your source computer, at least I hope that is the situation after spending money on this device you just built up.

The GUI update to 9.2.1.6 is being reported as fine overall. I have just performed the update myself and there were no noticeable issues, I can access all my files and write to the shares, the 2 jails I have are working fine. Keep in mind that I have a very simple setup, no AD or tons of jails, no apple anything. My system is in the basement so I didn't get to see how many times it rebooted but twice is typical so it takes a bit of time when using a USB Flash drive as the boot device. Of course I saved my 9.2.1.3 configuration file before the update.

 

[pschatz100]

I performed a GUI update from 9.2.1.5 to 9.2.1.6 five days ago, with no issues. As joeschmuck says, be patient and let it finish.

Just as a precaution, I deleted all CIFS shares before running the update, then after the update I re-established them. Once everything was back in order, I then did a power down and restart. Probably overkill, but I felt the extra bit of effort would be worth it.

Prior to the update, I was getting transfer rates ~50MB/sec when copying files from my source computer to FreeNAS. Now, I get ~65-70MB/sec. I'm copying to a ZFS mirror. I would expect that copying to a RaidZ1 or RaidZ2 array would be somewhat faster.

Transfer rates over a network connection are affected by a lot of things: Your router, how it handles frames and prioritizes traffic, the operating parameters of your NIC's, the cables, filesystems on the source and receiving machines, and of course the content of the files themselves. Lots of issues that are outside the scope of FreeNAS per se.

 

[NiceTry]

Just a gut feeling but I think you will find out the speed fluctuations are due to your source computer, at least I hope that is the situation after spending money on this device you just built up.

The GUI update to 9.2.1.6 is being reported as fine overall. I have just performed the update myself and there were no noticeable issues, I can access all my files and write to the shares, the 2 jails I have are working fine. Keep in mind that I have a very simple setup, no AD or tons of jails, no apple anything. My system is in the basement so I didn't get to see how many times it rebooted but twice is typical so it takes a bit of time when using a USB Flash drive as the boot device. Of course I saved my 9.2.1.3 configuration file before the update.

Interesting you should mention my computer contributing to the symptom. Last night I connected an external USB3 drive to my computer and did transfers to and from FreeNAS. The network transfer rates went up and down (per my graphing diagnostic) 80-125MB/s with occasional dips into the 50's. The average for the entire transfer of almost 8GB worth of video file was 75MB/s. So, I think you're gut is working quite well. I'm convinced the FreeNAS is working like it is supposed to with the drives it has.

Changing the "Discovery Interval" for minidlna seems to have cured the dropout problem, at least for the past two days. Not sure what the Discovery Interval actually is. I've not found any documentation except how to set it. My guess was a form of heartbeat broadcast on the local net.

9.2.1.6 is up and running. It was installing while I wrote this reply. I think it booted twice and thought a lot in between. It finally got through it all and popped up without a problem.

I found in the literature how it's not a good idea to have the same shares used by CIFS and AFP. I can attest that it is true. Weird behavior having to do with hidden files used to manage the directories and how CIFS and AFP use them (or ignore them) can muck up the file management. Not catastrophic but irritating. Since it is most convenient for me to use AFP, that's what I'll do.

I'm still going to test with a direct cable but I think I'm good. Thank you and the others that contributed to the solutions. I'm grateful for the education.

 

[NiceTry]

I performed a GUI update from 9.2.1.5 to 9.2.1.6 five days ago, with no issues. As joeschmuck says, be patient and let it finish.

Just as a precaution, I deleted all CIFS shares before running the update, then after the update I re-established them. Once everything was back in order, I then did a power down and restart. Probably overkill, but I felt the extra bit of effort would be worth it.

Prior to the update, I was getting transfer rates ~50MB/sec when copying files from my source computer to FreeNAS. Now, I get ~65-70MB/sec. I'm copying to a ZFS mirror. I would expect that copying to a RaidZ1 or RaidZ2 array would be somewhat faster.

Note that transfer rates over a network connection are affected by a lot of things: Your router, how it handles frames and prioritizes traffic, the operating parameters of your NIC's, the cables, the protocols being used, filesystems on the source and receiving machines, and of course the content of the files themselves. Lots of issues that are outside the scope of FreeNAS per se. If you are really interested in perfecting the maximum possible throughput, you will need to learn how to investigate and optimize each thing.

Thanks, pschatz100. And thanks for the benchmark info. I'd not used CIFS shares for awhile but I understand your paranoia in stripping it down and re-establishing them.

I just performed a file transfer test. My performance with 9.2.1.6 has also improved under AFP. A simple drag and drop file copy from FreeNAS to the computer gave me 94MB/s. The same file dropped onto FreeNAS gave 105MB/s. The file was a 7.39GB .mov file. The transfer itself did not exhibit nearly the same variation as I saw before. I hope there is a real reason for this improvement and its not just a fluke or phase of the moon or whatever that's making this happen. I'm happy with it.

Thanks for your advice and suggestions. It's appreciated.

 

[joeschmuck]

Changing the "Discovery Interval" for minidlna seems to have cured the dropout problem, at least for the past two days. Not sure what the Discovery Interval actually is. I've not found any documentation except how to set it. My guess was a form of heartbeat broadcast on the local net.

Discovery Interval is how often in seconds the DLNA server yells out and says "Hey everyone, I'm a DLNA server right here, yes it's me." And so making the time smaller just makes it shout out more frequently.

And I think everything sounds like it's working properly now, I'm glad.

 

[NiceTry]

Discovery Interval is how often in seconds the DLNA server yells out and says "Hey everyone, I'm a DLNA server right here, yes it's me." And so making the time smaller just makes it shout out more frequently.

And I think everything sounds like it's working properly now, I'm glad.

Yes, I think it is working as it should. Thanks again for the help. Apparently shouting at my WD TVLive more often keeps it awake. I hope it was that easy.

 

[NiceTry]

FYI Update...

I figured out how to create a RAM disk on my computer and performed drag and drop transfers through AFP to and from FreeNAS. It achieved 114MB/s from and 119MB/s to the FreeNAS using my 7.39GB .mov file.

I'm very happy with that.

My conclusion is that my desk computers' hard drives with their asynchronous caching and buffering timing was contributing to the variability in FreeNAS transfers. It looks like changes in 9.2.1.6 has also improved the situation. The variability is much less evident.

 

[NiceTry]

Other transfer symptoms have returned under stress testing. But, I also have another drive reporting extended smart test failure. Warranty replacement on the way. Should be here Thursday.

Would someone try transferring a large file (larger than the amount of memory you have) from one volume to another volume on FreeNAS? The idea is to invoke bi-directional simultaneous transfer. Volume to volume is necessary to actually get the transfer to go rather than just moving the file to a different folder on the same volume.

My result, under AFP, was that the transfer started at about 50_MB/s both read and write (~100MB/s total) then, after 3 or 4 GB, it began to slow down until it got to hundreds of bytes/s. Jail or no jail made no difference. Several tries yielded the same result. FreeNAS console on FreeNAS machine (as opposed to GUI) reported NIC (re0) timeout. I had to reboot. Might this be related to the RealTek NIC?

Thinking perhaps an AFP peculiarity, I tried CIFS. CIFS won't start due to sanity check failure. So I tried NFS. I can't seem to get NFS working either. When I attempt the NFS mount, it responds that I don't have permission to access the server. Very frustrating. The system is so secure, I can't use it.

If someone could try this test using whatever they have, I'd like to hear about it...

9.2.1.6, 8GB RAM, AMD FX-4130, ASUS M5A78L-L/USB3 (Realtek ethernet), 4-1TB RAID-Z1

 

[Eric Thompson]

I just ran what you asked on my FreeNAS system - a volume to volume file transfer (or at least from one dataset to another in the pool). Network activity indicated traffic in and out of my pc. Transferred a 40GB .iso file of a blu-ray movie, transfer sat at 40-50MB/s the whole time (~15 min transfer). Running 9.2.1.6 on a supermicro X10SLM+-LN4F with Intel Xeon E3-1230V3 cpu, 32gb ECC Ram and 4x4TB Seagate NAS drives in RaidZ1. Connected via wired gigabit link to FreeNAS machine via switch.

From what I've seen in the thread so far, it does sound like a realtek nic issue, but I don't have any experience running FreeNAS on anything without Intel NICs.

Hope this helps.

 

[NiceTry]

I just ran what you asked on my FreeNAS system - a volume to volume file transfer (or at least from one dataset to another in the pool). Network activity indicated traffic in and out of my pc. Transferred a 40GB .iso file of a blu-ray movie, transfer sat at 40-50MB/s the whole time (~15 min transfer). Running 9.2.1.6 on a supermicro X10SLM+-LN4F with Intel Xeon E3-1230V3 cpu, 32gb ECC Ram and 4x4TB Seagate NAS drives in RaidZ1. Connected via wired gigabit link to FreeNAS machine via switch.

From what I've seen in the thread so far, it does sound like a realtek nic issue, but I don't have any experience running FreeNAS on anything without Intel NICs.

Hope this helps.

Thank you Eric. It helps. My problem may be an artifact of whatever is wrong with the drive that didn't pass the extended SMART test. I hope so. I'll know after I get the replacement tomorrow. Was that a CIFS, AFP or NFS transfer?

 

[Eric Thompson]

I was running CIFS.

 

[NiceTry]

New disk installed and resilvered.

Tried 7.39GB volume to volume transfer (twice). Each time, about 3.5GB into it, the transfer ground to a halt and FreeNAS locked up on the net. "re0 (RealTek NIC) watchdog timeout."

See network throughput graph below. The peak read/write transfer you see at the beginning is about 75MB/s each direction, 150MB/s total (for scale). I think the high peak is an artifact of the measurement but it provides scale for the transfer. It averaged 35-45MB/s each way until it died off at the end.

Is this the kind of issue that can be brought on using the RealTek NIC?

 

[cyberjock]

Is this the kind of issue that can be brought on using the RealTek NIC?

Yes. That's why we don't recommend Realtek....

 

[NiceTry]

Yes. That's why we don't recommend Realtek....

cyberjock: Thanks for the confirmation. It's off to Fry's or mail-order for a GigE Intel NIC. Does it matter which format, model? PCI-e, PCI, ??

Edit: Wasted trip to Fry's. They've reduced their inventory of components.

 

[Ericloewe]

cyberjock: Thanks for the confirmation. It's off to Fry's or mail-order for a GigE Intel NIC. Does it matter which format, model? PCI-e, PCI, ??

Grab the single-port GbE 1x PCIe card. Nobody's ever complained (I'm running two of those in my pfSense box and they're both flawless) about them.

 

[joeschmuck]

I have never had that problem but I do not use AFP either. I'm curious what the results are after switching NICs. We already know the Apple computer isn't doing well at handling throughput well at all but I did a little research and your error message was greatly seen using RealTek NICs however the CPUs were low powered from what I quickly read.

Now if you replace "re0 watchdog timeout" with "em0 watchdog timeout" for a quick Google search then you come up with a ton of Intel NIC errors as well.

As for the NIC card, yes it does matter PCI or PCIe, they are different physical interface connections to the motherboard and you need PCIe. Once done, if you still get the same error then troubleshoot your main computer or network. Ensure you have a direct connection, no switches or routers and try the transfer again.

I did perform a test, I transferred a single 27.4GB Acronis backup file (already compressed) from one dataset (no compression) to another dataset (LZ4 compression), ~61MB/sec (each way) and took maybe 6 minutes (didn't really time it). I use CIFS as I use a windoze system. My NAS is in the basement and I run through two network switches for connectivity, no router in this specific path.

 

[NiceTry]

I have never had that problem but I do not use AFP either. I'm curious what the results are after switching NICs. We already know the Apple computer isn't doing well at handling throughput well at all but I did a little research and your error message was greatly seen using RealTek NICs however the CPUs were low powered from what I quickly read.

Do you have an Intel NIC or are you using the on-board Realtek?

When I set up the RAM disk on the Mac, the single direction transfers saturated the Enet. The Mac Mini has two laptop drives. It actually does better using an external USB3 drive. After I'm done messing with the server, I plan to stripe them to get the performance up.

Now if you replace "re0 watchdog timeout" with "em0 watchdog timeout" for a quick Google search then you come up with a ton of Intel NIC errors as well.

Hmmmm... I'm hoping the consensus opinion of the RealTek NIC is correct and I won't have to do more research.

As for the NIC card, yes it does matter PCI or PCIe, they are different physical interface connections to the motherboard and you need PCIe. Once done, if you still get the same error then troubleshoot your main computer or network. Ensure you have a direct connection, no switches or routers and try the transfer again.

I tested with a direct connection... Same result. I'm REALLY rooting for the Intel NIC as the salvation. I do have another carcass I can resurrect if I need to. Ubuntu or Fedora should be okay for that. Wait! I think the carcass has a Realtek controller as well. Darn!

BTW, the same situation existed with my 4GB, Core 2 Duo, MSI mobo set up I just replaced but I wrote it off as an infrequent occurrence. The MSI motherboard also had a RealTek net controller.

My original v8.x machine was on an Intel motherboard with a dual port Intel PCI NIC and some brand of hardware RAID controller I got at a flea market. It worked well as a server for several years but made me nervous on the data reliability side. I gave that machine to a friend who put it up as a server at his church.

I wanted to test with CIFS but I've somehow mucked up something and the CIFS won't start. Reports sanity check failure. I tried as many of the permission and other tricks I've found in the forum to no avail. I'm currently resilvering my last replacement drive. I have a fresh 9.2.1.6 USB stick ready to try after the Intel NIC tests.

I'll report the NIC finding as soon as I can. Thanks again for the information.

Ericloewe: I received your advice while at Fry's. Thanks. They didn't have anything Intel in the NIC aisle.

 

[joeschmuck]

Nope, I'm using the on board RealTek NIC. Also keep in mind that we are using different protocols. I hope the problem is more of a setup issue than the NIC for everyone's sake however I know you would like to see the problem solved.

I did install my Intel NIC add-on card and the throughput was virtually identical, ~62MB/sec and same test as above.

So back to the RealTek NIC and I tried a different file, this one was about 29GB and it didn't fail in the same way you experienced it, the transfer speed significantly slowed down, I mean to a crawl. I was probably 20GB into the transfer when the speed dropped but it didn't stop and I let it crawl for about 10 minutes. While the transfer was going on I was able to transfer a second file but it was extremely slow. Once I terminated the first transfer all transfers were fast again. I never received any error/warning messages.

I don't typically transfer large files from one dataset to another via ethernet, I use the cp or mv command as it's much faster, however now I'm curious if this is a limitation of the RealTek NIC or FreeNAS 9.2.1.6. I will never know because I don't feel like screwing up my pool again migrating between 9.2.1.3 and 9.2.1.6. For the time being I will leave the RealTek NIC operational and continue to test some transfers before I decide if I will stick with the RealTek or Intel NIC. If I come up with some answer, I'll put it out. I'm going to hate admitting that Cyberjock is 100% correct here with respect to RealTek NICs being junk but I will concede if I must.

As for the CIFS sanity check failure, yea, CIFS when moving from 9.2.1.3 to 9.2.1.6 messed up my system and I tried to change permissions to make things right but I ended up rebuilding my entire system (software wise) from the ground up, meaning I blew away my pool as well. Doing this fixed everything. It also forced me into doing some well needed house cleaning.

EDIT: I have transferred a few more files and the throughput drops to about 1.2MB/sec however still no flat out termination nor error messages. I have transferred files strictly to and from my main computer into the NAS and found out something is up, not sure what it is. I'll be ticked off if it has to do with 9.2.1.6 but I'm not going to lay blame there just yet as more testing will be needed. Time to connect just the Intel NIC and see if everything works without issue and rule out my own main computer as the issue.

 

[NiceTry]

EDIT: I have transferred a few more files and the throughput drops to about 1.2MB/sec however still no flat out termination nor error messages. I have transferred files strictly to and from my main computer into the NAS and found out something is up, not sure what it is. I'll be ticked off if it has to do with 9.2.1.6 but I'm not going to lay blame there just yet as more testing will be needed. Time to connect just the Intel NIC and see if everything works without issue and rule out my own main computer as the issue.

Thanks for the follow-up testing. Although I'm sorry for the trouble, I'm happy you've been able to replicate most of this symptom using CIFS.

My idea on the dataset to dataset transfer was to simulate maxed-out simultaneous read-write disk and network action from multiple users as a stress test.

I'm going to a local computer tech shop this morning and try to round up an Intel NIC. Perhaps from his parts bin. I'll hold off rebuilding my system until after I do that testing. I don't want to introduce more variables into this.

Your 16GB memory may be helping you stave this off for a larger file. Eric Thompson, above #70, that successfully tried my test with a 40GB file, has 32Gb with an on-board Intel controller. My symptom with 8Gb was a slowdown after 3-4GB to the low MB/s for awhile then stop. If I terminate the transfer before it stops, it will eventually clear. If I miss the opportunity to stop it before NIC timeout, I have to force a reboot from the NAS keyboard or power switch. I've not waited a long time (more than 10 minutes or so) in that state to see if it recovers on its own.

I'm still rooting for the Intel NIC...

BTW, when you rebuilt your pool, did you delete the pool and then import the volumes or did you actually rebuild from a backup?

 

[joeschmuck]

I had a serious CIFS permissions issue and I tried creating new datasets and that didn't work. Reinstalling FreeNAS 9.2.1.6 new didn't help. I had to destroy my pool and recreate it. For me this wasn't a big problems because all my data fits on about 500GB of space once you trim out the old backups. I've never had to do this in the past when upgrading FreeNAS versions before but for some reason I was running into issues using Acronis backup software but normal windows shares seemed to be working. If it wasn't for the difficulty with all the permissions, I'd drop back to 9.2.1.3 and test it all over again to see if I still experience the RealTek slowdown, but to be honest, I didn't see that issue before but then again I didn't really stress the system out. I have currently disabled the RealTek NIC in the BIOS and am just using the Intel NIC. I will update my signature to include this part. BTW, it only cost me about $30 when I ordered it last year for a test I conducted. http://www.newegg.com/Product/Product.aspx?Item=N82E16833106033