SOLVED Copying 1GB Files From NAS to Windows Causes Network Crash

[Baptiste]

So I'm curious, your system passed burn-in testing with flying colors but is crashing under load with FreeNAS?

Or did you not do any burn-in testing?

https://forums.freenas.org/index.php?threads/building-burn-in-and-testing-your-freenas-system.17750/

Like state above I have not run a burn-in test since building the NAS, but the hardware in there is all from my old build which I had run various burn-in tests (memtest86, passmark burn it in, Cinebench) and had passed all of it.

 

[Baptiste]

Quick recap:

Run Live Ubuntu on Client, make RAM disk and see if issue continues.

Run Memtest86 on the server.

I'll report back when I've done those.

As a side note, if one of you fine folks has Teamviewer or Skype we can possibly arrange a remote control so you could see if I messed a setting up or something?

 

[BigDave]

I do want to say, the RAM, CPU, and Mobo in the NAS was my old build which I just swapped this weekend for the new fancy Intel CPU and Asus board in my Desktop. And I never had RAM issues with it before.

Were these old desktop components now being used for FreeNAS, once water cooled & overclocked like your new desktop hardware is now?
If the answer is yes, then all the more reason to check/test/validate for a hardware shortcoming and/or failure. I'd start with the RAM...

 

[Baptiste]

They were never water

Were these old desktop components now being used for FreeNAS, once water cooled & overclocked like your new desktop hardware is now?
If the answer is yes, then all the more reason to check/test/validate for a hardware shortcoming and/or failure. I'd start with the RAM...

They were never water cooled but had a slight OC for a little bit. I of course reset the BIOS and removed any OC and custom settings before building the NAS.

Running Memtest86 right now on NAS, doing Hammer Test 2 passes. Will let you know result. That being said, it's been running for 5 minutes and no progress has changed, weird. And yes, Memtest86 detects the 2 chips, 8GB.
EDIT: After some online reading, turns out the Hammer test has a known bug, gonna select instead test 0-8 at 4 passes.

Gonna install Ubuntu Live meanwhile and try out this RAM disk.

 

[jgreco]

Like state above I have not run a burn-in test since building the NAS, but the hardware in there is all from my old build which I had run various burn-in tests (memtest86, passmark burn it in, Cinebench) and had passed all of it.

So you took it apart and rebuilt it, and never burned it in, thinking that your previous build's burn-in is somehow relevant.

Ah. Yeah. Not.

 

[Baptiste]

In Ubuntu now with RAM disk created. Waiting on Memtest86 to finish.

 

[Baptiste]

So you took it apart and rebuilt it, and never burned it in, thinking that your previous build's burn-in is somehow relevant.

Ah. Yeah. Not.

It wasn't really a rebuild, when removing the old components, I left the RAM, CPU and CPU heat Sync, all attached to the Mobo. Those parts never even moved. The only hardware change was removing the Graphics card to put in my Desktop, and adding the 5 1Tb Western Digital drives. And removing the CMOS battery to reset settings, but that's not a big deal.
EDIT: Yes the SMART status of the 5 drives checks out, I used HD Sentinel to check each drive individually.

 

[Baptiste]

After 2 hours and 30 minutes memtest86 finished, no issues. I attached a screenshot of the log.

 

[Baptiste]

Can't get NAS to show up in WebGUI under Ubuntu... Not sure why, tried Intel NIC and RealTec NIC which each has their own IP, could not connect. Will try bashing my head against it tomorrow some more.
If someone would like to tell me if Ubuntu has different rules or something for FreeNAS, please do let me know.

 

[jgreco]

It wasn't really a rebuild, when removing the old components, I left the RAM, CPU and CPU heat Sync, all attached to the Mobo. Those parts never even moved.

Did you move the mainboard to another chassis? Use a different power supply? (Didn't use a different power supply?) Etc?

After 2 hours and 30 minutes memtest86 finished, no issues. I attached a screenshot of the log.

So when are you going to actually run the full memtest?

2 hours of running memtest. Kidding, right? Try two WEEKS (or even months if you're suspicious of it). Especially when you're not using ECC memory, this is your *only* opportunity to detect and reject marginal memory. Bad memory often doesn't faze Windows, it may just silently corrupt something. Not as good a thing for NAS, especially one that is trying hard to protect its data.

Using a non-recommended consumer grade platform is hazardous in general. While loads of PC hardware vaguely qualifies as "FreeBSD" compatible, certain bits of hardware such as Realtek ethernets are known to be problematic. You've got a trifecta of the first three things on the list in the How to Fail guide.

See, the thing here is that you're not really understanding. There's a disconnect of some sort. You're saying that your NAS is having a ${problem} that usually tracks back to hardware, but you're showing extreme resistance to the idea that maybe you should be doing the things that are normally done to validate a platform prior to using it for FreeNAS. You've chosen to use a platform that we've seen be problematic for many people in the past, and one that's been stressed through overclocking at that.

None of this means that you absolutely cannot make this work. It means what you've got is suboptimal. It might mean it cannot be made to work, but even there, we're trying to drill down, playing "find the problem." But please try to take what we're saying seriously.

First, install the Intel ethernet card and do not use anything else until we identify where the problem is. This reduces the chances of the Realtek ethernet being a problem. A network adapter should never "fall off the network."

Second, post a full hardware manifest, including what you're using for a power supply, because this could easily be a case of using too-small a PSU (you should be using probably at least a 550W PSU).

Third, be prepared to be asked to do some more burn-in testing, because what you're reporting for problems is ambiguous at best.

Fourth, actually maybe second-and-a-halfth, see if you can arrange to test file transfers between your Windows and something else, and also between that something else and the FreeNAS, to see if maybe the problem's somewhere else.

 

[Baptiste]

Did you move the mainboard to another chassis? Use a different power supply? (Didn't use a different power supply?) Etc?

Different Power supply. And yes they moved cases, by not moved I mean I never removed them from the motherboard.

So when are you going to actually run the full memtest?

2 hours of running memtest. Kidding, right? Try two WEEKS (or even months if you're suspicious of it). Especially when you're not using ECC memory, this is your *only* opportunity to detect and reject marginal memory. Bad memory often doesn't faze Windows, it may just silently corrupt something. Not as good a thing for NAS, especially one that is trying hard to protect its data.

The memory (and rest of the components) in there was used as my main desktop over well over a year, if bad memory was there I figured maybe I'd have noticed it? But I am willing to run a more extensive memory test, possibly over night while I sleep, maybe tonight and all throughout tomorrow, giving it at least 24 hours.

Using a non-recommended consumer grade platform is hazardous in general. While loads of PC hardware vaguely qualifies as "FreeBSD" compatible, certain bits of hardware such as Realtek ethernets are known to be problematic. You've got a trifecta of the first three things on the list in the How to Fail guide.

Yes I understand that this is all consumer grade products but I feel like most (not all of course) FreeNAS users are probably using the consumer grade equipment too. I don't have a grand to dump into a server board with ECC RAM and a Xeon. Broke, like just graduated last month out of college with hundreds of thousands in debt kinda broke.

See, the thing here is that you're not really understanding. There's a disconnect of some sort. You're saying that your NAS is having a ${problem} that usually tracks back to hardware, but you're showing extreme resistance to the idea that maybe you should be doing the things that are normally done to validate a platform prior to using it for FreeNAS. You've chosen to use a platform that we've seen be problematic for many people in the past, and one that's been stressed through overclocking at that.

jgreco, I don't see how I've been showing extreme resistance... I know I can sometimes be very stubborn, but I would like to think here that in this forum I have done everything that was asked of me. I listed specifications, steps taking, results, spent my own money on a users recommendation. I've done the tests recommend to my by users and continue to do them. I'm not a FreeNAS master user by any means, in fact I'm a complete noob at it, but you have to understand that I am trying. I'm, trying my best that I can with my abilities and with what materials I have.

None of this means that you absolutely cannot make this work. It means what you've got is suboptimal. It might mean it cannot be made to work, but even there, we're trying to drill down, playing "find the problem." But please try to take what we're saying seriously.

I am, and I appropriate each and every post posted by the users in this forum, they have been very kind and helpful. I don't wish to waste yours, or anyone else time.

First, install the Intel ethernet card and do not use anything else until we identify where the problem is. This reduces the chances of the Realtek ethernet being a problem. A network adapter should never "fall off the network."

Got it, will only user Intel NIC from now on and its IP. Want me to delete the network interface from FreeNAS for the Realtek NIC?

Second, post a full hardware manifest, including what you're using for a power supply, because this could easily be a case of using too-small a PSU (you should be using probably at least a 550W PSU).

DESKTOP (CLIENT) FULL MANIFEST:

OS: Windows 10 Education
Case: Corsair 760T
CPU: Core i7 4770K (OC 4.5GHz, 45 Clock speed with I believable 1.200V, maybe 1.250V. But it is not consistent 4.5GHz. Under Idle goes between 1GHz and 2.8Ghz, saw it go to .9GHz a couple times. If you're wondering about temps, 29C Idle and 70C under extreme stress fo all 8 threads maxed out at 4.5GHz for extended time.)
Motherboard: Asus Sabertooth z87 (The board was actually recently shipped to Asus under warranty for a check and came back 100 percent fine. I sent the board out as I had bought it from a friend and wanted to confirm it was working fine before buying.)
Graphics Card: Nvidia 750 ti
RAM: Crucial Ballistix sport 24GB DDR3 ( 2 X 8GB, 2 X 4GB)
CPU Cooler: Corsair H115i
SSD: 128GB (Boot)
SSD: 500GB (Media)
HDD: 1TB (Games)
HDD: 1TB (Games)
HDD: 1TB (Dropbox)
HDD 1TB (Scratch Disk for editing)
HDD 1T (Temp Drive using cause Media SSD is full)
Power Supply: Cooler Master V550 Semi Modular
Disc Drive: LG M-Disc
Docking Station: Startech (I think there's no name on it) 2.5 and 3.5 HDD Docking Station
Misc PCI: eSATA to SATA card
Misc: 2 X Blue LEDs for effect

NAS (SERVER) FULL MANIFEST:

OS: FreeNAS 9.10-Stable
Case: Fractal Design Node 804
CPU: A8-6600K
Motherboard: Gigabyte F2A88XM-D3H
RAM: Ripjaws 8GB DDR3 (2 X 4GB)
CPU Cooler: Hyper 212 Evo
HDD: Western Digital Blue 5 X 1TB (Raid z2)
Power Supply: Corsair CX 650M
USB: Jetflash Transcend 8GB (Boot)
NIC: Intel Gigabit CT Desktop Adapter - OEM (http://www.newegg.com/Product/Product.aspx?Item=N82E16833106273)

That should be all of it.

Third, be prepared to be asked to do some more burn-in testing, because what you're reporting for problems is ambiguous at best.

Fourth, actually maybe second-and-a-halfth, see if you can arrange to test file transfers between your Windows and something else, and also between that something else and the FreeNAS, to see if maybe the problem's somewhere else.

Sorry if it's ambiguous, again, trying my best.

Yes like I said in my best before, was having issue with Ubuntu last night and will try again today. Was also thinking of temporarily installing Xpenology and transferring from there to Windows to see if that works. I'll hopefully report back soon with results.

Again, thank you and every user in this forum for the help.

 

[Baptiste]

I have a very long post above this one that is "awaiting moderator approval, and is invisible to normal visitors" so not sure if this message will be posted before the other one. If so, maybe just wait for the other one to pop up before reading this. Also, why does it need mod approval, is it cause it has a hyperlink in it?

I have some results to share. I booted the Windows Desktop into Ubuntu and got a 16GB RAM disk working and figured out how to connect to FreeNAS from Ubuntu. Did the 10MB/s transfer, worked fine. Did the 100MB/s transfer, worked fine.
Just a small reminder, I am using a 1GB movie file for the transfer tests, and in Windows I use that 1GB movie file and Blackmagic Speed Test as I can just leave it running without me having to keep dragging and dropping files. I did it a couple times for about 15 minutes, just moving files back and forth, no issues.
Went back to Windows, did 10MB/s, worked fine. Did the 100MB/s, crashed within 2 minutes.
I'll try booting Xpenology on the NAS and seeing if that works but I'll have to do that later today, need to head out right now.

Side Note 1: I forgot to mention this before, but for some odd reason, I can't just boot the NAS and Windows already directly attached and have them work, they do not detect. I need to plug both first into my piece of shit 10MB/s router, and then plug them in directly, and bam, connects. It's the weirdest thing, not sure at all what it means.

Side Note 2: While I was in Ubuntu I saw this pop up on the NAS screen, not sure what any of it means, see pic attached.

 

[maglin]

Turn the Realtek NIC off in the BIOS. I think something is wrong with Windows. You can create a CIFS share and use black magic on that network folder. It's dirty and not as good as other solutions but is quick. I use it for gee wiz stuff. I just tried it on my CIFS share and get 90-96 MB/s. Which is what I get when I have my unmanaged network hub attached. It really slows down my entire network.

Try removing the NIC drivers from Windows and reinstalling them. And don't change any settings in the driver properties. Also make sure you don't have jumbo frames turn on. It was mentioned earlier and I want to hit on it again. It almost always is slower with it on.


Sent from my iPhone using Tapatalk

 

[Baptiste]

Turn the Realtek NIC off in the BIOS. I think something is wrong with Windows. You can create a CIFS share and use black magic on that network folder. It's dirty and not as good as other solutions but is quick. I use it for gee wiz stuff. I just tried it on my CIFS share and get 90-96 MB/s. Which is what I get when I have my unmanaged network hub attached. It really slows down my entire network.

Try removing the NIC drivers from Windows and reinstalling them. And don't change any settings in the driver properties. Also make sure you don't have jumbo frames turn on. It was mentioned earlier and I want to hit on it again. It almost always is slower with it on.


Sent from my iPhone using Tapatalk

I'll see what I can do about turning off the built in RealTek NIC in the BIOS.

I'm sorry, I have no idea what Jumbo Frames is. Is it a Windows or FreeNAS setting?

 

[Baptiste]

Turned off "On board LAN Controller" which is only what I can assume to be the On board NIC being the Realtek.
Ran Blackmagic Speed Test, went fine for a little while and then failed. Funny enough, this time it failed on a write operation. But the rest was the same, had to unplug NAS and plug it back into router to be detected.

 

[ttabbal]

Jumbo frames is a network card Driver setting. Don't mess with it, it's pretty worthless these days. You seem to have some low level stuff having trouble....

Can you try this? If you haven't done so yet, set freenas to a static IP. Say, 10.1.0.2. Netmask /24 or 255.255.255.0, they mean the same thing, so whichever works for the OS you're using. Freenas web UI uses /24.

Now set the client machine static IP to 10.1.0.3. Same netmask. Test to see if you can connect to the web UI. If so, run a cat5 cable directly between the two. Test connection again. If it works, do some tests that way. No switches, routers, hubs, etc involved.

Now, we really need to test raw transfer here. Something like iperf. It's available on Windows as well, you just need to find an exe for it. The Ubuntu live boot will have it by default. Get a console session open on freenas and do "iperf - s". On the client, "iperf -c 10.1.0.2" what does it report? Maybe try a longer run, for the client "iperf -t 100 -c 10.1.0.2".. That should help simulate a large transfer.

 

[jgreco]

Different Power supply. And yes they moved cases, by not moved I mean I never removed them from the motherboard.

The fact you don't think you disturbed the stuff on the top ain't gonna matter if you got somethin' shortin' out on the bottom. Using a different PSU and installing into a different chassis definitely qualifies as a major rebuild.

The memory (and rest of the components) in there was used as my main desktop over well over a year, if bad memory was there I figured maybe I'd have noticed it?

Depends.

Yes I understand that this is all consumer grade products but I feel like most (not all of course) FreeNAS users are probably using the consumer grade equipment too.

No, really, most of them aren't. Back in the early transitional days from FreeNAS 0.7 to the new design, that was true, and things were bad. A bunch of us spent a lot of time explaining to people why Realtek ethernets are bad, why ECC is virtually mandatory, and why server grade boards are the right way to go. I spend a lot less time today listening to people whine about how their consumer board with Realtek isn't up to snuff than I did four or five years ago, so I think I can reasonably say that our campaign of documentation and encouragement has been extremely successful.

I don't have a grand to dump into a server board with ECC RAM and a Xeon. Broke, like just graduated last month out of college with hundreds of thousands in debt kinda broke.

And reality doesn't give a crap. Sorry, that's just the way it is. As much as I'd love for you to be able to run FreeNAS on a $25 Raspberry Pi, FreeNAS is intended to be a high performance large scale filer, and it isn't going to run on a janky 2005 era 32 bit PC with 1GB of RAM the way lots of people want it to. As a matter of fact, it isn't going to run particularly well on a lot of contemporary consumer grade stuff. This is because the consumer grade board will have things like Realtek ethernets, CPU-based video, no ECC, and a duty rating that's only fit for a grandma that uses her computer twice a week. The server grade board will have dual Intel ethernets, ECC, IPMI with onboard video, PCIe slots that are sensible to server use, the ability to manage power in useful ways, etc.

 

[Baptiste]

Jumbo frames is a network card Driver setting. Don't mess with it, it's pretty worthless these days. You seem to have some low level stuff having trouble....

Can you try this? If you haven't done so yet, set freenas to a static IP. Say, 10.1.0.2. Netmask /24 or 255.255.255.0, they mean the same thing, so whichever works for the OS you're using. Freenas web UI uses /24.

Now set the client machine static IP to 10.1.0.3. Same netmask. Test to see if you can connect to the web UI. If so, run a cat5 cable directly between the two. Test connection again. If it works, do some tests that way. No switches, routers, hubs, etc involved.

Now, we really need to test raw transfer here. Something like iperf. It's available on Windows as well, you just need to find an exe for it. The Ubuntu live boot will have it by default. Get a console session open on freenas and do "iperf - s". On the client, "iperf -c 10.1.0.2" what does it report? Maybe try a longer run, for the client "iperf -t 100 -c 10.1.0.2".. That should help simulate a large transfer.

I've been using 192.168.1.105 as my Static IP for the Intel NIC, is that much different then 10.10.2? I know 192.1.68.X.X is a local IP based on network, verses global IP seen from the outside, what kind of IP is 10.X.X.X, is it a IPv4 setting? The current netmask is 255.255.0.0.

 

[Baptiste]

The fact you don't think you disturbed the stuff on the top ain't gonna matter if you got somethin' shortin' out on the bottom. Using a different PSU and installing into a different chassis definitely qualifies as a major rebuild.

I may not be too savvy when it comes to a lot of things, but I worked at the computer repair center at my college for 2 years and I am extremely confident in my ability to properly handle gear and install it. I can guarantee there is no shorting out of any kind happening. If the chassis change and changing PSU counts as a major rebuild, what burn-in test (I assume this is what you're hinting at) do you want me to run?

I spend a lot less time today listening to people whine about how their consumer board with Realtek isn't up to snuff than I did four or five years ago, so I think I can reasonably say that our campaign of documentation and encouragement has been extremely successful.

Well at least you can take solace in knowing I'm not complaining about the gear ;)

And reality doesn't give a crap. Sorry, that's just the way it is. As much as I'd love for you to be able to run FreeNAS on a $25 Raspberry Pi, FreeNAS is intended to be a high performance large scale filer, and it isn't going to run on a janky 2005 era 32 bit PC with 1GB of RAM the way lots of people want it to. As a matter of fact, it isn't going to run particularly well on a lot of contemporary consumer grade stuff. This is because the consumer grade board will have things like Realtek ethernets, CPU-based video, no ECC, and a duty rating that's only fit for a grandma that uses her computer twice a week. The server grade board will have dual Intel ethernets, ECC, IPMI with onboard video, PCIe slots that are sensible to server use, the ability to manage power in useful ways, etc.

Listen, jgreco, my gear isn't perfect and amazing and won't make an Intel employee piss their pants in glee, but it works well and is in very good condition. With the exception of dust it looks brand new. It's not no 1GB DDR2 Intel atom 900 mhz Dell laptop from the 2005s with a floppy drive in it either. It's decent consumer grade equipment. And I have almost no money to change that (except small change like the $30 Intel NIC) and this is what I have to work with. No matter how much you tell me it's only consumer grade and how many good points you make about ECC, IPMI, and dedicate server PCI slots. Like you said, reality doesn't give a crap. And the reality is I can't change the gear.

I am only asking for help to a not so fun situation. I've done everything the users in this forum have asked for, given helpful feedback and waiting for more posts. If you would like to help me, please do so and thank you very much, I'd appropriate it very much. But repeatedly telling me that my gear is in cahoots isn't very constructive.

 

[Baptiste]

Ok, so I have some more news.

As a test I wiped FreeNAS and installed Xpenology. The 10MB/s test worked fine, BUT much like FreeNAS it failed on the 100MB/s. So what does this mean?

FreeNAS + Mac OS = PASS
FreeNAS + Ubuntu (Running on the Windows 10 hardware) = PASS
FreeNAS + Windows 10 = FAIL
Xpenology + Windows 10 = FAIL

A pattern quickly starts to show...
I don't know about the rest of you guys (and please feel free to correct me) but it doesn't seem like a hardware issue at all but a software issue. Tomorrow I will try to install a fresh copy of Windows 10 onto a USB and try that. And if that too fails I will try a copy of Windows 7.

What do you guys think? Could it be Widows 10 has some secret broken issue somewhere that I just need to wait and see if Microsoft will update it? Or some stupid hidden setting that needs to be turned on? Or something else?