SOLVED Data transfer issues

[CaedenV]

TL;DR Ver
Recently re-did my RAIDz2 array (added another drive), but when transferring data back (simple drag-and-drop) the speed starts out very fast, then crashes down to nothing for a few sec, then speeds up again, then crashes for a bit. During long transfers the crash will cause a time out and error the transfer out.

Long Version:
My FreeNAS box had been off since last Nov when I had 2 drives fail. I recently had enough money to replace the bad drives, and the process went smoothly. Re-Silvered the drive, all was well. Then I got a fun side project which gave me some fun money for upgrades. So I purchased a new motherboard that can hold an 8th HDD, and a drive to fill the slot.
I backed up all the data across several HDDs I am borrowing from friends for the week, destroyed the RAIDz2 setup, set up a new RAIDz2, set permissions, etc. All appeared to go well.
Went to start throwing data back on the fresh RAIDz2 and the first chunk of data (~2.5TB) went well, but then it started hanging. I started getting ~30sec hangs where data was not transmitting, followed by ~5-15 sec of full 113MB/s data flow. On occasion the pauses were long enough where it would error out the transfer and I would have to try again, but when moving TBs of data it becomes painfully slow.
My first thought was that there was perhaps something wrong with the temp drives I am borrowing, but I have tested data transfers to my wife's PC, and those go fine, and these pauses are happening with any connection into the FreeNAS box. So hopefully this means my data is safe :D *phew*

My next thought was that perhaps it was a system resource issue on the server. Checking CPU history (AMD A10 5800), but usage is pretty low, averaging ~10-25% load. RAM usage is pretty well maxed out (20GB, 2x8GB, 2x2GB), and I think I am going to give up 2 8GB sticks from my desktop that I am not really using to bump the server up to 32GB (4x8GB).

The obvious suspicion is that there is an issue either with the motherboard or HDD that I added last, but no errors are poping up in the UI of FreeNAS.

Any thoughts/ideas to help troubleshoot or to get the system to find/surface an existing error that may be causing this weird behavior?

System Info:
FreeNAS 9.10
CPU: AMD A10 5800
Mobo: ASRock FM2A88M Pro3+ FM2+
RAM: 20GB DDR3 2x2GB, 2x8GB
Drives: 5 Seagate consumer drives, 3 HGST server/SATA drives, all 7200 3TB drives
RAID: RAIDz2, 3TBx8 drives ~15GB usable space

Thanks!

 

[maafty]

That is a weird issue...
have you tried taking out all the drives and just putting 1 drive in as a test? using a desktop mobo means the sata ports will all be sharing bandwidth but i cant see that making it pause/hang, that should just make throughput lower, but since you are using 1gb network that will be the main bottleneck
also having a system that supports ECC memory is highly recommended.

not sure what to suggest but what i would try is just 1 stick of ram and 1 hdd, and see how that performs, if u still have issue u could try a different ram stick etc. start basic any build your way up until you see the issue, of course the issue could also be the NIC

hope some of this helps :)

 

[CaedenV]

ha! well, I never get the 'normal' issues lol
ECC ram is in my future; I believe I have dibs on an old work server when they decommission it this fall, which will be all sorts of fun to play with. But for the moment I am on cheap consumer hardware because that is what can be afforded. But the server from work should have ~128GB of ECC... which blows my mind that it is not enough to be useful in our work enviornment lol. Not good enough for work, and perhaps overkill for home.


The more I have thought about it today, the more I am sure that it is the new HDD. I had no issues during the week where I installed the new MoBo, and the new drive is a referb, so it would not be that surprising.


While we are at it... I am wondering if you can point me in the right direction on another potential issue:
At my last job we suffered a series of crypto attacks, and with small kids in the house I am becoming more and more hesitant to have full guest access on the server.
Is there a nice way for me to assign a specific user acct with full read/write/execute access, but still allow Guest access for read/execute access? I tried setting this up, but it seems as though I need to either set up full guest access, or everyone gets real accounts. But we have a lot of people go through our house (if only we could charge friends and family rental space hahahaha) and I like to give them access to the movie and music collection (all legally obtained! very proud of my collection!). But with many of them being 'less than technically literate' the idea of connecting with a different username and password is a no-go, and I am not going to make accts for every person who darkens my door. I have read the user setup guide several times, and either I don't understand it as well as I think I do, or else something is not working quite right.

Anywho; when I get home tonight I will try unplugging the new HDD and see if it runs better with a drive down, and if that doesn't fix it then I will do what you suggested with starting over very simply and slowly build it up to see where things fall apart.

 

[maafty]

loads of ram is the way to go, one of mine has 512GB.

i work in the NHS, know all about cyber attacks.... haha.
Im not sure what the best way to setup the user accounts, i only use my FreeNAS as iSCSI's, they link to my ESX servers (10gbe) into Windows/Linux VMs and file sharing is done from there, so i use AD/Radius authentication as i find that the best and most flexible method.

 

[CaedenV]

OK! so I think I figured it out, and it doesn't appear to be a hardware issue (other than a fan starting to go bad which I mistook for 8 HDDs in a crappy case lol, MUCH quieter now).

So I got home, and totally forgot that my mother-in-law was in town for a visit, and my wife had things to do this evening, so I was very delayed on working on this project. But, here is what I did:

1) removed the 2 2GB sticks of RAM, and took 2 8GB sticks out of my desktop. This brings the server up to 32GB, and it seems snappy again (system is using ~30GB!). FreeNAS certainly runs slow when choked on RAM. I will have to keep this in mind for my next upgrades. Still, while this helped a bit, I was still having the same pausing issues.

2) Removed the new HDD from the mix. Being RAIDz2 I figured I could do a transfer in a degraded state just fine. But I still had the same issue. But something just didn't sound right, like a light friction sound all the time. Turned out to be a bad fan; replaced with a spare and now it is nice and quiet.

3) I realized that this was a re-build of an existing install. I didn't want to go through a whole install process, but I did want to reset things as much to scratch as possible, so I started removing stuff. I destroyed the pool, reset settings back to default on various menus, etc.

This is when I found that Plex was still installed and running, so I turned it off, re-built the pool with as few changes as possible to defaults, set permissions for guest access, and lo and behold! Everything is working again! I am not getting the super fast 113MB/s I was getting previously... but a consistent 105-108 is certainly not bad (I think some SMB settings were giving me a little performance boost). I am now 100GB into my first transfer and everything is behaving as it should.

The fun bit though is; turned plex back on, and with every title it found it would mess with the system and cause a time-out as it gathered artwork and such. Turn it back off, and things ran well again. So I am leaving it off for now, and I will kick it back on after I am done moving into the system again.

Really hope that helps someone out there down the road!

 

[CaedenV]

loads of ram is the way to go, one of mine has 512GB.

i work in the NHS, know all about cyber attacks.... haha.
Im not sure what the best way to setup the user accounts, i only use my FreeNAS as iSCSI's, they link to my ESX servers (10gbe) into Windows/Linux VMs and file sharing is done from there, so i use AD/Radius authentication as i find that the best and most flexible method.

Oh boy! you guys as NHS really got hit! I feel for you! Up until a month ago I was working in a few school districts. One I was directly in charge of for about two years before the big ones hit, so I had plenty of time to play and learn in the environment and fix old permissions issues, update servers, etc. They were compromised, but the issues were contained to specific user folders and it really wasn't a big deal (dont even think my boss found out about that one now that I think about it). The other school I was sort of in charge of was a mess. The last week I worked there they were hit with 2 different attacks, and both compromised all of the network file shares. Thankfully the local tech was keeping manual backups of the important stuff, and I was there long enough so teacher's 'home' drives were secure (except for the few infected teachers), but it was such a pain. Not all of the shares were documented, and the local tech had been giving teachers and staff local admin rights, so the whole thing kept falling apart!
My new job is in a VDI environment with honeypots. When stuff gets encrypted in the pots, the IT staff gets and email, the user gets logged out before they can do much harm, and then their machine is restarted (which gives them a fresh install of Windows). The whole thing is largely automated and seems to work rather well. Then we roll back the server changes by a few minutes and all the damage is undone. Really wish my old job could afford that kind of setup! It is really slick!

Ya, I think I need to set up an AD/Radius setup. I was on the Server 2016 beta for a while, but my spare box went to help a friend whose PC died shortly after I started playing with it. Maybe I can scrounge enough parts to build another and get back in the beta... or play with Zential or something similar so I can finally learn a little linux while I am at it.

 

[maafty]

Oh boy! you guys as NHS really got hit! I feel for you! Up until a month ago I was working in a few school districts. One I was directly in charge of for about two years before the big ones hit, so I had plenty of time to play and learn in the environment and fix old permissions issues, update servers, etc. They were compromised, but the issues were contained to specific user folders and it really wasn't a big deal (dont even think my boss found out about that one now that I think about it). The other school I was sort of in charge of was a mess. The last week I worked there they were hit with 2 different attacks, and both compromised all of the network file shares. Thankfully the local tech was keeping manual backups of the important stuff, and I was there long enough so teacher's 'home' drives were secure (except for the few infected teachers), but it was such a pain. Not all of the shares were documented, and the local tech had been giving teachers and staff local admin rights, so the whole thing kept falling apart!
My new job is in a VDI environment with honeypots. When stuff gets encrypted in the pots, the IT staff gets and email, the user gets logged out before they can do much harm, and then their machine is restarted (which gives them a fresh install of Windows). The whole thing is largely automated and seems to work rather well. Then we roll back the server changes by a few minutes and all the damage is undone. Really wish my old job could afford that kind of setup! It is really slick!

Ya, I think I need to set up an AD/Radius setup. I was on the Server 2016 beta for a while, but my spare box went to help a friend whose PC died shortly after I started playing with it. Maybe I can scrounge enough parts to build another and get back in the beta... or play with Zential or something similar so I can finally learn a little linux while I am at it.

Luckily at our hospital we were ok, we have a really good team who are on the ball so we had no issues, we did patch quite a few servers afterwards just to make sure all is up to date, issue most hospitals had was they had port 445 open?!?!?!?!
VDI is great fun, i even have a VDI (Horizon View) at home, great if u need a windows desktop on the go on phone/tablet.
Most of my work is linux/unix based, its fairly easy to get used to, plenty of guides and help out on the web too, you can always drop me a message if u need any linux help.

Glad you sorted ur FreeNAS issue out, i too have Plex but its on its on VM in my ESX cluster so never has any issues :)

a tip that may help you squeeze that extra MB/s would be to enable jumbo frames, providing your NICs and switches etc support it, not sure how much you know on the networking side but basically rather than each packet being standard size of up to 1500bytes you can set it to 9000bytes, of course you will never get more than the theoretical max of 125MB/s (on 1gb) but ive seen 123MB/s, of course overheads etc will take up that last 2Mb/s but you can send the same data in less packets thus lowering the overheads to squeeze that last bit of speed out.

you can also lagg multi NICs (LACP) but that doesnt increase the connection to 250MB/s (based on 2 NICS) your overall bandwidth will be 250MB/s but only 125MB/s per connection, unless you setup multipath iSCSI using round robin then you can get the combined speeds, but think iSCSI is the only way to multiply speeds of combined NICs - i use this on some of my ESX iSCSI links and tested it fully and i get 240MB/s, it also works as resilience, if 1 connection drops it all falls back to 1 NIC at 123MB/s, of course you have to take into account speed of your HDDs as that could end up being a bottleneck, i use SAS-12gbps DualPort disks in mirored vdevs so i get very good speeds, my main link to my iSCSI is on 10gb and i get 800MB/s R/W :D

anyway thats enough me waffling on!

 

[CaedenV]

Luckily at our hospital...

you can also lagg multi NICs (LACP) but that doesnt increase the connection to 250MB/s (based on 2 NICS) your overall bandwidth will be 250MB/s but only 125MB/s per connection, unless you setup multipath iSCSI using round robin then you can get the combined speeds, but think iSCSI is the only way to multiply speeds of combined NICs - i use this on some of my ESX iSCSI links and tested it fully and i get 240MB/s, it also works as resilience, if 1 connection drops it all falls back to 1 NIC at 123MB/s, of course you have to take into account speed of your HDDs as that could end up being a bottleneck, i use SAS-12gbps DualPort disks in mirored vdevs so i get very good speeds, my main link to my iSCSI is on 10gb and i get 800MB/s R/W :D

anyway thats enough me waffling on!

My 'switch' is currently just an ASUS router, I believe it can do Jumbo Frames, but LACP is not an option (though I have set up LACP in work environments plenty of times, not terribly difficult to do). I am mostly waiting for a switch with 2 10gig ports to come down to the sub $200 mark before I buy a real switch. 1 10gig for the server, and 1 for my desktop, and a couple of cards. Should work quite nicely, and they are coming down in price. Probably another year or so.

Also, when I checked the server this morning, my data transfer went through quite nicely... but the server decided it needed to disconnect 2 HDDs for some reason... so there is yet more work to be done :( the joys of budget parts.

Anywho, that will be a different issue for a different thread if I need help on it, but I think I can troubleshoot this one once I get a chance to take a real look at it.

Thanks again for all the help!

 

[SweetAndLow]

Realtek nic? If so then that is your problem.

Sent from my Nexus 5X using Tapatalk