Slow Transfer Speeds . . . AFTER the disks wake from "HDD Standby" - Why?

[TDPsGM]

Howdy folks.

This is kinda odd; any transfers that I do "Immediately" after the system starts I am very pleased with 75-95Mb/s.

After the Disks go into Standby mode, and then spin up from that, the speeds drop down to 10-12Mb/s.
If I "Re-boot" the FreeNAS system and transfer the files BEFORE the system goes into Standby then the speeds are right back up again.

It doesn't matter if I am using a CIFS share or FTP'ing -- the results are the same for both.

Why?

Any suggestions?

 

[joeschmuck]

What are your drive settings (HDD Standby, ACC, APM)?
Do you have powerd turned on?
Does it sound like the drives are repeatedly spinning up and down during the problem time?

 

[TDPsGM]

HI joeschmuck,

Thanks for the reply - sorry I have been tied up.

Yes I have HDD Standby enabled on all drives - it is set to 5 minutes.
ACC, APM, & and "powered" turned on I am not familiar with. I looked through the Admin panel and I don't see them. Can you point me in the right direction so I can have a look?

As far as spinning up and down, I'll check that now and report back shortly.

Thanks again for the assistance.

 

[TDPsGM]

Re: Slow Transfer Speeds . . . AFTER the disks wake from "HDD Standby" - Why?

Hey joeschmuck,

It does not look like the drives are spinning up and down during the transfer.
All 4 are running steady as far as I can tell.

- - - Updated - - -

hmmmmm, this is weird.

I have all the disks set to spin down after 5 minutes, yet one of them is still running.

Should they all not spin down if I have them all set to go standby after 5 minutes?
Could that be a clue as to what is wrong?

Thanks for the assistance.

 

[joeschmuck]

I'd say 5 minutes is a bit aggressive and you will likely kill your drives early. You are also using more energy by frequently spinning them up. Try a setting of 30 minutes or more depending on how frequently you access the NAS.

When you change the HDD Standby setting (which is where the other hard drive settings are), they are currently not enabled until you reboot your NAS. So if you changed the time on one drive from Never to 5 Mins, you must reboot. This will be changing in the near future but that it true right now.

 

[tingo]

To figure out for sure if spindown is the problem; turn it off and see if the problem disappears (if you haven't done so already).

 

[TDPsGM]

ok - I can change the time to be less aggressive, but I have shut down and restarted several times and that one drive just keeps on spinning. I even tried changing it to 10 minutes (the one that does not spin down) and then rebooted and let the system sit. The other 3 spun down nicely at the 5 minute mark, and after 15-20 minutes this single drive is just humming away (set to 10 minutes).

Has anyone seen that before?

- - - Updated - - -

{Edit}
Sorry tingo - I get it now (turn the HDD Stby OFF). Ok, I'll give that a go, and report back.
{/Edit}

 

[tingo]

No. As part of your faultfinding, temporarily change settings for the disks so that they run the whole time. Next, run your FreeNAS box for some time (a bit longer than the time it took for the problem to appear before) and see if transfer speed is 75 - 90 Mb/s the whole time, or if it goes down.
This way you will know for sure if the problem is related to the spindown of the disks, or not.

 

[TDPsGM]

Good Call tingo. Spin down has nothing to do with the slow transfer speeds - although I am not sure where to look now.

I have enabled the drives to be 'Always On', and then re-booted the system.

On the reboot the transfer speeds were right up there 75+ easily both via ftp and using a CIFS share.

I walked away for about 15-20 minutes and tried another transfer and they were right down again.
I opened the case, and all the drives seem to be up and running - so I'd say it is safe to say that the "spin down" is not the issue.

. . . but now I really don't have any idea were to start.

Any suggestions?

 

[gpsguy]

Since spindown isn't the problem, I'll ask a couple of questions.

Are you using the onboard Realtek NIC? If so, replace it with an Intel pro/1000.

How full is the array (percentage wise)?

 

[TDPsGM]

Are you using the onboard Realtek NIC?

Yes. It is the: Realtek® 8111E , 1 x Gigabit LAN Controller

If so, replace it with an Intel pro/1000.

Ok, . . . why? The transfer speeds are good out of the gate. Has the Realtek been a problem in the past in this way?
If I have a transfer that goes on for 4-5 minutes it doesn't seem to slow down any during that transfer.

How full is the array (percentage wise)?

It is about 40% full.

 

[joeschmuck]

There is nothing wrong with the Realtek 8111E or F, I've done some testing on these (in the performance section).

One thing I haven't heard was if you leave the NAS on overnight, what is the speed in the morning?

Could you post the results of a zpool status for your system?

Also, the transfer speeds decrease, is this only with a read operation [with respect to the NAS], a write operation, or both?

 

[TDPsGM]

Nope. I don't leave the machine on over night.

zpool status -v

Code:

          raidz2-0                                      ONLINE       0     0    
 0                                                                              
            gptid/74271bb8-5554-11e2-9a32-f46d04d98f51  ONLINE       0     0    
 0                                                                              
            gptid/74d1a118-5554-11e2-9a32-f46d04d98f51  ONLINE       0     0    
 0                                                                              
            gptid/75c1c7f5-5554-11e2-9a32-f46d04d98f51  ONLINE       0     0    
 0                                                                              
            gptid/766b7256-5554-11e2-9a32-f46d04d98f51  ONLINE       0     0    
 0                                                                              
                                                                                
errors: No known data errors                                                    
                                                                                
  pool: volume1                                                                 
 state: ONLINE                                                                  
  scan: none requested                                                          
config:                                                                         
                                                                                
        NAME                                          STATE     READ WRITE CKSUM
        volume1                                       ONLINE       0     0     0
          gptid/004b94fb-0755-11e1-be94-f46d04d98f51  ONLINE       0     0     0
                                                                                
errors: No known data errors                                                    
[root@freenas ~]#       

The slow down is both ways - Read And Write

 

[joeschmuck]

I recommend you leave the machine on overnight at least for testing purposes, possibly for a few days. If you keep shutting it down, I'm curious when it will find the time to finish a scrub.

 

[TDPsGM]

Sorry, been away for a bit.

I recommend you leave the machine on overnight at least for testing purposes, possibly for a few days. If you keep shutting it down, I'm curious when it will find the time to finish a scrub.

This may seem like a dumb question, but, does it do a scrub automatically?

I have had it do one once. I manually initiated it and it took about 3-3.5 hrs.

Are you suggesting that it may be trying to do one on its own? If so where are the settings that control that and how often it does them?

 

[cyberjock]

Sorry, been away for a bit.



This may seem like a dumb question, but, does it do a scrub automatically?

I have had it do one once. I manually initiated it and it took about 3-3.5 hrs.

Are you suggesting that it may be trying to do one on its own? If so where are the settings that control that and how often it does them?

You should have it setup to do one at least every month automatically. If you haven't you are a bad admin(spank yourself!). Scrubs really hurt server performance while they are running, but are a necessary evil with ZFS. Not doing them regularly(in accordance with the manual) is a recipe for failure. Most people plan them to run at night or weekends.

 

[TDPsGM]

You should have it setup to do one at least every month automatically. If you haven't you are a bad admin(spank yourself!).

They are "Planned", but I wasn't doing them "Automatically". I have a task that pops up in which I trigger it when it is not in use.

Scrubs really hurt server performance while they are running, but are a necessary evil with ZFS.

THAT is why I plan them to be done when "I" want (and Know) that the load is low . . . however, we've digressed from the problem at hand.

WHY are the transfer speeds dying after a short period of time?
If I keep the transfers gonig, the speed will stay right up there for as long as they are transferring. It seems to be when there is a pause in the action that when we resume that the performance tanks.

 

[cyberjock]

I know that in my system(I had 12 GB of RAM) the system did weird things performance-wise. It got frustrating because I could find nothing "wrong". In fact, nothing was wrong except for the owner. The system was starved for RAM and an upgrade to 20GB fixed my problem. If you can add more RAM to the system temporarily to see if it helps I'd try that. It could be that when the system is left idle ZFS reallocates the read and write cache based on previous usage and is inappropriately starving one over the other.

 

[TDPsGM]

Interesting theory.

I hope that that is not the case though.
I have 8 GB of Ram in there now and don't have room for more.

Any ideas on how we could 'test' this theory?

Maybe there is a way to 'FORCE' set these limits so as not to starve one for the other and solve my current problem, because as long as I start the transfers (and don't stop) the speeds stay right up there (regardless if it is uploaded or downloaded, CIFS or an FTP transfer).

Comments?

 

[cyberjock]

My only comment is to be extremely careful what you try to tweak. Do lots and lots and lots of homework. It is important to keep in mind that the defaults are the defaults for a reason, and you should understand fully the reason why the tweak may help as well as how it could hurt you(or make your data unreliable). People have done very nasty things to their zpools by not doing their homework. I'm not sure I've ever seen someone show up and post a few tweaks that were mind-blowing. Plenty of mind-blowingly stupid, but definitely nothing to warrant me thinking about reading up on the setting. Also, I'm not sure how you could "test" this without actually using more RAM. I chose the "more RAM" option because I had no other ideas despite help from a few very smart friends in the forum.

To be honest, despite my thoroughness with understanding settings before I try them (I don't cut/paste and reboot just to 'test' a setting I found in a forum) I still am very hesitant to ever tweak ZFS. There's only 3 people on the forum that I would ever trust with any tweaks, and I'd only trust them if they were giving me one-on-one support for some imaginary issue I was having.