Register for the iXsystems Community to get an ad-free experience and exclusive discounts in our eBay Store.
HDD Spindown Timer

HDD Spindown Timer 1.3

Joined
Jun 26, 2019
Messages
12
Thanks
2
#1
ngandrass submitted a new resource:

HDD Spindown Timer - Monitors drive I/O and forces HDD spindown after a given idle period. Resistant to S.M.A.R.T. reads.

Disk spindown has always been an issue for various FreeNAS users. This script utilizes iostat to detect I/O operations (reads, writes) on each disk. If a disk didn't receive reads or writes for a given period of time it is considered idle and gets spun down.

This excludes periodic reads of S.M.A.R.T. data performed by the smartctl service which therefore enables users to have S.M.A.R.T. reporting turned on while still being able to...
Read more about this resource...
 

Ender117

FreeNAS Experienced
Joined
Aug 20, 2018
Messages
216
Thanks
39
#5
quick question, how do you know the drive has spun down? expect listening?
 
Joined
Jun 26, 2019
Messages
12
Thanks
2
#7
quick question, how do you know the drive has spun down? expect listening?

You can use camcontrol to check if a device has spun down properly. The following commands check whether $device (e.g. ada0) is spinning (output is 1) or spun down (output is 0):
Code:
camcontrol cmd $device -a "E5 00 00 00 00 00 00 00 00 00 00 00" -r - | cut -d " " -f 10 | sed "s/FF/1/" | sed "s/00/0/"

Further details are outlined in this thread: https://www.ixsystems.com/community...ut-if-a-drive-is-spinning-down-properly.2068/

I hope this helps :)
 
Joined
Jun 26, 2019
Messages
12
Thanks
2
#8
Which application we can use for IOPS monitoring.
I'm not quite sure that I fully understand your question but the spindown timer script doesn't interfere with the reporting module of FreeNAS. Therefore you can see historical IOPS data for each disk in the reporting section of your FreeNAS GUI. Just select "Disk" as a category and have a look at the "disk_ops" metric. Please refer to the documentation for further details: https://www.ixsystems.com/documentation/freenas/11.2-U6/reporting.html

Live IOPS-monitoring should be possible using iostat. In fact the script internally also uses iostat to determine whether or not there have been I/O operations on a drive.
 

Ender117

FreeNAS Experienced
Joined
Aug 20, 2018
Messages
216
Thanks
39
#9
You can use camcontrol to check if a device has spun down properly. The following commands check whether $device (e.g. ada0) is spinning (output is 1) or spun down (output is 0):
Code:
camcontrol cmd $device -a "E5 00 00 00 00 00 00 00 00 00 00 00" -r - | cut -d " " -f 10 | sed "s/FF/1/" | sed "s/00/0/"

Further details are outlined in this thread: https://www.ixsystems.com/community...ut-if-a-drive-is-spinning-down-properly.2068/

I hope this helps :)
Good to know, but this seems not work with da drives (HBA attached instead of AHCI), I got greeted by a "error sending command"
Code:
root@freenas:~ # camcontrol cmd da1 -a "E5 00 00 00 00 00 00 00 00 00 00 00" -r -
camcontrol: error sending command
root@freenas:~ # camcontrol cmd da13 -a "E5 00 00 00 00 00 00 00 00 00 00 00" -r -
camcontrol: error sending command
root@freenas:~ # camcontrol cmd ada1 -a "E5 00 00 00 00 00 00 00 00 00 00 00" -r -
50 00 00 00 00 00 00 00 00 FF 00
 
Joined
Jun 26, 2019
Messages
12
Thanks
2
#10
Good to know, but this seems not work with da drives (HBA attached instead of AHCI), I got greeted by a "error sending command"
Which exact drive model are you using? Could it be that it is an SCSI device instead of an ATA one? You can try one of the following solutions:

If you have an ATA drive the following command gives you it's current power mode: camcontrol epc $drive -c status -P. It should return Current power state: Standby_z(0x00) for a spun down drive.

If you are using SCSI drives you should be able to read the modepage 0x1a and have a look at the STANDBY line: camcontrol modepage $drive -m 0x1a. However, since I don't own any SCSI drives I can't confirm this. The pages and their content are described in /usr/share/misc/scsi_modes.

You can see the usage of the commands in the script starting at line 154: https://github.com/ngandrass/freenas-spindown-timer/blob/master/spindown_timer.sh#L154

Let me know if one of the solutions above work for you :)
 

Glorious1

FreeNAS Guru
Joined
Nov 23, 2014
Messages
950
Thanks
173
#11

Ender117

FreeNAS Experienced
Joined
Aug 20, 2018
Messages
216
Thanks
39
#12
Which exact drive model are you using? Could it be that it is an SCSI device instead of an ATA one? You can try one of the following solutions:

If you have an ATA drive the following command gives you it's current power mode: camcontrol epc $drive -c status -P. It should return Current power state: Standby_z(0x00) for a spun down drive.

If you are using SCSI drives you should be able to read the modepage 0x1a and have a look at the STANDBY line: camcontrol modepage $drive -m 0x1a. However, since I don't own any SCSI drives I can't confirm this. The pages and their content are described in /usr/share/misc/scsi_modes.

You can see the usage of the commands in the script starting at line 154: https://github.com/ngandrass/freenas-spindown-timer/blob/master/spindown_timer.sh#L154

Let me know if one of the solutions above work for you :)
Code:
root@freenas:~ # camcontrol epc da13 -c status -P
Current power state: PM0:Active or PM1:Idle(0xff)

root@freenas:~ # camcontrol epc da21 -c status -P
Current power state: Idle_a(0x81)

root@freenas:~ # camcontrol modepage da2 -m 0x1a
PM_BG_PRECEDENCE:  0
STANDBY_Y:  0
IDLE_C:  0
IDLE_B:  0
IDLE_A:  0
STANDBY_Z:  0
IDLE_A Condition Timer:  0
STANDBY_Z Condition Timer:  0
IDLE_B Condition Timer:  0
IDLE_C Condition Timer:  0
STANDBY_Y Condition Timer:  0
CCF Idle:  1
CCF Standby:  1
CCF Stopped:  2


So there are different idle modes? The SAS power mode is not very intuitive.
 
Joined
Jun 26, 2019
Messages
12
Thanks
2
#13
So there are different idle modes? The SAS power mode is not very intuitive.
According to the specification (See: T13/BSR INCITS 529 Revision 14) Standby_y and Standby_z are both substates of the general PM2: STANDBY state a drive can enter. As the standard indicates the power consumption of a drive in Standby_z is less than or equal to the power consumption of the drive in Standby_y state (Standby_y power >= Standby_z power) (See 4.9.2 Power conditions). However, I think the implementation of both states is highly vendor dependent but if you are seeing your drive in either of the standby states you should be fine.

Anyway... None of your drives above seem to be spun down. Have you tried manually spinning down one of the drives an check if it is then reported as spun down correctly? Make sure to have no iops on the drive while performing the test.

To manually spin down an ATA drive issue camcontrol standby $drive or camcontrol stop $drive for a SCSI drive.
 

Ender117

FreeNAS Experienced
Joined
Aug 20, 2018
Messages
216
Thanks
39
#14
According to the specification (See: T13/BSR INCITS 529 Revision 14) Standby_y and Standby_z are both substates of the general PM2: STANDBY state a drive can enter. As the standard indicates the power consumption of a drive in Standby_z is less than or equal to the power consumption of the drive in Standby_y state (Standby_y power >= Standby_z power) (See 4.9.2 Power conditions). However, I think the implementation of both states is highly vendor dependent but if you are seeing your drive in either of the standby states you should be fine.

Anyway... None of your drives above seem to be spun down. Have you tried manually spinning down one of the drives an check if it is then reported as spun down correctly? Make sure to have no iops on the drive while performing the test.

To manually spin down an ATA drive issue camcontrol standby $drive or camcontrol stop $drive for a SCSI drive.
Those drives are actually hot spares and I have set them to spin down after 60min, APM127, AAM medium, not sure why it kept spinning.

Anyway it appears I can manually spin down SATA drives but not SAS (it reports stopped successfully but power state did not change)
Code:
root@freenas:~ # camcontrol standby da21
root@freenas:~ # camcontrol epc da21 -c status -P
Current power state: Standby_z(0x00)
root@freenas:~ # camcontrol epc da16 -c status -P
Current power state: PM0:Active or PM1:Idle(0xff)
root@freenas:~ # camcontrol standby da16
root@freenas:~ # camcontrol epc da16 -c status -P
Current power state: Standby_z(0x00)
root@freenas:~ # camcontrol modepage da4 -m 0x1a
PM_BG_PRECEDENCE:  0
STANDBY_Y:  0
IDLE_C:  0
IDLE_B:  0
IDLE_A:  0
STANDBY_Z:  0
IDLE_A Condition Timer:  0
STANDBY_Z Condition Timer:  0
IDLE_B Condition Timer:  0
IDLE_C Condition Timer:  0
STANDBY_Y Condition Timer:  0
CCF Idle:  1
CCF Standby:  1
CCF Stopped:  2
root@freenas:~ # camcontrol stop da4
Unit stopped successfully
root@freenas:~ # camcontrol modepage da4 -m 0x1a
PM_BG_PRECEDENCE:  0
STANDBY_Y:  0
IDLE_C:  0
IDLE_B:  0
IDLE_A:  0
STANDBY_Z:  0
IDLE_A Condition Timer:  0
STANDBY_Z Condition Timer:  0
IDLE_B Condition Timer:  0
IDLE_C Condition Timer:  0
STANDBY_Y Condition Timer:  0
CCF Idle:  1
CCF Standby:  1
CCF Stopped:  2
root@freenas:~ #


About your script, I see it defaults to work on all drives with option to exclude certain drives. I would be nice if it can do the opposite: only monitor and work on specified drives, like the hot spares for example.
 
Joined
Jun 26, 2019
Messages
12
Thanks
2
#15
Joined
Jun 26, 2019
Messages
12
Thanks
2
#16
Those drives are actually hot spares and I have set them to spin down after 60min, APM127, AAM medium, not sure why it kept spinning.

Anyway it appears I can manually spin down SATA drives but not SAS (it reports stopped successfully but power state did not change)

This is strange... Can you physically check if the drive is spun down after issuing the stop command? Maybe the power information of your specific drive is encoded in the CCF-values at the bottom of the modepage. However, I can't find any information on those values right now :(


About your script, I see it defaults to work on all drives with option to exclude certain drives. I would be nice if it can do the opposite: only monitor and work on specified drives, like the hot spares for example.

I introduced the manual mode (see release notification above). It disables automatic detection of drives and an explicit list of drives to monitor can be specified using the -i switch. Usage examples can be found in the advanced usage section of the README.
 

Ender117

FreeNAS Experienced
Joined
Aug 20, 2018
Messages
216
Thanks
39
#17
This is strange... Can you physically check if the drive is spun down after issuing the stop command? Maybe the power information of your specific drive is encoded in the CCF-values at the bottom of the modepage. However, I can't find any information on those values right now :(





I introduced the manual mode (see release notification above). It disables automatic detection of drives and an explicit list of drives to monitor can be specified using the -i switch. Usage examples can be found in the advanced usage section of the README.
I don't think I can, the disks are in a DAS filled with 20 drives, noise from individual drives is inaudible.

Thanks for introducing manual mode, I will take a look.
 

mgittelman

FreeNAS Experienced
Joined
Dec 8, 2017
Messages
145
Thanks
30
#18
@ngandrass Thanks so much for providing this resource!

I'm having a little bit of trouble understanding what the poll time means. Is that how often the script checks to see what the status of the drives are? How is it different than the timeout period? I also notice that my logs are filling up with messages "unfreezing devq for target" every hour when the default timeout is set to every hour. Could you explain more about what's happening and how the poll time and timeout functions interact?

*Edit - more specfics on my logs:

Code:
Oct 17 19:21:05 nas     (pass20:mpr0:0:20:0): INQUIRY. CDB: 12 01 00 00 ff 00 length 255 SMID 533 Aborting command 0xfffffe00015f2e30
Oct 17 19:21:05 nas mpr0: Sending reset from mprsas_send_abort for target ID 20
Oct 17 19:21:08 nas mpr0: Unfreezing devq for target ID 20
Oct 17 19:21:15 nas     (pass21:mpr0:0:21:0): INQUIRY. CDB: 12 01 00 00 ff 00 length 255 SMID 499 Aborting command 0xfffffe00015efd50
Oct 17 19:21:15 nas mpr0: Sending reset from mprsas_send_abort for target ID 21
Oct 17 19:21:17 nas mpr0: Unfreezing devq for target ID 21
Oct 17 19:21:24 nas     (pass22:mpr0:0:22:0): INQUIRY. CDB: 12 01 00 00 ff 00 length 255 SMID 443 Aborting command 0xfffffe00015eacd0
Oct 17 19:21:24 nas mpr0: Sending reset from mprsas_send_abort for target ID 22
Oct 17 19:21:27 nas mpr0: Unfreezing devq for target ID 22
Oct 17 19:21:33 nas     (pass23:mpr0:0:23:0): INQUIRY. CDB: 12 01 00 00 ff 00 length 255 SMID 435 Aborting command 0xfffffe00015ea150
Oct 17 19:21:33 nas mpr0: Sending reset from mprsas_send_abort for target ID 23
Oct 17 19:21:36 nas mpr0: Unfreezing devq for target ID 23
Oct 17 20:21:43 nas     (pass20:mpr0:0:20:0): INQUIRY. CDB: 12 01 00 00 ff 00 length 255 SMID 162 Aborting command 0xfffffe00015d18e0
Oct 17 20:21:43 nas mpr0: Sending reset from mprsas_send_abort for target ID 20
Oct 17 20:21:46 nas mpr0: Unfreezing devq for target ID 20
Oct 17 20:21:53 nas     (pass21:mpr0:0:21:0): INQUIRY. CDB: 12 01 00 00 ff 00 length 255 SMID 322 Aborting command 0xfffffe00015dfee0
Oct 17 20:21:53 nas mpr0: Sending reset from mprsas_send_abort for target ID 21
Oct 17 20:21:55 nas mpr0: Unfreezing devq for target ID 21
Oct 17 20:22:02 nas     (pass22:mpr0:0:22:0): INQUIRY. CDB: 12 01 00 00 ff 00 length 255 SMID 211 Aborting command 0xfffffe00015d5f50
Oct 17 20:22:02 nas mpr0: Sending reset from mprsas_send_abort for target ID 22
Oct 17 20:22:04 nas mpr0: Unfreezing devq for target ID 22
Oct 17 20:22:11 nas     (pass23:mpr0:0:23:0): INQUIRY. CDB: 12 01 00 00 ff 00 length 255 SMID 396 Aborting command 0xfffffe00015e6940
Oct 17 20:22:11 nas mpr0: Sending reset from mprsas_send_abort for target ID 23
Oct 17 20:22:14 nas mpr0: Unfreezing devq for target ID 23


Also getting:

Code:
(pass21:mpr0:0:21:0): INQUIRY. CDB: 12 01 00 00 ff 00
(pass21:mpr0:0:21:0): CAM status: SCSI Status Error
(pass21:mpr0:0:21:0): SCSI status: Check Condition
(pass21:mpr0:0:21:0): SCSI sense: NO SENSE asc:0,0 (No additional sense information)
camcontrol: The epc subcommand only works with ATA protocol devices
(pass22:mpr0:0:22:0): INQUIRY. CDB: 12 01 00 00 ff 00
(pass22:mpr0:0:22:0): CAM status: SCSI Status Error
(pass22:mpr0:0:22:0): SCSI status: Check Condition
(pass22:mpr0:0:22:0): SCSI sense: NO SENSE asc:0,0 (No additional sense information)
camcontrol: The epc subcommand only works with ATA protocol devices
(pass23:mpr0:0:23:0): INQUIRY. CDB: 12 01 00 00 ff 00
(pass23:mpr0:0:23:0): CAM status: SCSI Status Error
(pass23:mpr0:0:23:0): SCSI status: Check Condition
(pass23:mpr0:0:23:0): SCSI sense: NO SENSE asc:0,0 (No additional sense information)
camcontrol: The epc subcommand only works with ATA protocol devices


Thank you!!
 
Last edited:
Joined
Jun 26, 2019
Messages
12
Thanks
2
#19
I'm having a little bit of trouble understanding what the poll time means. Is that how often the script checks to see what the status of the drives are? How is it different than the timeout period?
There are two timing parameters you are able to specify. The most important one is the timeout (-t TIMEOUT). This determines the number of seconds a drive has to experience no reads or writes before being considered as idle and spun down. Therefore when setting TIMEOUT to 3600 seconds, a drive that hasn't been used for at least one hour will be spun down.

Whether or not a drive experiences I/O is monitored using iostat. The poll time (-p POLL_TIME) determines how long a single iostat call waits before evaluating if there have been reads or writes during the last time period. This is necessary since the idle timer for each drive can only be updated in-between the iostat calls. Therefore the value of POLL_TIME simply specifies how long a single iostat call listens for I/O before the script updates the per disk idle counters. Generally a value between 60 and 600 seconds should be suitable for most applications :)

A quick note regarding the POLL_TIME value selection: The worst case time a drives idle status is detected is TIMEOUT + POLL_TIME seconds since we are only able to update the disk idle counters in-between the iostat calls.

I also notice that my logs are filling up with messages "unfreezing devq for target" every hour when the default timeout is set to every hour.
I'm not quite sure whats going on here. If I understand the situation correctly you are using SATA drives connected trough an SAS controller, correct? This somehow seems to result in the drives being detected as ATA drives but understanding only SCSI commands because of the SAS controller.

Can you provide the full output of all the following commands for at least one of your drives?

Code:
$ camcontrol identify $DRIVE
$ camcontrol epc $DRIVE -c status -P
$ camcontrol modepage $DRIVE -m 0x1a

P.S.: Unfortunately I only have access to my home FreeNAS machine which exclusively uses directly connected SATA drives. Therefore I can't test SCSI/SAS setups right now and possible fixes are based on the documentation I'm able to find online :(
 
Top