HDD Spindown Timer

HDD Spindown Timer 2.2.0

ngandrass

Explorer
Joined
Jun 26, 2019
Messages
68
ngandrass submitted a new resource:

HDD Spindown Timer - Monitors drive I/O and forces HDD spindown after a given idle period. Resistant to S.M.A.R.T. reads.

Disk spindown has always been an issue for various FreeNAS users. This script utilizes iostat to detect I/O operations (reads, writes) on each disk. If a disk didn't receive reads or writes for a given period of time it is considered idle and gets spun down.

This excludes periodic reads of S.M.A.R.T. data performed by the smartctl service which therefore enables users to have S.M.A.R.T. reporting turned on while still being able to...

Read more about this resource...
 

Ender117

Patron
Joined
Aug 20, 2018
Messages
219
quick question, how do you know the drive has spun down? expect listening?
 

ngandrass

Explorer
Joined
Jun 26, 2019
Messages
68
quick question, how do you know the drive has spun down? expect listening?


You can use camcontrol to check if a device has spun down properly. The following commands check whether $device (e.g. ada0) is spinning (output is 1) or spun down (output is 0):
Code:
camcontrol cmd $device -a "E5 00 00 00 00 00 00 00 00 00 00 00" -r - | cut -d " " -f 10 | sed "s/FF/1/" | sed "s/00/0/"

Further details are outlined in this thread: https://www.ixsystems.com/community...ut-if-a-drive-is-spinning-down-properly.2068/

I hope this helps :)
 

ngandrass

Explorer
Joined
Jun 26, 2019
Messages
68
Which application we can use for IOPS monitoring.

I'm not quite sure that I fully understand your question but the spindown timer script doesn't interfere with the reporting module of FreeNAS. Therefore you can see historical IOPS data for each disk in the reporting section of your FreeNAS GUI. Just select "Disk" as a category and have a look at the "disk_ops" metric. Please refer to the documentation for further details: https://www.ixsystems.com/documentation/freenas/11.2-U6/reporting.html

Live IOPS-monitoring should be possible using iostat. In fact the script internally also uses iostat to determine whether or not there have been I/O operations on a drive.
 

Ender117

Patron
Joined
Aug 20, 2018
Messages
219
You can use camcontrol to check if a device has spun down properly. The following commands check whether $device (e.g. ada0) is spinning (output is 1) or spun down (output is 0):
Code:
camcontrol cmd $device -a "E5 00 00 00 00 00 00 00 00 00 00 00" -r - | cut -d " " -f 10 | sed "s/FF/1/" | sed "s/00/0/"

Further details are outlined in this thread: https://www.ixsystems.com/community...ut-if-a-drive-is-spinning-down-properly.2068/

I hope this helps :)
Good to know, but this seems not work with da drives (HBA attached instead of AHCI), I got greeted by a "error sending command"
Code:
root@freenas:~ # camcontrol cmd da1 -a "E5 00 00 00 00 00 00 00 00 00 00 00" -r -
camcontrol: error sending command
root@freenas:~ # camcontrol cmd da13 -a "E5 00 00 00 00 00 00 00 00 00 00 00" -r -
camcontrol: error sending command
root@freenas:~ # camcontrol cmd ada1 -a "E5 00 00 00 00 00 00 00 00 00 00 00" -r -
50 00 00 00 00 00 00 00 00 FF 00
 

ngandrass

Explorer
Joined
Jun 26, 2019
Messages
68
Good to know, but this seems not work with da drives (HBA attached instead of AHCI), I got greeted by a "error sending command"

Which exact drive model are you using? Could it be that it is an SCSI device instead of an ATA one? You can try one of the following solutions:

If you have an ATA drive the following command gives you it's current power mode: camcontrol epc $drive -c status -P. It should return Current power state: Standby_z(0x00) for a spun down drive.

If you are using SCSI drives you should be able to read the modepage 0x1a and have a look at the STANDBY line: camcontrol modepage $drive -m 0x1a. However, since I don't own any SCSI drives I can't confirm this. The pages and their content are described in /usr/share/misc/scsi_modes.

You can see the usage of the commands in the script starting at line 154: https://github.com/ngandrass/freenas-spindown-timer/blob/master/spindown_timer.sh#L154

Let me know if one of the solutions above work for you :)
 

Glorious1

Guru
Joined
Nov 23, 2014
Messages
1,211

Ender117

Patron
Joined
Aug 20, 2018
Messages
219
Which exact drive model are you using? Could it be that it is an SCSI device instead of an ATA one? You can try one of the following solutions:

If you have an ATA drive the following command gives you it's current power mode: camcontrol epc $drive -c status -P. It should return Current power state: Standby_z(0x00) for a spun down drive.

If you are using SCSI drives you should be able to read the modepage 0x1a and have a look at the STANDBY line: camcontrol modepage $drive -m 0x1a. However, since I don't own any SCSI drives I can't confirm this. The pages and their content are described in /usr/share/misc/scsi_modes.

You can see the usage of the commands in the script starting at line 154: https://github.com/ngandrass/freenas-spindown-timer/blob/master/spindown_timer.sh#L154

Let me know if one of the solutions above work for you :)
Code:
root@freenas:~ # camcontrol epc da13 -c status -P
Current power state: PM0:Active or PM1:Idle(0xff)

root@freenas:~ # camcontrol epc da21 -c status -P
Current power state: Idle_a(0x81)

root@freenas:~ # camcontrol modepage da2 -m 0x1a
PM_BG_PRECEDENCE:  0
STANDBY_Y:  0
IDLE_C:  0
IDLE_B:  0
IDLE_A:  0
STANDBY_Z:  0
IDLE_A Condition Timer:  0
STANDBY_Z Condition Timer:  0
IDLE_B Condition Timer:  0
IDLE_C Condition Timer:  0
STANDBY_Y Condition Timer:  0
CCF Idle:  1
CCF Standby:  1
CCF Stopped:  2


So there are different idle modes? The SAS power mode is not very intuitive.
 

ngandrass

Explorer
Joined
Jun 26, 2019
Messages
68
So there are different idle modes? The SAS power mode is not very intuitive.

According to the specification (See: T13/BSR INCITS 529 Revision 14) Standby_y and Standby_z are both substates of the general PM2: STANDBY state a drive can enter. As the standard indicates the power consumption of a drive in Standby_z is less than or equal to the power consumption of the drive in Standby_y state (Standby_y power >= Standby_z power) (See 4.9.2 Power conditions). However, I think the implementation of both states is highly vendor dependent but if you are seeing your drive in either of the standby states you should be fine.

Anyway... None of your drives above seem to be spun down. Have you tried manually spinning down one of the drives an check if it is then reported as spun down correctly? Make sure to have no iops on the drive while performing the test.

To manually spin down an ATA drive issue camcontrol standby $drive or camcontrol stop $drive for a SCSI drive.
 

Ender117

Patron
Joined
Aug 20, 2018
Messages
219
According to the specification (See: T13/BSR INCITS 529 Revision 14) Standby_y and Standby_z are both substates of the general PM2: STANDBY state a drive can enter. As the standard indicates the power consumption of a drive in Standby_z is less than or equal to the power consumption of the drive in Standby_y state (Standby_y power >= Standby_z power) (See 4.9.2 Power conditions). However, I think the implementation of both states is highly vendor dependent but if you are seeing your drive in either of the standby states you should be fine.

Anyway... None of your drives above seem to be spun down. Have you tried manually spinning down one of the drives an check if it is then reported as spun down correctly? Make sure to have no iops on the drive while performing the test.

To manually spin down an ATA drive issue camcontrol standby $drive or camcontrol stop $drive for a SCSI drive.
Those drives are actually hot spares and I have set them to spin down after 60min, APM127, AAM medium, not sure why it kept spinning.

Anyway it appears I can manually spin down SATA drives but not SAS (it reports stopped successfully but power state did not change)
Code:
root@freenas:~ # camcontrol standby da21
root@freenas:~ # camcontrol epc da21 -c status -P
Current power state: Standby_z(0x00)
root@freenas:~ # camcontrol epc da16 -c status -P
Current power state: PM0:Active or PM1:Idle(0xff)
root@freenas:~ # camcontrol standby da16
root@freenas:~ # camcontrol epc da16 -c status -P
Current power state: Standby_z(0x00)
root@freenas:~ # camcontrol modepage da4 -m 0x1a
PM_BG_PRECEDENCE:  0
STANDBY_Y:  0
IDLE_C:  0
IDLE_B:  0
IDLE_A:  0
STANDBY_Z:  0
IDLE_A Condition Timer:  0
STANDBY_Z Condition Timer:  0
IDLE_B Condition Timer:  0
IDLE_C Condition Timer:  0
STANDBY_Y Condition Timer:  0
CCF Idle:  1
CCF Standby:  1
CCF Stopped:  2
root@freenas:~ # camcontrol stop da4
Unit stopped successfully
root@freenas:~ # camcontrol modepage da4 -m 0x1a
PM_BG_PRECEDENCE:  0
STANDBY_Y:  0
IDLE_C:  0
IDLE_B:  0
IDLE_A:  0
STANDBY_Z:  0
IDLE_A Condition Timer:  0
STANDBY_Z Condition Timer:  0
IDLE_B Condition Timer:  0
IDLE_C Condition Timer:  0
STANDBY_Y Condition Timer:  0
CCF Idle:  1
CCF Standby:  1
CCF Stopped:  2
root@freenas:~ #


About your script, I see it defaults to work on all drives with option to exclude certain drives. I would be nice if it can do the opposite: only monitor and work on specified drives, like the hot spares for example.
 

ngandrass

Explorer
Joined
Jun 26, 2019
Messages
68
Those drives are actually hot spares and I have set them to spin down after 60min, APM127, AAM medium, not sure why it kept spinning.

Anyway it appears I can manually spin down SATA drives but not SAS (it reports stopped successfully but power state did not change)


This is strange... Can you physically check if the drive is spun down after issuing the stop command? Maybe the power information of your specific drive is encoded in the CCF-values at the bottom of the modepage. However, I can't find any information on those values right now :(


About your script, I see it defaults to work on all drives with option to exclude certain drives. I would be nice if it can do the opposite: only monitor and work on specified drives, like the hot spares for example.


I introduced the manual mode (see release notification above). It disables automatic detection of drives and an explicit list of drives to monitor can be specified using the -i switch. Usage examples can be found in the advanced usage section of the README.
 

Ender117

Patron
Joined
Aug 20, 2018
Messages
219
This is strange... Can you physically check if the drive is spun down after issuing the stop command? Maybe the power information of your specific drive is encoded in the CCF-values at the bottom of the modepage. However, I can't find any information on those values right now :(





I introduced the manual mode (see release notification above). It disables automatic detection of drives and an explicit list of drives to monitor can be specified using the -i switch. Usage examples can be found in the advanced usage section of the README.
I don't think I can, the disks are in a DAS filled with 20 drives, noise from individual drives is inaudible.

Thanks for introducing manual mode, I will take a look.
 

MikeyG

Patron
Joined
Dec 8, 2017
Messages
442
@ngandrass Thanks so much for providing this resource!

I'm having a little bit of trouble understanding what the poll time means. Is that how often the script checks to see what the status of the drives are? How is it different than the timeout period? I also notice that my logs are filling up with messages "unfreezing devq for target" every hour when the default timeout is set to every hour. Could you explain more about what's happening and how the poll time and timeout functions interact?

*Edit - more specfics on my logs:

Code:
Oct 17 19:21:05 nas     (pass20:mpr0:0:20:0): INQUIRY. CDB: 12 01 00 00 ff 00 length 255 SMID 533 Aborting command 0xfffffe00015f2e30
Oct 17 19:21:05 nas mpr0: Sending reset from mprsas_send_abort for target ID 20
Oct 17 19:21:08 nas mpr0: Unfreezing devq for target ID 20
Oct 17 19:21:15 nas     (pass21:mpr0:0:21:0): INQUIRY. CDB: 12 01 00 00 ff 00 length 255 SMID 499 Aborting command 0xfffffe00015efd50
Oct 17 19:21:15 nas mpr0: Sending reset from mprsas_send_abort for target ID 21
Oct 17 19:21:17 nas mpr0: Unfreezing devq for target ID 21
Oct 17 19:21:24 nas     (pass22:mpr0:0:22:0): INQUIRY. CDB: 12 01 00 00 ff 00 length 255 SMID 443 Aborting command 0xfffffe00015eacd0
Oct 17 19:21:24 nas mpr0: Sending reset from mprsas_send_abort for target ID 22
Oct 17 19:21:27 nas mpr0: Unfreezing devq for target ID 22
Oct 17 19:21:33 nas     (pass23:mpr0:0:23:0): INQUIRY. CDB: 12 01 00 00 ff 00 length 255 SMID 435 Aborting command 0xfffffe00015ea150
Oct 17 19:21:33 nas mpr0: Sending reset from mprsas_send_abort for target ID 23
Oct 17 19:21:36 nas mpr0: Unfreezing devq for target ID 23
Oct 17 20:21:43 nas     (pass20:mpr0:0:20:0): INQUIRY. CDB: 12 01 00 00 ff 00 length 255 SMID 162 Aborting command 0xfffffe00015d18e0
Oct 17 20:21:43 nas mpr0: Sending reset from mprsas_send_abort for target ID 20
Oct 17 20:21:46 nas mpr0: Unfreezing devq for target ID 20
Oct 17 20:21:53 nas     (pass21:mpr0:0:21:0): INQUIRY. CDB: 12 01 00 00 ff 00 length 255 SMID 322 Aborting command 0xfffffe00015dfee0
Oct 17 20:21:53 nas mpr0: Sending reset from mprsas_send_abort for target ID 21
Oct 17 20:21:55 nas mpr0: Unfreezing devq for target ID 21
Oct 17 20:22:02 nas     (pass22:mpr0:0:22:0): INQUIRY. CDB: 12 01 00 00 ff 00 length 255 SMID 211 Aborting command 0xfffffe00015d5f50
Oct 17 20:22:02 nas mpr0: Sending reset from mprsas_send_abort for target ID 22
Oct 17 20:22:04 nas mpr0: Unfreezing devq for target ID 22
Oct 17 20:22:11 nas     (pass23:mpr0:0:23:0): INQUIRY. CDB: 12 01 00 00 ff 00 length 255 SMID 396 Aborting command 0xfffffe00015e6940
Oct 17 20:22:11 nas mpr0: Sending reset from mprsas_send_abort for target ID 23
Oct 17 20:22:14 nas mpr0: Unfreezing devq for target ID 23


Also getting:

Code:
(pass21:mpr0:0:21:0): INQUIRY. CDB: 12 01 00 00 ff 00
(pass21:mpr0:0:21:0): CAM status: SCSI Status Error
(pass21:mpr0:0:21:0): SCSI status: Check Condition
(pass21:mpr0:0:21:0): SCSI sense: NO SENSE asc:0,0 (No additional sense information)
camcontrol: The epc subcommand only works with ATA protocol devices
(pass22:mpr0:0:22:0): INQUIRY. CDB: 12 01 00 00 ff 00
(pass22:mpr0:0:22:0): CAM status: SCSI Status Error
(pass22:mpr0:0:22:0): SCSI status: Check Condition
(pass22:mpr0:0:22:0): SCSI sense: NO SENSE asc:0,0 (No additional sense information)
camcontrol: The epc subcommand only works with ATA protocol devices
(pass23:mpr0:0:23:0): INQUIRY. CDB: 12 01 00 00 ff 00
(pass23:mpr0:0:23:0): CAM status: SCSI Status Error
(pass23:mpr0:0:23:0): SCSI status: Check Condition
(pass23:mpr0:0:23:0): SCSI sense: NO SENSE asc:0,0 (No additional sense information)
camcontrol: The epc subcommand only works with ATA protocol devices


Thank you!!
 
Last edited:

ngandrass

Explorer
Joined
Jun 26, 2019
Messages
68
I'm having a little bit of trouble understanding what the poll time means. Is that how often the script checks to see what the status of the drives are? How is it different than the timeout period?

There are two timing parameters you are able to specify. The most important one is the timeout (-t TIMEOUT). This determines the number of seconds a drive has to experience no reads or writes before being considered as idle and spun down. Therefore when setting TIMEOUT to 3600 seconds, a drive that hasn't been used for at least one hour will be spun down.

Whether or not a drive experiences I/O is monitored using iostat. The poll time (-p POLL_TIME) determines how long a single iostat call waits before evaluating if there have been reads or writes during the last time period. This is necessary since the idle timer for each drive can only be updated in-between the iostat calls. Therefore the value of POLL_TIME simply specifies how long a single iostat call listens for I/O before the script updates the per disk idle counters. Generally a value between 60 and 600 seconds should be suitable for most applications :)

A quick note regarding the POLL_TIME value selection: The worst case time a drives idle status is detected is TIMEOUT + POLL_TIME seconds since we are only able to update the disk idle counters in-between the iostat calls.

I also notice that my logs are filling up with messages "unfreezing devq for target" every hour when the default timeout is set to every hour.
I'm not quite sure whats going on here. If I understand the situation correctly you are using SATA drives connected trough an SAS controller, correct? This somehow seems to result in the drives being detected as ATA drives but understanding only SCSI commands because of the SAS controller.

Can you provide the full output of all the following commands for at least one of your drives?

Code:
$ camcontrol identify $DRIVE
$ camcontrol epc $DRIVE -c status -P
$ camcontrol modepage $DRIVE -m 0x1a

P.S.: Unfortunately I only have access to my home FreeNAS machine which exclusively uses directly connected SATA drives. Therefore I can't test SCSI/SAS setups right now and possible fixes are based on the documentation I'm able to find online :(
 

MikeyG

Patron
Joined
Dec 8, 2017
Messages
442
Thank you for the explanation. Yes, SATA drives on a SAS controller (LSI 9305).

If I'm understanding "iostat call" correctly, you basically run iostat for the polling period (by default 600 seconds) and then seeing if drive activity happens during that 600 second period to determine if the drive is idle or not. As soon as that polling period ends, you get the results, update the idle status, spindown if needed, and then immediately start another polling period (or what I'm thinking of as sampling period).

I've included the following from running the camcontrol on one of the disks. Results are the same whether active or in standby:

Code:
root@nas:~ # camcontrol identify da16
pass20: <ST4000VN000-1H4168 SC44> ACS-2 ATA SATA 3.x device
pass20: 600.000MB/s transfers, Command Queueing Enabled

protocol              ATA/ATAPI-9 SATA 3.x
device model          ST4000VN000-1H4168
firmware revision     SC44
serial number         Z30142NR
WWN                   5000c5005d00087a
cylinders             16383
heads                 16
sectors/track         63
sector size           logical 512, physical 4096, offset 0
LBA supported         268435455 sectors
LBA48 supported       7814037168 sectors
PIO supported         PIO4
DMA supported         WDMA2 UDMA6
media RPM             5900
Zoned-Device Commands no

Feature                      Support  Enabled   Value           Vendor
read ahead                     yes      yes
write cache                    yes      yes
flush cache                    yes      yes
overlap                        no
Tagged Command Queuing (TCQ)   no       no
Native Command Queuing (NCQ)   yes              32 tags
NCQ Queue Management           no
NCQ Streaming                  no
Receive & Send FPDMA Queued    no
SMART                          yes      yes
microcode download             yes      yes
security                       yes      no
power management               yes      yes
advanced power management      yes      no      0/0x00
automatic acoustic management  no       no
media status notification      no       no
power-up in Standby            yes      no
write-read-verify              yes      no      0/0x0
unload                         no       no
general purpose logging        yes      yes
free-fall                      no       no
Data Set Management (DSM/TRIM) no
Host Protected Area (HPA)      yes      no      7814037168/7814037168
HPA - Security                 no
root@nas:~ # camcontrol epc da16 -c status -P
Current power state: PM0:Active or PM1:Idle(0xff)
root@nas:~ # camcontrol modepage da16 -m 0x1a
camcontrol: error sending mode sense command


The errors are faily easy to trigger by specifying a short timeout and short polling. As soon as the timeout counter runs down and it polls, i get the error:

Code:
root@nas:/mnt/Main/Files/Software/Scripts # ./spindown_timer.sh -m -t 60 -p 9 -v -i da16 -i da17 -i da18 -i da19
Monitoring drives with a timeout of 60 seconds: da16 da17 da18 da19
I/O check sample period: 9 sec
2019-10-18 15:25:44 Drive timeouts: [da16]=60 [da17]=60 [da18]=60 [da19]=60
2019-10-18 15:25:53 Drive timeouts: [da16]=51 [da17]=51 [da18]=51 [da19]=51
2019-10-18 15:26:02 Drive timeouts: [da16]=42 [da17]=42 [da18]=42 [da19]=42
2019-10-18 15:26:11 Drive timeouts: [da16]=33 [da17]=33 [da18]=33 [da19]=33
2019-10-18 15:26:20 Drive timeouts: [da16]=24 [da17]=24 [da18]=24 [da19]=24
2019-10-18 15:26:29 Drive timeouts: [da16]=15 [da17]=15 [da18]=15 [da19]=15
2019-10-18 15:26:38 Drive timeouts: [da16]=6 [da17]=6 [da18]=6 [da19]=6
(pass20:mpr0:0:20:0): INQUIRY. CDB: 12 01 00 00 ff 00
(pass20:mpr0:0:20:0): CAM status: SCSI Status Error
(pass20:mpr0:0:20:0): SCSI status: Check Condition
(pass20:mpr0:0:20:0): SCSI sense: NO SENSE asc:0,0 (No additional sense information)
camcontrol: The epc subcommand only works with ATA protocol devices
2019-10-18 15:26:57 Spun down idle drive: da16
2019-10-18 15:27:06 Spun down idle drive: da17
(pass22:mpr0:0:22:0): INQUIRY. CDB: 12 01 00 00 ff 00
(pass22:mpr0:0:22:0): CAM status: SCSI Status Error
(pass22:mpr0:0:22:0): SCSI status: Check Condition
(pass22:mpr0:0:22:0): SCSI sense: NO SENSE asc:0,0 (No additional sense information)
camcontrol: The epc subcommand only works with ATA protocol devices
2019-10-18 15:27:16 Spun down idle drive: da18


A longer timeout period with a short polling period does not seem to be a problem though. It appears that there is a conflict with timeout and polling:
Code:
root@nas:/mnt/Main/Files/Software/Scripts # ./spindown_timer.sh -m -t 3600 -p 9 -v -i da16 -i da17 -i da18 -i da19
Monitoring drives with a timeout of 3600 seconds: da16 da17 da18 da19
I/O check sample period: 9 sec
2019-10-18 15:28:50 Drive timeouts: [da16]=3600 [da17]=3600 [da18]=3600 [da19]=3600
2019-10-18 15:28:59 Drive timeouts: [da16]=3591 [da17]=3591 [da18]=3591 [da19]=3591
2019-10-18 15:29:08 Drive timeouts: [da16]=3582 [da17]=3582 [da18]=3582 [da19]=3582
2019-10-18 15:29:17 Drive timeouts: [da16]=3573 [da17]=3573 [da18]=3573 [da19]=3573
2019-10-18 15:29:26 Drive timeouts: [da16]=3564 [da17]=3564 [da18]=3564 [da19]=3564
2019-10-18 15:29:35 Drive timeouts: [da16]=3555 [da17]=3555 [da18]=3555 [da19]=3555
2019-10-18 15:29:44 Drive timeouts: [da16]=3546 [da17]=3546 [da18]=3546 [da19]=3546
2019-10-18 15:29:53 Drive timeouts: [da16]=3537 [da17]=3537 [da18]=3537 [da19]=3537
2019-10-18 15:30:02 Drive timeouts: [da16]=3528 [da17]=3528 [da18]=3528 [da19]=3528
2019-10-18 15:30:11 Drive timeouts: [da16]=3519 [da17]=3519 [da18]=3519 [da19]=3519
2019-10-18 15:30:20 Drive timeouts: [da16]=3510 [da17]=3510 [da18]=3510 [da19]=3510
2019-10-18 15:30:29 Drive timeouts: [da16]=3501 [da17]=3501 [da18]=3501 [da19]=3501
2019-10-18 15:30:38 Drive timeouts: [da16]=3492 [da17]=3492 [da18]=3492 [da19]=3492
2019-10-18 15:30:47 Drive timeouts: [da16]=3483 [da17]=3483 [da18]=3483 [da19]=3483
2019-10-18 15:30:56 Drive timeouts: [da16]=3474 [da17]=3474 [da18]=3474 [da19]=3474
2019-10-18 15:31:06 Drive timeouts: [da16]=3465 [da17]=3465 [da18]=3465 [da19]=3465
2019-10-18 15:31:15 Drive timeouts: [da16]=3456 [da17]=3456 [da18]=3456 [da19]=3456
2019-10-18 15:31:24 Drive timeouts: [da16]=3447 [da17]=3447 [da18]=3447 [da19]=3447
2019-10-18 15:31:33 Drive timeouts: [da16]=3438 [da17]=3438 [da18]=3438 [da19]=3438
2019-10-18 15:31:42 Drive timeouts: [da16]=3429 [da17]=3429 [da18]=3429 [da19]=3429
 
Last edited:
Top