SOLVED Very slow write speeds during sas disk badblock testing

[zenon1823]

Hey all wondering if anyone has any thoughts on this. I'm continuing the shakedown on my new freenas hardware and am having an issue with write speeds to one particular model of SAS drive, and I'm not sure what to look into next. I initially had the DAS populated with 12 Hitachi SAS drives and all exhibited the same slow writes about 25MB/s and reads 150-190MB/s when running badblocks under FreeNAS. I previously ran badblock on all the same drives on the same hardware under Centos 7 and attained read & write speeds 150-190MB. I initially assumed the write speed issue was a FreeNAS OS issue, but I've since populated the DAS with the drives outlined below in the system specs and only the Hitachi drives exhibit the slow write speeds. Any thoughts?

System Details:
Dell R210 ii chassis with Xeon E3-1220V2 & 8GB ECC RAM
Dell H200E crossflashed to LSI 9200-8e - P20firmware
Dell MD1200 Chassis Populated with:
>> da0 - HUS156060VLS600 - Hitachi 600GB 15k SAS
>> da1 - HUS156060VLS600 - Hitachi 600GB 15k SAS
>> da2 - ST3600057SS - Seagate 600GB 15k SAS
>> da3 - ST3600057SS - Seagate 600GB 15k SAS
>> da4 - WD1601ABYS - WD 160GB SATA
>> da5 - WD1601ABYS - WD 160GB SATA

R/W Speeds running badblock under freenas:


Write speeds running badblock under centos 7 with the DAS fully populated with Hitachi HUS156060VLS600 drives:


EDIT: (later in the evening I had the great idea of seeing what kind of write speeds I get if I setup 3 mirror pools of each matching pair of drives)
The 25MB/s write speed doesn't seem to be an issue when writing a /dev/urandom file to the Hitachi mirror so its only badblocks... I'm stumped.

 

[HoneyBadger]

Since these are SAS drive and they aren't yet in a pool, I suspect it's that the drive write cache is set to disabled by default.

ZFS will try to enable the write cache when they are made members of a pool (and on subsequent imports) but I believe there's also a camcontrol command you can use to enable the cache.

Check the output of camcontrol identify daX -v and look for the line under Features for "write cache"

I believe the enable command on SAS drives is

Code:

echo "WCE: 1" | camcontrol mode /dev/daX -m 0x08 -e

But this might need to be added after every reboot.

 

[zenon1823]

Good Call @HoneyBadger! Those 2 disks did indeed have their write cache disabled.

Also as a side note - adding the drives to a pool did not enable the write cache, nor did subsequent imports.

A couple comments to help anyone who comes along this later. The camcontrol command provided by HoneyBadger works for retrieving write cache status on SATA drives but not for SAS. To check SAS use smartctl -x /dev/daX | grep Write (this also works for SATA)

Now onto the issue of enabling the writecache. I did some reading and found a few posts talking about using the command you mentioned, most posts did reference that it needs to be done after every reboot or by way of a post init script. However if you enable Writeback cache using smartctl then the setting is persistent over reboots. I did find this referenced in a couple posts but it seems largely not the more common solution despite it being a better solution(???) so hopefuly this helps someone else.
You can check the status with smartctl -g wcache /dev/sdX
You can enable it with smartctl -s wcache,on /dev/sdX
you can disable it with smartctl -s wcache,off /dev/sdX

I attached a capture of the performance now with writecache enabled on all drives. the first test (with both a write and read) is badblocks and the second (only write) is a dd test.

So my last question, if anyone has an idea, is to explain why (with writecache disabled) did badblocks experience such poor write performance but dd writing urandom to the disk was fine?

PS I also adjusted the subject of the post to better reflect what is discussed.

 

[HoneyBadger]

However if you enable Writeback cache using smartctl then the setting is persistent over reboots.

Interesting, I'll have to check this. Did you test it after a full power cycle yet? I suspect that unplugging and reconnecting a drive might reset the write cache but if the smartctl method generates persistent results that's fantastic.

Thanks for documenting the fix, and for updating the title for easy searching. I'm betting this will help future SAS users.

 

[zenon1823]

Interesting, I'll have to check this. Did you test it after a full power cycle yet? I suspect that unplugging and reconnecting a drive might reset the write cache but if the smartctl method generates persistent results that's fantastic.

You ask a great question and now I feel silly... Yes I had power cycled the server, but wasn't thinking about not having power cycled the disk shelf... And unfortunately enabling the writeback cache is not persistent across drive power cycles. Sorry for the false hope, more research required.

 

[HoneyBadger]

Don't feel bad. You've found a much simpler and easier to remember way to enable it via smartctl. Now we can combine that with the post-init scripts and have a very clear solution that doesn't seem like arcane voodoo involving SCSI mode page commands. :)

(And I'll go update my post-init scripts later today!)