Slow writing but fast reading (ZFS RAID-Z1)

[super-hornet]

Hi all
I'm using FreeNAS 8.0.1 Beta 3 and now Beta 4.
I have 3pieces of 2TB SATA III HDD (but connect to SATA II port due to FreeNAS 8.0.1 do not support the SATA III my motherboard chipset).

The HDD is configured as ZFS RAID-Z1

When I start up with those 3 HDD empty, the read and write speed over the 1GBps network is like 80MBytes/second.

Now, that those RAID-Z1 is filled up to like 2TB++, the performance in writing now like 11MBytes/seconds. But Reading it can achieved 80MB/s.

Anyone experiencing the similar symptom? Anything that I missed out?

Oh.. the system is dual core, Athlon Phenom II with 8GB of RAM.

SH

 

[matthewowen01]

how are you testing?

 

[joeschmuck]

I suspect you are running the 64 bit flavor as well. Also could you be a bit more specific about drive capacity with relation to the volume you created? I would like to know the volume size and how much free space is available for that volume. As Matt said, how are you testing the throughput?

As for SATA II v.s. SATA III for this type of project, it will not make a difference. Could you also post your MB info. It would be helpful for when someone asks about what to buy. If your MB isn't currently fully supported, that would be nice info to have for the folks here who give advice.

 

[super-hornet]

The HDD I'm using is Western Digital 2TB Green, SATA III.

Motherboard I'm using is:
Gigabyte GA-770TA-UD3. The onboard SATA III controller chipset is from Marvel 9128. If I connect SATA III hard disk to the sata3 controller, the bios can detect but FreeNAS cannot recognize it. So I have no choice but to use the onboard SATA 2 instead.

I configured 3 HDD in ZFS RAID-Z1 with one dataset. When I copy file to and from Windows 7, (CIFS), the read and write speed I can achieve is around average 80MBps...during when the dataset/RAID-Z1 was empty. But after around 2TB used, the write speed dropped but reading/copying files from FreeNAS to Windows can still achieve 80MBps.

Now, come to the strange thing. I tried reboot a few times and still the same. Then I change the CIFS settings (FreeNAS) to enable "Enable AIO". Do a reboot and the write speed now shoot up 50 to 60MBps.

Then I tried to simulate again by disabling the "Enable AIO" to test whether is that the one that cause it, and do a reboot.

Now, even when I turn the "Enable AIO" to off (revert back to default), the read and write speed is back to normal of 80MBps. STRANGE!


EDIT: Yup! I'm using 64bit version.
FreeNAS-8.0.1-BETA4-amd64
Since the motherboard only have 2 ports of SATA3, I add in a SATA3+USB3 Controller from ASUS. It is "U3S6". This controller's SATA3 chipset is from Marvel as well but the chipset i think is "88SE9123". If I connect my SATA3 HDD to that controller (ASUS), FreeNAS can recognize the SATA3 hard disk BUT too bad this controller only have 2 ports.

SH

 

[joeschmuck]

Enable AIO is now set to one by default as of Beta-3. Your configuration must be older than that or you turned off AIO.

As for the strange reaction, if it comes back please post again in this thread. If you can think of anything you might have tweaked or done that might have caused the slow transfer that would be good to know. Maybe a ticket will need to be opened if it's repeatable.

-Mark

 

[jenksdrummer]

If you have a WD EARS drive (common), they are advanced format, but report to BIOS/OS that they have 512b sectors, instead of 4k sectors. The OS (ie, Microsoft Vista-level and up) are supposed to "know" this and format them correctly. Check to make sure you forced 4K when you formatted the volume - this is an option in 8.0.1-Beta4, but I think it's existed since 8.0.1-Beta1, and existed in the FreeNAS 7.x releases I've used.

Either that, or use the jumper, as it'll kick all the writes over by 1 sector, causing the alignment to be corrected. 63 sectors/512b is what is expected by systems...4096b isn't divisible by 63...but is by 64. The jumper is used on pins 7 and 8 (reference drive label for which pins are which) - this is how WD gets around the problem.

It would have been so much easier if they didn't do this 512b-e mode...either report 512b and mean it, or 4096b and mean it!

 

[joeschmuck]

I think the jumper is a great idea and wish more manufacturers used it (Samsung, hint hint). It allows the consumer the flexibility of using it in a newer or older OS. Eventually there will be 4K only drives but they might all be SSD. I don't think the drives are what slowed down the throughput.

 

[super-hornet]

Thanks for the CIFS setting. I will keep on monitoring it.

That HDD is 4k format. When I create the RAID, I did set the "Force 4096 sector".
I did post it on other thread that, for this hard disk, in openSUSE 11.4, it shows as 4k sector. But in FreeNAS, it shows 512.

The motherboard SATA2 and SATA3 port can recognize the hard disk without having to enable the jumper. Same goes for openSUSE. But when using FreeNAS, for that particular SATA3 marvell chipset, FreeNAS do not recognize it.

I also think it is not because of 4k sector relation that it is slow down since the reading do not affected. If it is 4k related problem, then the read should be affected as well.

SH

 

[joeschmuck]

Good thing your problem is currently fixed. Should it occur again please post and maybe you will figure out what is causing it. Hopefully it never returns.

 

[Daisuke]

If you have a WD EARS drive (common), they are advanced format, but report to BIOS/OS that they have 512b sectors, instead of 4k sectors. The OS (ie, Microsoft Vista-level and up) are supposed to "know" this and format them correctly. Check to make sure you forced 4K when you formatted the volume - this is an option in 8.0.1-Beta4, but I think it's existed since 8.0.1-Beta1, and existed in the FreeNAS 7.x releases I've used.

Either that, or use the jumper, as it'll kick all the writes over by 1 sector, causing the alignment to be corrected. 63 sectors/512b is what is expected by systems...4096b isn't divisible by 63...but is by 64. The jumper is used on pins 7 and 8 (reference drive label for which pins are which) - this is how WD gets around the problem.

It would have been so much easier if they didn't do this 512b-e mode...either report 512b and mean it, or 4096b and mean it!

Just to complete your post: I did a lot of research on this, as I run 6 WD20EARS disks on my ZFS RaidZ2 array. The purpose of the jumper is to allow Windows XP to perform normally with a start sector of 63. Forcing the 4K sectors in FreeNAS will start the partition at sector 2048 (like Vista and Windows 7), in which case you do not want the jumper. The fact that FreeNAS uses 4K sectors internally (and still claims is 512 bytes) has no bearing on anything other than the optimal alignment.

Personally, I don't use the jumper method (WD explanation).
That does not changes the fact that is WD's fault, the drive presents itself to the OS as having 512B sectors. WD fixed the issue into WD20EARX.