Good day:
I've been building out my new FreeNAS box, and I've come across an interesting performance limitation. Here's the system config:
Chassis - Supermicro CSE-847E16-R1K28LPB (36-bay chassis with a 24-port SAS2 expander backplane up front and a 12-port SAS2 expander backplane in back)
Motherboard - Supermicro X9DRi-LN4F+
Processor - 2x Xeon E5-2670 (8C/16T, 2.6GHZ, each)
Memory - 128GB Samsung DDR3-1600 1.5V ECC Registered (16x8GB)
Boot Drives - 2x Intel 320 SSD, 40GB, mirrored (on rear backplane)
SLOG - 2x Intel S3700 SSD, 200GB, mirrored (on rear backplane - not yet configured)
Fast storage - 14x HGST HUS156045VLS600 450GB 15K SAS, 2-way mirrors (on front backplane)
Big storage - 6x HGST HUS724040ALS640 4TB 7.2K SAS, RAIDZ2 (on rear backplane)
HBA - LSI 9211-8i, flashed to IT mode, P20 firmware
OS - latest stable FreeNAS 9.3 build
Ignore the various RAID/ZFS configs, I haven't even reached that point yet.
While doing burn-in testing with the above drives, I observed some interesting performance behavior. The 15K SAS drives would cruise along about 100MB/sec read... but dropped down to ~24MB/sec write. The 7.2K SAS drives exhibited "normal" performance - with relatively equal read and write speeds, running between 120-200MB/sec.
The 15K drives spent some time in a NetApp, so they had the 520-byte sectors. I reformatted these to 512-byte, but suspected this could be an issue. So, I pulled all of the drives, leaving only the two boot drives installed on the rear backplane, and installed two drives I had around spare. These are brand-new, generic 512-byte sector, drives showing zero SMART issues and 2 accumulated start/stop cycles. One is a Seagate ST3300555SS (300GB 10K SAS), the other is a Fujitsu MAX3147RC (146GB 15K SAS).
Much to my surprise, I saw the same performance with these two new drives. Around 100MB/sec read, around 25MB/sec write.
I'm doing performance testing using simple DD commands:
dd if=/dev/zero of=/dev/daX bs=1M count=2000 (write)
dd if=/dev/daX of=/dev/null bs=1M count=2000 (read)
I've also done checks with iozone, and observed the performance charts in the FreeNAS GUI while running badblocks... all showed the same behavior and fairly similar performance numbers.
Doing reads from the boot devices, and reads/writes from the soon-to-be-SLOG devices, all SSDs, renders expected performance north of 200MB/sec. in both directions.
I'm kinda flummoxed at this point. I've moved the drives to different bays, no change. If it was a SAS vs. SATA thing, it would make more sense. But the 4TB SAS drives perform as expected.
Any thoughts would be greatly appreciated!
I've been building out my new FreeNAS box, and I've come across an interesting performance limitation. Here's the system config:
Chassis - Supermicro CSE-847E16-R1K28LPB (36-bay chassis with a 24-port SAS2 expander backplane up front and a 12-port SAS2 expander backplane in back)
Motherboard - Supermicro X9DRi-LN4F+
Processor - 2x Xeon E5-2670 (8C/16T, 2.6GHZ, each)
Memory - 128GB Samsung DDR3-1600 1.5V ECC Registered (16x8GB)
Boot Drives - 2x Intel 320 SSD, 40GB, mirrored (on rear backplane)
SLOG - 2x Intel S3700 SSD, 200GB, mirrored (on rear backplane - not yet configured)
Fast storage - 14x HGST HUS156045VLS600 450GB 15K SAS, 2-way mirrors (on front backplane)
Big storage - 6x HGST HUS724040ALS640 4TB 7.2K SAS, RAIDZ2 (on rear backplane)
HBA - LSI 9211-8i, flashed to IT mode, P20 firmware
OS - latest stable FreeNAS 9.3 build
Ignore the various RAID/ZFS configs, I haven't even reached that point yet.
While doing burn-in testing with the above drives, I observed some interesting performance behavior. The 15K SAS drives would cruise along about 100MB/sec read... but dropped down to ~24MB/sec write. The 7.2K SAS drives exhibited "normal" performance - with relatively equal read and write speeds, running between 120-200MB/sec.
The 15K drives spent some time in a NetApp, so they had the 520-byte sectors. I reformatted these to 512-byte, but suspected this could be an issue. So, I pulled all of the drives, leaving only the two boot drives installed on the rear backplane, and installed two drives I had around spare. These are brand-new, generic 512-byte sector, drives showing zero SMART issues and 2 accumulated start/stop cycles. One is a Seagate ST3300555SS (300GB 10K SAS), the other is a Fujitsu MAX3147RC (146GB 15K SAS).
Much to my surprise, I saw the same performance with these two new drives. Around 100MB/sec read, around 25MB/sec write.
I'm doing performance testing using simple DD commands:
dd if=/dev/zero of=/dev/daX bs=1M count=2000 (write)
dd if=/dev/daX of=/dev/null bs=1M count=2000 (read)
I've also done checks with iozone, and observed the performance charts in the FreeNAS GUI while running badblocks... all showed the same behavior and fairly similar performance numbers.
Doing reads from the boot devices, and reads/writes from the soon-to-be-SLOG devices, all SSDs, renders expected performance north of 200MB/sec. in both directions.
I'm kinda flummoxed at this point. I've moved the drives to different bays, no change. If it was a SAS vs. SATA thing, it would make more sense. But the 4TB SAS drives perform as expected.
Any thoughts would be greatly appreciated!