Dear community, admins, zfs-heads and co-enthusiasts,
I am stuck with an unexpected (mal-)performance issue I haven't been able to solve for some time, and now I feel it's time to seek your help and ideas. So what happened so far?
After setting up a HP microserver to respectable read iSCSI performance (10Gbe, 500+MB/s seq. read performance within client-VM on VMware, 800+MB/s from a windows client) I now am baffled by a full-fledged server setup offering reads between 250 and 400 MB/s. The latter is rare though and prone to drop to 250MB/s without warning and staying there through reboots etc (I noticed this behaviour several times with different setups, further details below).
Small server:
HP microserver gen8
XEON E3-1265L V2 @ 2.5GHz
16GB RAM
Intel 10GbE pcie
RaidZ2 with 4 Intel SSD DC S3700 800GB
still running FreeNAS 9.2.1.9 (without experimental target or multithreaded mode)
Big server:
Supermicro Mainboard X10DRH-IT
2x Xeon E5-2673v3 3.5GHz
128MB RAM
using on-board 10Gbe ports
3x LSI SAS 9300-8i (Avago? 12Gb/s SAS just to make sure IOs won't be a bottleneck)
22x SSD Crucial M550
2x Samsung 845DC PRO for slogd and L2ARC (the latter in order to ease on r/w load)
running FreeNAS 9.3 (I fetched the most recent update today)
purpose: VM storage for operating system and database volumes, file services will reside on separate NAS
I tried several setups
- a 22-disk RaidZ3
- a 3x7 RaidZ3 (3x 7 disks)
- a 2x9 RaidZ3 (2x 9 disks: max 3 per controller)
- a 3x RaidZ2 (3x 6 disks: max 2 per controller)
- a 21-disk double-mirrored-setup (7 mirrors each with one disk per controller)
Originally I paid attention mostly to worst-case write performance (using sync=always) which didn't change as much as I had hoped, but at some point I noticed that reads hardly seem to differ at all, only ranging from 250 to 350 MB/s (and once 450).
Also, sometimes results (read and write) differed
a. after a reboot
b. after a complete destroy and re-create using EXACTLY the same setup/configuration
c. after a completely new installation of freenas using the same version
I was also very careful to choose exactly the disks I intended to use, spreading them out over the different controllers to increase (theoretical) resilience.
The local performance seems to be much higher so I suspect the problem is somewhere in between... iSCSI performance, network setup (I tuned some sysctls for 10Gbe with only marginal to moderate effect), the windows network card is identical and also tuned identically to my former tests with the microserver. I've been at this for weeks now, configuring, testing, pondering, changing, cursing, googleing (not necessarily in that order though). Ok, I guess should have come here sooner, but I wanted to get some solid base, some reliable numbers to build on - but after hundreds of tests the numbers still won't add up.
Performance testing was done with CrystalDiskMark (using between 1 and 4 GB of pseudo-random data) and AS SSD benchmark. (Btw: AS SSD seems to flush the cache after each write, CrystalDiskMark doesn't. I only noticed by coincidence because for testing I always configured the zvol for sync=always or sync=disabled respectively in order to get comparable results.)
Can anyone point me in a more promising direction where I should keep digging?
Thanks in advance!
Kjartan
PS: as an aside, I found that having all disks on one controller results in a somewhat better performance compared to spreading the disks out over several controllers. Still, I like the extra security spreading offers...
I am stuck with an unexpected (mal-)performance issue I haven't been able to solve for some time, and now I feel it's time to seek your help and ideas. So what happened so far?
After setting up a HP microserver to respectable read iSCSI performance (10Gbe, 500+MB/s seq. read performance within client-VM on VMware, 800+MB/s from a windows client) I now am baffled by a full-fledged server setup offering reads between 250 and 400 MB/s. The latter is rare though and prone to drop to 250MB/s without warning and staying there through reboots etc (I noticed this behaviour several times with different setups, further details below).
Small server:
HP microserver gen8
XEON E3-1265L V2 @ 2.5GHz
16GB RAM
Intel 10GbE pcie
RaidZ2 with 4 Intel SSD DC S3700 800GB
still running FreeNAS 9.2.1.9 (without experimental target or multithreaded mode)
Big server:
Supermicro Mainboard X10DRH-IT
2x Xeon E5-2673v3 3.5GHz
128MB RAM
using on-board 10Gbe ports
3x LSI SAS 9300-8i (Avago? 12Gb/s SAS just to make sure IOs won't be a bottleneck)
22x SSD Crucial M550
2x Samsung 845DC PRO for slogd and L2ARC (the latter in order to ease on r/w load)
running FreeNAS 9.3 (I fetched the most recent update today)
purpose: VM storage for operating system and database volumes, file services will reside on separate NAS
I tried several setups
- a 22-disk RaidZ3
- a 3x7 RaidZ3 (3x 7 disks)
- a 2x9 RaidZ3 (2x 9 disks: max 3 per controller)
- a 3x RaidZ2 (3x 6 disks: max 2 per controller)
- a 21-disk double-mirrored-setup (7 mirrors each with one disk per controller)
Originally I paid attention mostly to worst-case write performance (using sync=always) which didn't change as much as I had hoped, but at some point I noticed that reads hardly seem to differ at all, only ranging from 250 to 350 MB/s (and once 450).
Also, sometimes results (read and write) differed
a. after a reboot
b. after a complete destroy and re-create using EXACTLY the same setup/configuration
c. after a completely new installation of freenas using the same version
I was also very careful to choose exactly the disks I intended to use, spreading them out over the different controllers to increase (theoretical) resilience.
The local performance seems to be much higher so I suspect the problem is somewhere in between... iSCSI performance, network setup (I tuned some sysctls for 10Gbe with only marginal to moderate effect), the windows network card is identical and also tuned identically to my former tests with the microserver. I've been at this for weeks now, configuring, testing, pondering, changing, cursing, googleing (not necessarily in that order though). Ok, I guess should have come here sooner, but I wanted to get some solid base, some reliable numbers to build on - but after hundreds of tests the numbers still won't add up.
Performance testing was done with CrystalDiskMark (using between 1 and 4 GB of pseudo-random data) and AS SSD benchmark. (Btw: AS SSD seems to flush the cache after each write, CrystalDiskMark doesn't. I only noticed by coincidence because for testing I always configured the zvol for sync=always or sync=disabled respectively in order to get comparable results.)
Can anyone point me in a more promising direction where I should keep digging?
Thanks in advance!
Kjartan
PS: as an aside, I found that having all disks on one controller results in a somewhat better performance compared to spreading the disks out over several controllers. Still, I like the extra security spreading offers...
Last edited: