12 Disk SSD Mirror - Weird performance

[clipsco]

Hi everyone,

I recently put together a Supermicro FreeNAS server and the performance is not making much sense to me. Yes, I know having sync on will slow down writes, but the performance is all over the place. My issue is that a single 860 EVO gets the same results as my VM in this giant pool...And the 900p SLOG will do 550k, so I REALLY should be seeing better performance than a single disk. To confuse me even more, Im getting faster writes than reads on the VMs!!! How does that work in a pool of mirror vdevs?? I tried two other 10gb NICS with the same results. Does anyone have any idea as of whats going on or maybe the next step in troubleshooting? I dropped a lot of coin...

Local dd no compression disabled sync: 1.82GB/s write and 2.29G/s read
Local dd no compression sync always: 990MB/s write and 2.7GB/s read.
iSCSI VM (Crystaldiskmark) with compression and sync always sequential: 1200MB/s write and 600MB/s read...???
iSCSI VM (Crystaldiskmark) with compression and sync always 4k q8t8: 295MB/s write and 345MB/s read BOOOOOOOOO
I got similar results with Atto.

I have MPIO/port binding set up in ESXi with RR switching at 1 iop. I had a raidz2 array with better performa

specs as follows:
Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz
96GB DDR4 RAM
LSI SAS 9003-8i IT mode
Intel X520 10GB NIC
12x Samsung 860 EVO 2TB SSDs
Intel 900p SLOG
One storage vdev at the painfully recommended 50% pool size.
The main purpose is for a VMware iSCSI datastore.

Sync off
capacity operations bandwidth pool alloc free read write read write -------------------------------------- ----- ----- ----- ----- ----- ----- freenas-boot 790M 231G 0 0 0 0 da14p2 790M 231G 0 0 0 0 -------------------------------------- ----- ----- ----- ----- ----- ----- san2vol1 87.5G 12.6T 0 17.4K 0 1.82G mirror 12.5G 1.80T 0 2.53K 0 267M gptid/8be06408-2001-11e9-9812-0cc47ad29eba - - 0 527 0 267M gptid/8c394202-2001-11e9-9812-0cc47ad29eba - - 0 529 0 267M mirror 12.5G 1.80T 0 2.48K 0 266M gptid/8c85a54b-2001-11e9-9812-0cc47ad29eba - - 0 511 0 266M gptid/8ccc22e6-2001-11e9-9812-0cc47ad29eba - - 0 509 0 266M mirror 12.5G 1.80T 0 2.37K 0 266M gptid/8d1a76f0-2001-11e9-9812-0cc47ad29eba - - 0 460 0 266M gptid/8d6220cd-2001-11e9-9812-0cc47ad29eba - - 0 464 0 266M mirror 12.6G 1.80T 0 2.41K 0 267M gptid/8db20e7c-2001-11e9-9812-0cc47ad29eba - - 0 527 0 267M gptid/8dfb3c25-2001-11e9-9812-0cc47ad29eba - - 0 527 0 267M mirror 12.5G 1.80T 0 2.47K 0 266M gptid/8e4e60a8-2001-11e9-9812-0cc47ad29eba - - 0 491 0 266M gptid/8ea62f13-2001-11e9-9812-0cc47ad29eba - - 0 494 0 266M mirror 12.4G 1.80T 0 2.53K 0 266M gptid/8f02e24d-2001-11e9-9812-0cc47ad29eba - - 0 518 0 266M gptid/8f56dc51-2001-11e9-9812-0cc47ad29eba - - 0 519 0 266M mirror 12.5G 1.80T 0 2.56K 0 265M gptid/8fbb3155-2001-11e9-9812-0cc47ad29eba - - 0 530 0 266M gptid/9015ed13-2001-11e9-9812-0cc47ad29eba - - 0 535 0 265M logs - - - - - - gptid/907012f3-2001-11e9-9812-0cc47ad29eba 300K 260G 0 0 0 0
Sync off

Code:

                                           capacity     operations    bandwidth
pool                                    alloc   free   read  write   read  write
--------------------------------------  -----  -----  -----  -----  -----  -----
freenas-boot                             791M   231G      8      0  22.6K      0
  da14p2                                 791M   231G      8      0  22.6K      0
--------------------------------------  -----  -----  -----  -----  -----  -----
san2vol1                                 130G  12.6T  18.3K      0  2.29G      0
  mirror                                18.6G  1.79T  2.62K      0   335M      0
    gptid/8be06408-2001-11e9-9812-0cc47ad29eba      -      -  1.26K      0   171M      0
    gptid/8c394202-2001-11e9-9812-0cc47ad29eba      -      -  1.22K      0   164M      0
  mirror                                18.5G  1.79T  2.61K      0   334M      0
    gptid/8c85a54b-2001-11e9-9812-0cc47ad29eba      -      -  1.18K      0   160M      0
    gptid/8ccc22e6-2001-11e9-9812-0cc47ad29eba      -      -  1.28K      0   174M      0
  mirror                                18.6G  1.79T  2.61K      0   334M      0
    gptid/8d1a76f0-2001-11e9-9812-0cc47ad29eba      -      -  1.29K      0   175M      0
    gptid/8d6220cd-2001-11e9-9812-0cc47ad29eba      -      -  1.17K      0   160M      0
  mirror                                18.7G  1.79T  2.62K      0   335M      0
    gptid/8db20e7c-2001-11e9-9812-0cc47ad29eba      -      -  1.25K      0   171M      0
    gptid/8dfb3c25-2001-11e9-9812-0cc47ad29eba      -      -  1.20K      0   164M      0
  mirror                                18.6G  1.79T  2.62K      0   335M      0
    gptid/8e4e60a8-2001-11e9-9812-0cc47ad29eba      -      -  1.26K      0   166M      0
    gptid/8ea62f13-2001-11e9-9812-0cc47ad29eba      -      -  1.26K      0   169M      0
  mirror                                18.6G  1.79T  2.62K      0   335M      0
    gptid/8f02e24d-2001-11e9-9812-0cc47ad29eba      -      -  1.29K      0   175M      0
    gptid/8f56dc51-2001-11e9-9812-0cc47ad29eba      -      -  1.19K      0   160M      0
  mirror                                18.6G  1.79T  2.62K      0   335M      0
    gptid/8fbb3155-2001-11e9-9812-0cc47ad29eba      -      -  1.28K      0   173M      0
    gptid/9015ed13-2001-11e9-9812-0cc47ad29eba      -      -  1.19K      0   162M      0
logs                                        -      -      -      -      -      -
  gptid/907012f3-2001-11e9-9812-0cc47ad29eba   300K   260G      0      0      0      0
--------------------------------------  -----  -----  -----  -----  -----  -----

=

 

[Chris Moore]

You have something configured incorrectly. I can't tell what, from what you have posted.
Just to give you a point of comparison, this is the speed I get with a pool of OLD 500GB hard drives:

 

[Chris Moore]

Did you look at the settings that were suggested in these threads:

"Absolutely must virtualize FreeNAS!" ... a guide to not completely losing your data.
https://forums.freenas.org/index.ph...ide-to-not-completely-losing-your-data.12714/

Build Report: Node 304 + X10SDV-TLN4F [ESXi/FreeNAS AIO]
https://forums.freenas.org/index.ph...node-304-x10sdv-tln4f-esxi-freenas-aio.57116/

 

[clipsco]

FreeNAS is running on bare metal and the VM I'm using is on ESXi on two 10Gb links. One VLAN and portal per 10gb interface. The other freeNAS server currently has a few small z1 pools that are currently out performing the new server. I have reinstalled FreeNAS and created a new pool, zvol, and iSCSI target with the even lower results. I'm going to try a z1 pool Friday for diagnostic purposes.

I tried running Bonnie++ to get random iops numbers but it kept putting "+++++" in the output instead of the numbers for random seeks. I'm probably doing something wrong there.

I'm leaning towards a PCIe or driver issue. Maybe I'll delete this pool too and check performance of each drive to rule out a bottleneck. I think I recall disk info having a function that will do that.

Any place else you think I should look?

 

[Mlovelace]

The X520 is going to need some tuning to get those 10gbps speeds. Also, with an all SSD pool, depending on your IOPS requirements, you would most likely be fine setting up a raidz pool instead of mirrors. It would be worth testing as you're still configuring your system.
Those Samsung drives are rated for:

Code:

Random Read (4 KB, QD32)Up to 98,000 IOPS Random Read
Random Write (4KB, QD32)Up to 90,000 IOPS Random Write
Random Read (4 KB, QD1)Up to 10,000 IOPS Random Read
Random Write (4 KB, QD1)Up to 42,000 IOPS Random Write

 

[clipsco]

The X520 is going to need some tuning to get those 10gbps speeds. Also, with an all SSD pool, depending on your IOPS requirements, you would most likely be fine setting up a raidz pool instead of mirrors. It would be worth testing as you're still configuring your system.
Those Samsung drives are rated for:

Code:

Random Read (4 KB, QD32)Up to 98,000 IOPS Random Read
Random Write (4KB, QD32)Up to 90,000 IOPS Random Write
Random Read (4 KB, QD1)Up to 10,000 IOPS Random Read
Random Write (4 KB, QD1)Up to 42,000 IOPS Random Write

Great idea! I was excited to get to tuning the NIC and benchmarked it to have a baseline. Unfortunately I'm getting 9.32Gb/s per link. :(

Code:

[  3] local 10.31.70.2 port 19249 connected with 10.x port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  10.9 GBytes  9.32 Gbits/sec

 

[Elliot Dierksen]

Great idea! I was excited to get to tuning the NIC and benchmarked it to have a baseline. Unfortunately I'm getting 9.32Gb/s per link. :(

Not to be a killjoy, but what were you expecting to get? You are getting 93.2% of the maximum physical speed. There is protocol overhead at almost every layer, so I would say your results are pretty darn good.

 

[clipsco]

Not to be a killjoy, but what were you expecting to get? You are getting 93.2% of the maximum physical speed. There is protocol overhead at almost every layer, so I would say your results are pretty darn good.

Oh for the NIC? I was hoping the NIC was slow and could be tuned.

 

[HoneyBadger]

Can you show me the network config for your ESX/iSCSI setup? Often times I see port binding being used when it's not actually necessary, and I noticed that in your config.

Try creating the pool manually/advanced settings and use ashift=13 - I believe the Samsung TLC NAND uses an 8KB programming page.

Check into your BIOS for any "power saving" settings. When you get into the domain of all-flash, every microsecond of latency matters, and being in a lower-power "sleep" state can add to that.

Set the tunable vfs.zfs.metaslab.lba_weighting_enabled: 0 - since you don't have spinning drives and all LBAs on NAND are equally fast, you might as well disable this. You could also try the more extreme vfs.zfs.metaslab.fragmentation_factor_enabled: 0 which will tell ZFS to ignore metaslab fragmentation when allocating data - again, this tunable is for all-SSD pools only, and will be severely detrimental to the performance of HDDs.

edit: more suggestions and fixed formatting for the new forum code

 

[clipsco]

I think you're correct, port binding is not necessary here because the target ports are in different subnets. I don't see how that could negatively impact performance though.

I was thinking that random seek performance must heavily depend on latency, so I like your suggestions. I'll remove port binding on one host and make the suggested changes and report back.

 

[HoneyBadger]

I think you're correct, port binding is not necessary here because the target ports are in different subnets. I don't see how that could negatively impact performance though.

Unnecessary port binding usually just results in excessively long storage system rescans, but removing it would be good practice here.

I was thinking that random seek performance must heavily depend on latency, so I like your suggestions. I'll remove port binding on one host and make the suggested changes and report back.

Let me know how things go. Something like iozone may also give you some performance insights, although that's local to FreeNAS itself rather than over your network link.

 

[clipsco]

Try creating the pool manually/advanced settings and use ashift=13 - I believe the Samsung TLC NAND uses an 8KB programming page.

I have not gotten familiar with creating pools via the CLI and cant for the life of me find someone using a command that specifies the ashift. Could you or someone please give me a quick rundown?

Also I checked all the disks with diskinfo -wS and they're consistently 406-409MB/s on 8k.

Thanks

 

[Snow]

Could the SLOG be the cause some type of bottle neck? I just would think with the SSD mirrored and they all return to one SLOG if I read your setup right would that not slow it down? Is your SLOG the PCI Card type or the SATA Version?

 

[Scharbag]

You have something configured incorrectly. I can't tell what, from what you have posted.
Just to give you a point of comparison, this is the speed I get with a pool of OLD 500GB hard drives:

View attachment 28001

Could you give details as to what kind of pool this is?

Cheers,

 

[clipsco]

Could the SLOG be the cause some type of bottle neck? I just would think with the SSD mirrored and they all return to one SLOG if I read your setup right would that not slow it down? Is your SLOG the PCI Card type or the SATA Version?

The SLOG only comes into play during synced writes. My issue is that over iSCSI writes are FASTER than reads on a pool of mirrors. In mirrors, each write has to occur twice, while reads should be accelerated. Locally the reads are faster like theyre supposed to be (and honestly slower than I expected), but over iSCSI the writes are FASTER than locally and read speed is 80% slower.

 

[Snow]

Have you tried any real world testing to confirm that Crystaldiskmark is not being weird? what motherboard are you using ?

 

[HoneyBadger]

I have not gotten familiar with creating pools via the CLI and can't for the life of me find someone using a command that specifies the ashift. Could you or someone please give me a quick rundown?

There's a very old thread from @jpaetzel here about how to use gpart from the command line to set up a pool from the CLI in the same manner (a little swap on each disk, etc) here: https://forums.freenas.org/index.php?threads/building-pools-from-the-cli.17540/ - however, the thread is old, and creating pools from the CLI generally isn't recommended as it's easy to mistype the options or miss them entirely (Trying to remember what the defaults used by FreeNAS are now.)

When you're creating the partitions with gpart you'll probably want to specify 8K alignment using the -a flag for both partitions:

Code:

gpart add -a 8192 -i 1 -s 2g -t freebsd-swap /dev/da0
gpart add -a 8192 -i 2 -t freebsd-zfs /dev/da0

Then when creating the pool, you'll use -o ashift=13 in the zpool create line.

Also I checked all the disks with diskinfo -wS and they're consistently 406-409MB/s on 8k.

Thanks

Can you paste the diskinfo output in code tags? I'd be interested to see how it breaks down for write latency.

 

[Mlovelace]

Trying to remember what the defaults used by FreeNAS are now

Do not include the <> symbols in the command they are to delineate where the user is to add their own information.

Code:

zpool create -o cachefile=/data/zfs/zpool.cache -o failmode=continue -o autoexpand=on -O compression=lz4 -O aclmode=passthrough -O aclinherit=passthrough -f -m /<pool name> -o altroot=/mnt <pool name> mirror /dev/gptid/<partition 2 gptid> /dev/gptid/<partition 2 gptid>

Edit: I for got you'll want to change the ashift to 13 before making the ssd pool.

Code:

sysctl -w vfs.zfs.min_auto_ashift=13

 

[HoneyBadger]

You should be able to just add -o ashift=13 in the "zpool create" line, I believe:

Code:

zpool create -o cachefile=/data/zfs/zpool.cache -o failmode=continue -o autoexpand=on -o ashift=13 -O compression=lz4 -O aclmode=passthrough -O aclinherit=passthrough -f -m /<pool name> -o altroot=/mnt <pool name> mirror /dev/gptid/<partition 2 gptid> /dev/gptid/<partition 2 gptid>

(Or am I getting my OpenZFS implementations mixed up here, and that's how it's done in Linux?)