IOPS?

[kspare]

We think we've got our storage servers tweaked pretty good right now.

Moving 8 vms, from another freenas box, both with 40gb adapters, i'm seeing write operations in zpool iostat sitting around 40k at a peak and averaging about 28k.

The server is running dual E5-2697 V2, 256gb ECC Ram, Mirrrored P3700 800gb, OP to 100GB formatted with 4k for Log, Single P3700 2TB OP to 500GB, formatted to 4K, Chelsios 40GB adapter, LSI 12GB SAS Adapter, Hitached 4TB SAS 3.5 Drives, latest freenas, over NFS.

Hyper Threading turned off

For tunables, I let it auto tune, and changed overhead to 16.

Seems to be working nicely, it runs about 54 terminal servers with around 450-500 users. no performance issues.

We seem to be constantly testing and looking for ways to squeeze more out.

I'm curious what others with larger operations are doing?

 

[kspare]

Some other tweaks.

Data Set record size is set to 64K
Mirrored vdevs.

I'm *not* running 9000mtu aka jumbo packets.
Atime off
lz4 compression

 

[kspare]

a snip of it rocking and rolling.

capacity operations bandwidth
pool alloc free read write read write
-------------------------------------- ----- ----- ----- ----- ----- -----
ESXi 5.03T 34.8T 108 36.0K 4.38M 1.60G
mirror 474G 3.16T 19 2.42K 1009K 60.8M
gptid/aa9f2e97-a96a-11e9-87cd-0cc47a12d23c - - 7 186 483K 54.5M
gptid/ab4606bf-a96a-11e9-87cd-0cc47a12d23c - - 11 175 526K 56.6M
mirror 469G 3.17T 7 2.13K 239K 56.7M
gptid/abf2e7da-a96a-11e9-87cd-0cc47a12d23c - - 3 158 116K 55.8M
gptid/aca23d25-a96a-11e9-87cd-0cc47a12d23c - - 3 185 124K 55.5M
mirror 469G 3.17T 6 2.35K 167K 60.1M
gptid/ad50b421-a96a-11e9-87cd-0cc47a12d23c - - 3 255 99.7K 56.1M
gptid/ae122ba4-a96a-11e9-87cd-0cc47a12d23c - - 2 178 67.8K 58.9M
mirror 469G 3.17T 12 2.23K 578K 55.1M
gptid/aea4ae42-a96a-11e9-87cd-0cc47a12d23c - - 9 194 447K 55.1M
gptid/af3d5468-a96a-11e9-87cd-0cc47a12d23c - - 2 277 132K 50.0M
mirror 474G 3.16T 6 1.87K 303K 56.9M
gptid/afe40e03-a96a-11e9-87cd-0cc47a12d23c - - 3 154 144K 54.8M
gptid/b08a1387-a96a-11e9-87cd-0cc47a12d23c - - 2 176 160K 53.6M
mirror 466G 3.17T 13 2.30K 602K 60.6M
gptid/b129c812-a96a-11e9-87cd-0cc47a12d23c - - 9 215 399K 53.5M
gptid/b1e253e1-a96a-11e9-87cd-0cc47a12d23c - - 3 169 203K 58.4M
mirror 466G 3.17T 6 2.06K 271K 60.7M
gptid/b29b6cb9-a96a-11e9-87cd-0cc47a12d23c - - 4 140 211K 58.0M
gptid/b369b444-a96a-11e9-87cd-0cc47a12d23c - - 1 152 59.8K 54.2M
mirror 467G 3.17T 9 2.42K 395K 63.6M
gptid/b406968f-a96a-11e9-87cd-0cc47a12d23c - - 5 163 259K 60.0M
gptid/b4a44c92-a96a-11e9-87cd-0cc47a12d23c - - 3 139 136K 55.9M
mirror 469G 3.17T 10 2.24K 526K 54.2M
gptid/b549e433-a96a-11e9-87cd-0cc47a12d23c - - 3 196 195K 52.7M
gptid/b5f16dcf-a96a-11e9-87cd-0cc47a12d23c - - 6 240 331K 53.2M
mirror 464G 3.17T 2 2.03K 75.8K 54.2M
gptid/b6b3d8df-a96a-11e9-87cd-0cc47a12d23c - - 0 246 0 51.4M
gptid/b773f402-a96a-11e9-87cd-0cc47a12d23c - - 2 193 75.8K 51.0M
mirror 468G 3.17T 8 2.19K 319K 52.4M
gptid/b838898d-a96a-11e9-87cd-0cc47a12d23c - - 6 309 235K 47.3M
gptid/b8f20d58-a96a-11e9-87cd-0cc47a12d23c - - 1 207 83.7K 50.2M
logs - - - - - -
mirror 2.74G 74.3G 0 11.8K 0 999M
gptid/99f44f6f-e3a9-11e9-9e2f-0cc47a12d23c - - 0 11.8K 0 999M
gptid/9a63170c-e3a9-11e9-9e2f-0cc47a12d23c - - 0 11.8K 0 999M
cache - - - - - -
gptid/517180ce-e85b-11e9-ba42-0cc47a12d23c 447G 18.2G 31 1.77K 1.27M 194M