Iscsi Performance Freenas 9.3 ESXi6

[wreedps]

Here is what my disks are doing on a average Wednesday night

 

[jgreco]

Chill out :) I was interpreting "steady state" as being a streaming/bulk write, not random write. Obviously, this is great performance for random writes to a single drive!

Yes, but we were discussing iSCSI performance, where the only time you get anything like a bunch of contiguous writes on a long virgin region of disk blocks is when you're building your VM on a new pool. For everything else, you're already somewhere on the graph above.

 

[jgreco]

Here is what my disks are doing on a average Wednesday night

Ok, so, like, I'm reading a whole hell of a lot into the situation, but you look like you've got a pool that's fairly busy at a constant rate. If this is something like traffic to/from the Internet (BitTorrent maybe?) it could be that you're limited by the speed of the Internet connection, in which case the rest of what I'm about to say may not actually apply since you've got an artificial throttle elsewhere in the system.

Your pool is showing relatively constant read and write activity. On an iSCSI volume, the constant stream of write activity implies that you're very likely to be getting fragment-y write behaviours, either now, or in the future as fragmentation slowly increases. You're at ~20-25% pool utilization, which is great, and you may already have hit some sort of steady state if your normal usage patterns are like what you posted. If so, your pool will get slower if you fill it more.

The constant stream of reads suggests that there's a lot of stuff on your pool that isn't being held in ARC/L2ARC. You should take some time to consider what you're storing on the iSCSI. Is there an identifiable working set? That is, a set of blocks that are read more frequently than others? For a VM, for example, the working set would be the blocks that hold files required to boot/run the VM and any apps on it. A VM might be 20GB but only have 1-2GB of working set.

A FreeNAS system with too-little RAM has a difficult time identifying the working set when using a block storage protocol. It needs to be able to keep data around long enough that it can see that this block is accessed a second time whereas that other one hasn't been. You can check your ARC statistics to see how that's working out for you.

So, then, if you have a large enough ARC, you can add L2ARC to the system. The L2ARC will do very little for you if your ARC isn't big enough; ZFS will be evicting essentially random blocks to the L2ARC in a futile attempt to cache things. The ARC has to be big enough to identify useful things. But once it is, the ARC and L2ARC may be able to substantially reduce the amount of reads currently being sourced from your pool.

And the final point is that if you can follow along with what I've discussed here, then, yes, you can improve your performance, because your pool is currently spending a lot of time reading. Any of that reading which is offloaded onto ARC/L2ARC means that those IOPS are served from hella-fast non-HDD storage and therefore gains you more capacity on your HDD pool to be doing other things like writing.

 

[wreedps]

I believe most of the activity is coming from my 3MP 1080P IP Security Cameras. They are recording to a VM on the FreeNas. I use Video Insight.
I may consider moving them to local storage on one of the ESXi hosts or putting them on my Qnap.

Well I just looked at Veeam One and the VMs with the most datastore usage over the last hour are my Exchange 2016 servers. I have 3 currently on this FreeNas.
I do not want to move those and they are lightening fast on the FreeNas.

It has calmed down a little tonight.

 

[wreedps]

All 52 VMs are running on the FreeNas I am discussing now.

 

[wreedps]

Big Cluster at work...

 

[tvsjr]

So... you're running 52 VMs - including serious stuff like video surveillance and Exchange - to a 6-disk/3-vdev, 7.2k SATA array? And you're surprised that you're having performance challenges? IMO, it's a testament to FreeNAS and its underpinnings that it's working as well as it is. You simply need more spindles, or faster spindles (15K drives), or both.

 

[wreedps]

So... you're running 52 VMs - including serious stuff like video surveillance and Exchange - to a 6-disk/3-vdev, 7.2k SATA array? And you're surprised that you're having performance challenges? IMO, it's a testament to FreeNAS and its underpinnings that it's working as well as it is. You simply need more spindles, or faster spindles (15K drives), or both.

Where did I say I was having poor performance? Everything I have ever said about my FreeNas is positive and performance is great. I was merely asking questions about performance because I have seen peoples devices MB/sec reach in the 100s+ Mine tops out around 50-60MB/sec

 

[tvsjr]

Have you conducted your tests with all VMs shut down? I suspect you'd see similar numbers if the array wasn't constantly busy...

 

[wreedps]

No I cant shut them down at the moment and my hosts do not have any local storage right now.

I am in the process of building:
1x E5-1620 Supermicro X9SRL
10x 900GB HP 10k SAS in mirrors
1x 128GB X110 Sandisk L2ARC
128GB Memory DDR3 ECC
Chelsio 10gbe dual port

I am just waiting on a chassis and proc. Both are in the mail.
We can test with the new box before I put it live. I need to decomm the current Freenas as I need the proc and mobo to build my little cousin a gaming PC for xmas.

 

[wreedps]

The 2609 is a contemptible CPU (that means "it SUCKS!!"). 32GB is probably too small in terms of memory; block access protocols need LOTS of RAM to make good choices of what to evict to L2ARC.

You'd probably see some improvement with an E5-1620 and 64GB (or more) of RAM.

Well it turns out the 1600s CPUs are NOT compatible with all the Load Reduced DIMMs I have amassed. It only supports LDIMMs with 2600 series chips....

Sigh. I had the whole box together and it wont boot because the CPU is not compatible with the memory.