What type of NVMe performs best in FreeNAS

[mouseskowitz]

I'm thinking of adding a NVMe drive to my FreeNAS. Do I need it, no, but it would be fun to play with. Any why, I'm trying to figure out what what specs on a drive are best suited to the way ZFS reads and writes.

My understanding of the way ZFS works is that it is usually writing data sequentially and will be reading data randomly. So this would mean that in the write category higher sequential write performance is more important than higher write IOPS. Then in the read category, IOPS are the key. More specifically, I'm comparing the performance of the 2TB versions of the Intel DC P4510 and DC P4600. If my reasoning is correct, the DC P4510 would be a better fit because it has higher sequential write, and random IOPS.

Am I on the right track or am I missing something in my line of thought?

 

[StoreMore]

What is the layout of your existing system (type and # of drives)? How much ram? What kind of network connection? Connection speed of clients? How many users?

What do you use the NAS for (file server, vms, video editing, etc)?

The reason I ask all of this above is that you might not notice much of a difference with or without SSDs unless you have specific use cases. Just for some info, I have an 8 drive nas in z2, along with 2 905P optane drives. However, even without the optane drives in use I can get file transfers of 600MBs per second or more. If I was doing VMs and sync transactions regular HDs would be slowwww.

 

[mouseskowitz]

Right now I have 6 3TB HDDs in raidz2 for bulk storage and 2 1.9TB SSDs (Intel DC S4600) in a mirror for OS drives. The system has 32GB RAM and dual 10G fiber and a Xeon 2609v4. It's backing about 20 ESXi VMs over ISCSi.

I realize that in daily operation there really won't be much difference between the SSDs and the NVMe. The main advantage would come when I'm doing maintenance to some other operations that hits the disks hard. Right now I usually get 3-4Gbps since it's a read/write mix off the SSDs. If it's pure read or write I will occasionally see as high as 5 or 6Gbps.

This is largely a learning exercise of the best way to configure multiple tiers of storage and what will give the best performance potential for the use case.

 

[StoreMore]

The iSCSI interface for VMs is pretty efficient compared to the VMs of NFS when it comes to sync writes.

I am guessing your are running the VMs off the S4600s? If this is the case I am not sure you'd feel a huge difference with just NVME. So, if you really do want to upgrade I'd go straight to optane for the better durability and faster sustained speeds/lower QD speeds.

 

[mouseskowitz]

Optane would be nice, but I don't have a couple grand for a 2TB drive. My budget is about $100-120/TB which is doable for NVMe. I'm not sure that I'd even see much difference in the day to day with Optane, my latency for the S4600 pool is under 1ms 99% of the time already. It's the times when the bottleneck is the sata bandwidth that I'm trying to remedy.