Planning for Performance

[TravisT]

I've recently upgraded my FreeNAS box, but I've had some self-induced performance issues which I think have stemmed from how my vdevs were initially setup.

I currently have a 4-drive (4TB each) RAIDZ vdev that I use for typical file server duties, which includes electronic files, media (pictures, music, movies), and a software repository. I also have 2, 2-disk striped vdevs that are mirrored that I am currently using for block storage to my VMware ESXi box via iSCSI.

My iSCSI performance is good enough for what I use it for. Where I really notice a lack of performance is my other RAIDz vdev. I recently upgraded my FreeNAS box (see signature), so I now have plenty of room for expansion. I'm trying to understand how I can plan for the most performance, hopefully in a iterative process.

A lot of my performance issues are noticed with smaller files (pdfs, digital photos). I have expanded my vdevs a couple times through replacement of the individual drives with larger ones then allowing them to resilver. This was largely due to not having space in my case for more disks. Now I think my better option is to add like vdevs to my pools, but want to set myself up for success. Would love to hear suggestions, and I'll gladly fill in any blanks I may have overlooked.

 

[Heracles]

Hey Travis,

Whenever you talk about FreeNAS and performance, RAM is almost always part of the answer. More RAM will almost always mean more performance.

As for the drives, better to have all of them working together. A single pool of mirrors will probably help you with that.

Once done, you can always add more RAM and more mirrors...

Hope this gives you some ideas...

 

[TravisT]

I do know that ZFS is very RAM-hungry, but I assumed that 32GB of RAM would be enough in my current setup. Maybe that was a bad assumption. The good news is that have plenty of room for expansion.

While performance is an issue, I do care about the data, so I'm not sure a single pool of mirrors would be best in my case. Would you care to elaborate on that recommendation? To be clear, I'm only concerned about my NAS pool, not the one I'm currently using for iSCSI/VMs.

My initial thought was to increase my disk count and move from a RAIDZ1 to RAIDZ2, then in the future add another vdev with the same configuration as the first (doubling my usable size). I may be way off, and hope I can better understand your recommendation if applicable to the pool I'm focusing on.

 

[Heracles]

Hey Travis,

ZFS can do 3-way mirrors if you wish. 3 way mirrors will help you with read but not write. That way, no vDev will drop to a single drive in case of problem with any drive. Of course, you drop to 1/3 of your usable space. Still, that may be enough space for you... To keep cold spare is another way to protect yourself. But in all cases, never consider that a single FreeNAS is enough to guarantee your data will not be lost. No matter how many redundant drives you have, a single fire will destroy them all. If your data are important, you should plan for offline / offsite backups.

To transform a Raid-Z1 to a Raid-Z2 requires to destroy the pool and rebuild it from scratch. Also, if you load a first vDev and add a second one way later, ZFS will write almost exclusively to that second drive until both vDev end up with the same load. As a matter of performance, it will not be that good.

As for the RAM, 32G is much closer to the minimum. Even more when doing iSCSI. When you do iSCSI, ZFS does not know as well what is important and what is not. As such, it can not manage its caching as efficiently as it does with files. Caching being not as efficient, you need more to achieve the same result.

Hope this help you envisage the best solution...

 

[TravisT]

Based on my other thread that you responded to, I may be losing my iSCSI need in the not too distant future. That being said, I can throw some more RAM at this server to make it happy. Is there a way to determine if RAM is the issue versus something else?

I'm not very familiar with what you mean on the 3-way mirror, but I will research that more. I would, however, like to increase performance on writes. That is the first place I notice slow performance but I can't say for certain that I don't have issues elsewhere contributing to that. One of my biggest hits is noticed when copying RAW digital photos via Adobe Lightroom to my CIFS share on FreeNAS.

Understand that changing the size of the vdev will require destroying/rebuilding it, and I'm ok with that if it is a better solution in the long term. I originally started my ZFS quest on openSolaris (forget the distro now), and eventually made the switch over to FreeNAS. My pool configuration now probably closely resembles my original setup (with slightly larger drives now). I'm sure there's room for improvement, and I hope to come up with a plan for that.

Finally, I completely understand backups and the need for them, which I currently do (offsite). I know it has to be said, otherwise someone will get flamed for not saying it. Thanks for putting that to rest ;-)

Signing off for the night...

 

[kdragon75]

Falling asleep... You want more vdevs in your NAS pool. Performance scales with vdevs not drives. Think raid 50 (don't flame me bro)