New to FreeNAS, looking for storage pool advice

[Andymiller6891]

Hi there.

I’m toying with the idea of trying FreeNAS on my new server, I have an ASUS dual Xeon V4 board with 64gb DDR4 and have ordered a Norco 4224 and have 16x 4tb Seagate SAS2 disks.

I’m looking for advice on the best way to setup the ZFS pool to maximise performance but with a good balance of size. I usually run a raid10 on an Adaptec raid card.

I plan on adding 2x 2.5inch 1tb SSDs I have available if they will offer any potential performance upgrade.

My aim is to get as close as I can to saturating a 10gb link that I have between my server and my clients PCs.

Any advice would be appreciated.

Thanks.

 

[danb35]

maximise performance

What kind of performance? Sequential reads? Random I/O? Something else?

I usually run a raid10

You can certainly do the same thing with ZFS, though it's generally referred to as striped mirrors. You'd obviously be sacrificing half your total capacity to redundancy, but it should be good for IOPS.

 

[Andymiller6891]

I’m used to loosing half capacity.

I’m mainly going to be using the box for plex media server, file sharing and a couple of windows VMs for domain services etc.

 

[PhiloEpisteme]

Hi @Andymiller6891, welcome to the forums! Could you give us a bit more info about your budget and use cases? It will help narrow down the build.

I’m toying with the idea of trying FreeNAS on my new server, I have an ASUS dual Xeon V4 board with 64gb DDR4 and have ordered a Norco 4224 and have 16x 4tb Seagate SAS2 disks.

That sounds like a great start for sure. Do you have the exact board, cpu, and memory models?

I’m looking for advice on the best way to setup the ZFS pool to maximise performance but with a good balance of size. I usually run a raid10 on an Adaptec raid card.

Great question. You may already know this, but I'll say it just in case. FreeNAS uses ZFS, which is a software RAID solution. It wants direct access to the disks to manage the pools. Thus, you don't need to spend the extra cash on a RAID card. In fact, it is generally advised that you NOT use any kind of hardware RAID with FreeNAS. Just use an HBA or on-board ports to give the file system direct access to the disks and you're all set.

The RAID10 analogue in FreeNAS is a pool composed of striped mirror vdevs. That is a bit of a mouthful, so I'll elaborate. ZFS exposes datasets for use in storing your data; a dataset is stored on a pool; a pool is made of vdevs; and devs are made of disks. A vdev can have many disks; a pool can have many vdevs; and a pool can have many datasets. If you single vdev in your pool goes down you lose the entire pool! Thus, it is extremely important that you have redundancy within your vdevs.

The redundancy options are mirrored vdevs, which use 2 or more drives as a mirror. These vdevs have the best IOPS but the highest cost per usable TB. RAIDZ1 built of n disks gives 1 drive worth of parity (though it is spread across all disks) thus giving you n-1 drives worth or storage. RAIDZ2 is like RAIDZ1 except it tolerates 2 drive failures and gives you n-2 drives worth or storage. RAIDZ3 extends this to 3 drive failure and n-3 storage.

It is common to use multiple vdevs to build a pool. In fact, adding vdevs to a pool is a great way to expand the size of your pool. If you've used RAID10 in the past you could stripe a bunch of mirror vdevs to get high IOPS and good redundancy.

Note Keep in mind that no level of redundancy is a replacement for a quality backup.

I plan on adding 2x 2.5inch 1tb SSDs I have available if they will offer any potential performance upgrade.

My aim is to get as close as I can to saturating a 10gb link that I have between my server and my clients PCs.

Those SSDs could be used as SLOG devices, but you'll want to make sure they have PLP if so. If you're using sync writes, they will offer a lot of performance improvements.

If you want to saturate that 10G connection you're going to want to take a look at exactly what your load is like. If you're using sync writes you're probably going to want to either get a good SLOG device or two. The striped mirrors are probably a good idea here.

Anyway, looking forward to hearing your thoughts. As I said, welcome to the forums. Hopefully you're able narrow down your build a bit here.

 

[Andymiller6891]

The mainboard is a Z10PE-D8 with 2x 2630 V3 cpus. 64gb DDR4 Registered ECC. Norco 4224 case, Corsair HX850 currently though may switch to a 1200, currently undecided on the HBA, have seen a few 12g SAS LSI ones.

 

[PhiloEpisteme]

The mainboard is a Z10PE-D8 with 2x 2630 V3 cpus. 64gb DDR4 Registered ECC. Norco 4224 case, Corsair HX850 currently though may switch to a 1200,

Thanks for the extra info :)

currently undecided on the HBA, have seen a few 12g SAS LSI ones.

This just came up in another thread. I bet you don't need SAS3; especially if you're only using 16 spinning disks. SAS2 will likely be more than enough. Or do you have a reason to think SAS2 isn't sufficient? The real pain is the need for 6 plugs to power that backplane. Check out my comments here for some advice in this regard.

 

[Andymiller6891]

I don’t really need sas3, sas2 will be fine for spindles as long as I can find a card to server all 6 8087 plugs for the backplanes.

I have been looking as the Adaptec 72405 but don’t know how the JBOD mode will play with FreeNAS. Would prefer a single card replacement as I play to utilise the other slots for my dual 10gb mellanox card and a quadro p2000 for plex.

 

[PhiloEpisteme]

10gb mellanox card

Do these have good support in FreeBSD and FreeNAS? You may want to search the forums for folks using it and if they had any issues.

I don’t really need sas3, sas2 will be fine for spindles as long as I can find a card to server all 6 8087 plugs for the backplanes.

Totally fair. Doesn't that board have a TON of PCIe slots? It looks like you can run 5x PCIe3.0x8 slots and 2x PCIe3.0x16? If so, you have plenty for what you need even if you go with two SAS2 HBAs and an expander. :)

I'm jealous of that board, I wish I had one lying around.

 

[Andymiller6891]

Yeah the board has 7 slots but I wanna try and minimise the amount populated.

I’m not sure if the mellanox is supported but had planned on changing it out for an Intel 520-t2 to use Ethernet rather than fiber.

 

[PhiloEpisteme]

I’m not sure if the mellanox is supported but had planned on changing it out for an Intel 520-t2 to use Ethernet rather than fiber.

Ah, that makes sense. Folks also have great luck with the Chelsio cards around here.

Yeah the board has 7 slots but I wanna try and minimize the amount populated.

Worried about temps? If so you could mount a fan right over those slots using one of those GPU fan brackets that mount on top of the expansion slot; I did that for an older build and had good luck keeping temps lower. You can absolutely go with the more expensive card so long as it is flashed to IT mode or isn't a RAID controller. I just mentioned the above as a way to save money without hurting performance in any way. But, as you rightly point out, there may be other considerations you're balancing. :)

 

[Andymiller6891]

so the best way to configure the storage would be to set up 8x 2hdd mirrored vdevs??

 

[PhiloEpisteme]

so the best way to configure the storage would be to set up 8x 2hdd mirrored vdevs??

I'll quote @danb35

What kind of performance? Sequential reads? Random I/O? Something else?

It really depends on what you want out of your pool. If you're looking for the setup which protects your data the best and you're willing to sacrifice speed you should go with RAIDZ3. If you're looking for IOPS and you have quality backups in place then yes, striped mirrored vdevs are a great way to go.

 

[Andymiller6891]

My main concern with this box is speed. I have a 2u supermicro box that will be used for a redundant backup and also have a backblaze b2 account that backups personal data.

 

[PhiloEpisteme]

My main concern with this box is speed. I have a 2u supermicro box that will be used for a redundant backup and also have a backblaze b2 account that backups personal data.

I'd say go with mirror vdevs then.

 

[Andymiller6891]

I’ve seen this page

https://calomel.org/zfs_raid_speed_capacity.html

Going by this a striped pair of raid z2’s is just as quick as 2disk mirrors. Is there any truth to this and how easy would it be to expand from 6disk raid z2’s to 7 or 8 disks?? I’m assuming I would need to add a 7th disk to both the striped z2’s?? Or is it more difficult?

 

[PhiloEpisteme]

Going by this a striped pair of raid z2’s is just as quick as 2disk mirrors. Is there any truth to this and how easy would it be to expand from 6disk raid z2’s to 7 or 8 disks?? I’m assuming I would need to add a 7th disk to both the striped z2’s?? Or is it more difficult?

Sorry, I was a bit unclear. The mirror setups can give you more IOPS with the same number of disks by giving you more vdevs in the pool.

 

[danb35]

how easy would it be to expand from 6disk raid z2’s to 7 or 8 disks?

Not easy. Destroy the pool and recreate it in the desired configuration. There's ongoing work that would allow vdev expansion, but I'm not aware of any ETA to see it in FreeNAS.