New Build - Looking for the 'best' parts.

[belcboo]

Hello, this will be my first "big" FreeNAS build and I'm looking for advice just to make sure that the pieces that I have chosen are the best for my needs. This server will be used to back up all the information that we have sparced in all the building, as well as storage of ESXi using a NFS share.

SERVER:
I'm planning to buy a refurbished server including the following:

Chassis: CSE-847E16-R1K28LPB
Backplanes: BPN-SAS2-826EL1 and BPN-SAS2-846EL1
Motherboard: X9DRi-LN4F+
CPU: 2 x E5-2650 [8C@2.00Ghz]
RAM: 192GB [24x8GB]
NIC: 4 x Intel i350 and planning to add 2 x Chelsio T420-BT.
RAID Card: LSI 9211-8I
PSU: Dual 1400w Gold.

*I'll probably add an SSD as a boot drive, I don't know if that one needs to be in mirror mode, or if you recommend that only for USB thumbs. [I also learned that it is very important to do periodical backups of the config file so you can restore your setup in case of disaster]

**It also include the Rails.

DISKS:
I was planning to buy 20 WD Red 8TB drives (I like more the HGST drives, but there's a huge gap in the price), I was planning to use mirror setup, but after reading the "FreeNAS noobs guide" I think that RAIDZ3 offers me more fail drive tolerance, I'm trying to decide if I should go with 21 or 22 (I'm still in doubts about 3 vDevs of 7 drives or 2 vDevs or 11 drives, both in RaidZ3). I plan to buy a 36 bay super micro server, so if we need more space I can add another 11 or 14 drives to create another vDev and add it to the Zpool (again, 1 vDev of 11 to have a total of 33 drives, having 3 bays without use, or 2 vdev of 7 drives to have a total of 35 drives, all will depend on the initial vdev config that I choose). Personally, I think that the 7 drives per vDev is the best choice in my case, but I don't know if I'm missing something.


CONCERNS:

- I haven't seen many posts about the T420-BT, only other versions of this card, I'm not sure if its will work.
- Yesterday I had a lot of concerns, but after reading the "FreeNAS for noobs" guide most of them were solved lol.

So basically that's all, any comment or advice will be helpful.

Thank you!

 

[snaptec]

What will you serve?
Vm space?
Users?
What Kind of performance do you expect?
Iscsi? Smb? Nfs?


Gesendet von iPhone mit Tapatalk

 

[belcboo]

What will you serve?
Vm space?
Users?
What Kind of performance do you expect?
Iscsi? Smb? Nfs?


Gesendet von iPhone mit Tapatalk

- Storage for 40 to 50 TB of film production. Using SMB and AFP.
- 2 ESXi Servers with 10VM (Most of them are ubuntus as WebServers, and two windows). Both will use NFS
- 3 WorkStations to Edit video (In case the speed is enough). Using SMB and AFP.

*We're also checking the option of include a pool of SSD for the editors workstations.

And basically that's it. We also have a tape backup system, I'm doing the research to have both things working so we can have our information "safer".

 

[Spearfoot]

- Storage for 40 to 50 TB of film production. Using SMB and AFP.
- 2 ESXi Servers with 10VM (Most of them are ubuntus as WebServers, and two windows). Both will use NFS
- 3 WorkStations to Edit video (In case the speed is enough). Using SMB and AFP.

*We're also checking the option of include a pool of SSD for the editors workstations.

And basically that's it. We also have a tape backup system, I'm doing the research to have both things working so we can have our information "safer".

Mirrors are recommended for ESXi datastores because IOPS scale with vdevs: 1 or 2 very wide RAIDZn vdevs won't perform very well compared to 6 or 8 mirrors. You will also need a SLOG device or your VM performance will be terrible (unless you disable synchronous writes on the VM dataset, which is not recommended). See "Some insights into SLOG ZIL with ZFS on FreeNAS" for details. But bear in mind that, if your VMs aren't working very hard, i.e., if performance isn't a big consideration, you might get by with storing them on the single wide RAIDZn-based pool you've described.

If your VM space requirements are low, you might want to consider setting up a SSD-based pool for the ESXi datastore and then configure a separate HDD-based pool for your other storage needs.

There's always more than one way to skin the cat...

 

[snaptec]

You already know that smb is a per core service?
Fewer faster cores are better than many slow cores.
That may be a problem if you want to max out 10g.
I will answer tomorrow again and give some other hints. As spearfoot already pointed out, 2 vdevs in raidz wont give you many iops


Gesendet von iPhone mit Tapatalk

 

[belcboo]

Mirrors are recommended for ESXi datastores because IOPS scale with vdevs: 1 or 2 very wide RAIDZn vdevs won't perform very well compared to 6 or 8 mirrors. You will also need a SLOG device or your VM performance will be terrible (unless you disable synchronous writes on the VM dataset, which is not recommended). See "Some insights into SLOG ZIL with ZFS on FreeNAS" for details. But bear in mind that, if your VMs aren't working very hard, i.e., if performance isn't a big consideration, you might get by with storing them on the single wide RAIDZn-based pool you've described.

If your VM space requirements are low, you might want to consider setting up a SSD-based pool for the ESXi datastore and then configure a separate HDD-based pool for your other storage needs.

There's always more than one way to skin the cat... :)

That was at first the idea, use mirror, but then I read that I could have better protection using RAIDZ3. Now after your comment I started to think the following scenario: If I acquire 31 drives could I use the following configuration?

- Pool 1: 3 vDev with 7 drives each in RAIDZ3 for storage. [Optional: 2 vDev with 11 drives each always in RAIDZ3]
- Pool 2: 5 vDev with 2 drives each in Mirror.

(The idea of having the whole pool in mirror with 3 drives its tempting me, but that will result really expensive to achieve the space I need.)

*Additional I will have from 3 to 4 free bays, I could add some SSD but that will depends in how much I can pull the budget.

You already know that smb is a per core service?
Fewer faster cores are better than many slow cores.
That may be a problem if you want to max out 10g.
I will answer tomorrow again and give some other hints. As spearfoot already pointed out, 2 vdevs in raidz wont give you many iops

Respecting to this, SMB won't be the main use of the Storage, I mean, we have a couple of clients who will use it, but that will be rarely, the main use will be from the ESXi servers (where most of the servers are internal web servers) and from the editors who have some Mac Pro's.

But I have started to check for CPU upgrades and I can have a E5-2670@2.6Ghz, or even get a couple of E5-2690@2.9Ghz in the future.

Thanks both of you for your help, I'll be waiting to check your answer.

 

[Spearfoot]

That was at first the idea, use mirror, but then I read that I could have better protection using RAIDZ3. Now after your comment I started to think the following scenario: If I acquire 31 drives could I use the following configuration?

- 3 vDev with 7 drives each in RAIDZ3 for storage. [Optional: 2 vDev with 11 drives each always in RAIDZ3]
- 5 vDev with 2 drives each in Mirror.

(The idea of having the whole pool in mirror with 3 drives its tempting me, but that will result really expensive to achieve the space I need.)

Yes, you can use 3-way mirrors but, as you pointed out, that would be horribly expensive and inefficient, too.

I believe RAID-Z3 is overkill for 7-disk vdevs; you'd be safe running these as RAID-Z2 arrays. With 3 vdevs, using RAID-Z2 instead of RAID-Z3 will increase the capacity of your 21-disk storage pool by the size of the 3 disks you wouldn't be using for parity. However, I would definitely use RAID-Z3 for 11-drive vdevs, though. Your call, either way!

I really do like the idea of a separate mirrored pool for VM storage. Again, you will need a SLOG device for this pool. You're building a pretty high-end system, so you might consider an NVMe device like the Intel P3700.

 

[tvsjr]

I have a nearly identical box serving as my FreeNAS. A few thoughts:
You can add two internal drive brackets, which will each hold 2 2.5" drives. This is a great option for your boot and SLOG devices, leaving your external drive bays free.

I'd suggest two pools, as I have. One pool of mirrors for your VM store, another pool for your video storage. I'd consider a 6-drive array (3x2 drive vdevs) for your VMs, assuming they aren't demanding (that's only ~225 IOPS with 7200RPM drives). For your media storage, 6-drive vdevs in RAIDZ2. Since you're looking at a 36-drive chassis, staying in multiples of 6 makes sense in my mind.

 

[belcboo]

I have a nearly identical box serving as my FreeNAS. A few thoughts:
You can add two internal drive brackets, which will each hold 2 2.5" drives. This is a great option for your boot and SLOG devices, leaving your external drive bays free.

I'd suggest two pools, as I have. One pool of mirrors for your VM store, another pool for your video storage. I'd consider a 6-drive array (3x2 drive vdevs) for your VMs, assuming they aren't demanding (that's only ~225 IOPS with 7200RPM drives). For your media storage, 6-drive vdevs in RAIDZ2. Since you're looking at a 36-drive chassis, staying in multiples of 6 makes sense in my mind.

I was thinking that, a pool of 6x4TB HGST@7200 in mirror for VM's, that will give me about ~12TB [Right now I have only 4TB for my VM's, but I have provisioned more than that], and with that change save a little of money. And the 6 drives in RAIDZ2 make sense too (Although is a little scary for me LOL).

Now I only have to read more about the SLOG. In your signature I see that you have Intel S3700, drives for SLOG and L2ARC, that drive is SATA II, does it affects the performance?

Thank you so much,

 

[tvsjr]

Do a little searching around the forums... using FreeNAS as a VM store has been discussed ad nauseum. There are points that you are missing... such as never exceeding 50% utilization on your VM pool. So, your 12TB is actually only 6TB usable.

The S3700s are well-respected drives and won't pose a performance bottleneck unless you start doing some very exotic stuff (10 or 40gig-e, very large pools, etc.) You do want a drive that is rated for many writes and that has power loss protection. This typically means the Intel data-center grade drives. If you have the coin, going to NVMe drives is nice.

You'll also want to ensure you have a properly-sized UPS that is configured to shut the system down gracefully should an extended power event occur. These chassis are not exactly low-power... with 18 drives in mine plus the SSDs, with the CPUs idling (load average is 0.00... I've got way more CPU than I really need), I average about 320 watts consumed. When I briefly had the chassis fully populated, I was well over 500 watts. This is out of the range of most "home" UPSs.

 

[snaptec]

Many points already written.
If the main purpose is esx storage go with some mirrored vdevs for that. Never use more than 50% of that pool.
If you use nfs, get a ssd or write performance will be horrible. If you can afford and need the performance, choose a NVMe one.
I think the esx will be connected with 10gbe?

For the video storage raidz2 is my favourite. The more vdevs, the more performance. RAM should be more than enough.

Nevertheless, you have a backup?

 

[belcboo]

Just an update in this thread after a couple of weeks:

- I have started to receive the pieces, I ended buying:

Chassis: CSE-847E16-R1K28LPB
Backplanes: BPN-SAS2-826EL1 and BPN-SAS2-846EL1
Motherboard: X9DRi-LN4F+
CPU: 2 x E5-2690 [8C@2.90Ghz]
RAM: 192GB [24x8GB]
NIC: 4 x Intel i350 and 2 x Chelsio T420-BT.
RAID Card: LSI 9211-8I
PSU: Dual 1400w Gold.
HDD's: 2 x Patriot Torch 60GB, 2 x Intel S3700 200GB, 24 x WD Red 5400RPM 8TB

I almost received everything, there's only a group of HDD's that Im waiting (I bought them in 3 different stores).

At this moment Im mounting all together and starting to do some testing with the drives I have, I'll try to keep this updated.

Many points already written.
I think the esx will be connected with 10gbe?

Yes, the ESXi server is one of the clients in 10gbe, and I also want to connect at least one work station to check how goes the edition working over the network.

Many points already written.
For the video storage raidz2 is my favourite. The more vdevs, the more performance. RAM should be more than enough.

Nevertheless, you have a backup?

I don't have a backup, we have a few little (~12TB) storages that we want to replace with this storage, and that was the reason why I was looking for the RAIDZ-3 lol. I do have a tape backup system and I'm checking if somehow I can use it to backup some files there just in case.

And basically that's it at this moment. Thank you for your help!