2nd try and a couple of questions

[carlob]

Hello there, still reading a lot of docs but I think I came out with something that could work even if I need some help to finalize the parts I need to buy.

Let's start from my requirements: this server should be able to run Freenas with the aim to be a first AND second backup for my data, to run Plex (trascoding maybe one stream to 4k) and for a couple of VMs. So it won't be only a NAS but also my home (lab) server. And now comes my first question:

1) should I install Freenas and run everything out of it or maybe run Freenas in a VM? VMware?

The case I have selected is a tower (because my rack is a 60cm or 23.5" inches of depth, I only have Unifi switches there), precisely the Supermicro 743TQ-865B-SQ with 8 x 3.5" hot-swap drives and 3 x standard 5.25" drive bay. Now there are accessories to make it a 8 x 3.5" hot-swap drives plus 8 x 2.5" hot-swap drives plus 2 x 2.5" fixed drives (I'd use these for 2 SSDs). So there could be 16+2=18 drives at the most:

2) will I need a SAS HBA card and an expander card (and forward breakout cables)? Motherboard, which will be a Supermicro X10SRi-F has only ten SATA ports so I'd better off connecting the 18 drives to the SAS card? or maybe 10 on the motherboard and 8 on the SAS card?

Next is ZFS: I'd like to setup two pools and sync them maybe once a day. The idea is to have the first pool as a primary NAS that backups on the second pool like it was a secondary NAS (I'll have an external for a third backup):

3) Is this possible in Freenas? Ideas on the pool sizes? I was thinking to setup the first pool in Z2 and the second in Z1. Assuming 8 X 4Tb HD for the first pool I should have about 17/18Tb of storage capacity, if I use 7 X 4Tb in the second pool I have 18Tb of storage so a backup from one pool to the other should be feasible (?) and I have a spare slot for another 2.5"HD. Is it overkill and a waste of money or it's ok?

4) What kind of mirrored SATA SSD should I use?

5) Any other thing I should consider HD wise?

Hardware list is as follows:

Supermicro 743TQ-865B-SQ case that comes with a 865W 80-Plus PSU
Supermicro X10SRi-F MB
E5-2680 v3 CPU (18467 Passmark - 120 TDP) Too much?
Noctua NH-U9DXi4 cooler (need to check if fits)
8 X Seagate IronWolf NAS 4 TB 3.5"
8 X Seagate IronWolf NAS 4 TB 2.5"
2 X Unknown SSD
4 X 16Gb ECC RAM (selected according to the Supermicro compatibility list)
1 X SAS HBA IT mode like this one
1 X SAS expander like this one
a lot of forward breakout cables
money, here is going to be quite an expensive build

i know that it's a very long message but I'd really appreciate your comments or suggestions!
Thanks a lot
C.

 

[carlob]

too long?

 

[HoneyBadger]

I assume this will be your only system.

1) Normally, FreeNAS works best when it is left to be a NAS/SAN only - while there is the ability to create VMs via bhyve, I wouldn't say the hypervisor is at a level where I'd suggest it for day-to-day use; however, installing FreeNAS as a VM in VMware brings its own particular set of issues up. It can be done, it just needs to be handled with care and done right.

2) If you are doing FreeNAS in a VM, you'll need two separate SAS/SATA devices - one that can be addressed by the host OS (ESXi) and another that is passed through to your FreeNAS VM via PCI passthrough. Regarding cabling; have you considered using the 743A series Supermicro chassis? They have SAS cabled backplanes, so you wouldn't need to use breakout cables, rather just an SFF-to-SFF SAS cable. Would help to reduce the spaghetti.

3) If you consider the data valuable enough to be backed up onto a separate pool, then this is absolutely a good ideal. Only you can decide what "overkill" really means. But if you are after performance, I would suggest mirrors for the first pool, and a RAIDZ2 for the secondary "backup" pool.

4) Avoid cheap, DRAMless TLC SSDs like the WD Green and other bargain-basement options. There seems to be a long-standing bug with their TRIM implementation that causes data integrity issues. I don't know if it's been isolated down to a specific controller or NAND configuration yet. If it's just for booting, find some off-lease Intel DC series SATA SSDs with a decent amount of life left and use those.

5) High-capacity 2.5" HDDs tend to be SMR - shingled magnetic recording. What's the exact model of those IronWolf 4TB 2.5" drives? I can only find the Barracuda line in that form factor/size, and those (ST4000LM024) are SMR.

 

[carlob]

I assume this will be your only system.

1) Normally, FreeNAS works best when it is left to be a NAS/SAN only - while there is the ability to create VMs via bhyve, I wouldn't say the hypervisor is at a level where I'd suggest it for day-to-day use; however, installing FreeNAS as a VM in VMware brings its own particular set of issues up. It can be done, it just needs to be handled with care and done right.

2) If you are doing FreeNAS in a VM, you'll need two separate SAS/SATA devices - one that can be addressed by the host OS (ESXi) and another that is passed through to your FreeNAS VM via PCI passthrough. Regarding cabling; have you considered using the 743A series Supermicro chassis? They have SAS cabled backplanes, so you wouldn't need to use breakout cables, rather just an SFF-to-SFF SAS cable. Would help to reduce the spaghetti.

3) If you consider the data valuable enough to be backed up onto a separate pool, then this is absolutely a good ideal. Only you can decide what "overkill" really means. But if you are after performance, I would suggest mirrors for the first pool, and a RAIDZ2 for the secondary "backup" pool.

4) Avoid cheap, DRAMless TLC SSDs like the WD Green and other bargain-basement options. There seems to be a long-standing bug with their TRIM implementation that causes data integrity issues. I don't know if it's been isolated down to a specific controller or NAND configuration yet. If it's just for booting, find some off-lease Intel DC series SATA SSDs with a decent amount of life left and use those.

5) High-capacity 2.5" HDDs tend to be SMR - shingled magnetic recording. What's the exact model of those IronWolf 4TB 2.5" drives? I can only find the Barracuda line in that form factor/size, and those (ST4000LM024) are SMR.

Thank you, this is really helpful and I appreciate the time you spent to read through the entire thing and your points.
Yes, this is going to be my only server or at least I hope so. My home environment is made by several iMacs, Macbook Pros, Mac minis and IOS devices so I pretty much live in a Apple ecosystem, actually I need to run some Windows stuff so I have Parallels installed on a couple of machines: one of my goals is to get rid of it and run a centralized Windows VM. Besides that my needs are mostly backup, Plex and maybe move the Unifi controller to another VM.

I'll look into the 743A series Supermicro chassis as your suggestion makes perfect sense, in the meantime I found out a couple of compact 4U chassis that could work in my rack like the Silverstone RM400, need to investigate before committing to a tower chassis.

My reference to "overkill" is something that is bothering me: since is generally impossible in Europe to buy server stuff (e.g. Supermicro) dead cheap as in the US most of the parts have to come new or priced very close to new. To give you an idea I'm going to spend something in the ballpark of 2k euro for this machine which is about 2,2K $.

Now is it worth the hassle (money and time) to put together a huge monster made of server components, 18 HDs and the like, spend more than 2K to only have 17/18Tb of space? Should I just buy a 4-bay Synology NAS, stick 4 12TB drives in RAID10 and call it a day?

 

[HoneyBadger]

The real question is what you plan to do with the storage. If the option of the 4-bay Synology is viable, then I would have to assume that you could also build a smaller, less-overkill FreeNAS machine using the same 4x12TB drives and store your bulk media there, while also adding a pair of moderately sized SSDs that you could use for VMs.

When considering how many backups to have (ie: your primary pool, syncing to a second one) I generally feel that any data valuable to have that kind of protection should have those copies on a separate system, possibly even offsite. This could be a cloud-tier backup for your most critical pieces, but obviously if you want to back up several TB this becomes less cost-effective.

For most media storage I don't consider that valuable enough to justify the expense. Yes, the cost of my time might be frustrating if I had to re-rip CDs and DVDs, but they are recoverable eventually. In the worst-case scenario where a fire consumes both my storage and my original media; well, I have house insurance.

But house insurance can't retake pictures or recreate critical documents. So those are backed up offsite as well.

 

[carlob]

The real question is what you plan to do with the storage. If the option of the 4-bay Synology is viable, then I would have to assume that you could also build a smaller, less-overkill FreeNAS machine using the same 4x12TB drives and store your bulk media there, while also adding a pair of moderately sized SSDs that you could use for VMs.

When considering how many backups to have (ie: your primary pool, syncing to a second one) I generally feel that any data valuable to have that kind of protection should have those copies on a separate system, possibly even offsite. This could be a cloud-tier backup for your most critical pieces, but obviously if you want to back up several TB this becomes less cost-effective.

For most media storage I don't consider that valuable enough to justify the expense. Yes, the cost of my time might be frustrating if I had to re-rip CDs and DVDs, but they are recoverable eventually. In the worst-case scenario where a fire consumes both my storage and my original media; well, I have house insurance.

But house insurance can't retake pictures or recreate critical documents. So those are backed up offsite as well.

I have about 4 TB of pictures and videos plus my music library (a couple of TB) and this stuff is not recoverable in case of loss, plus less of a TB of misc documents. This stuff is really important to me and need to be protected. In addition to that I'd like this server to be a central backup location for all my computers (currently an iMac, a Mac mini, a Macbook Pro and a windows machine so total of 4 computers each with 1 TB SSD onboard) and for ios devices (iphone, couple of ipads) and a movies repository for Plex.

I have already got most of the hardware, need to make a decision on chassis (half size rack/tower) and number/type of HDs but I think I will go with 8 HDs (2 for parity) and an external hp microserver (4-bay) for backup. The idea to have two pools is nice but after further consideration it is probably better to have an external device that you can keep offsite.

 

[carlob]

Another idea would be to buy a couple of 4U half-size rack chassis, put the server in one of them and use the others for a DIY jbod of n disks, each used as a pool. It should not be difficult using a cheap motherboard and a SAS expander. Need to investigate.

 

[Jessep]

I have already got most of the hardware,

What hardware do you already have? If this is old consumer/gaming gear rethink your solution.

This stuff is really important to me and need to be protected

If your data is important put in the time and money to protect it.

It's much more important to have a good backup strategy than a kick ass server.
https://www.backblaze.com/blog/the-3-2-1-backup-strategy/

Based on what I have read in your posts so far, I suggest developing your backup strategy BEFORE building any servers. You will be much better off if your data is protected before you start messing around.

 

[carlob]

What hardware do you already have? If this is old consumer/gaming gear rethink your solution.


If your data is important put in the time and money to protect it.

It's much more important to have a good backup strategy than a kick ass server.
https://www.backblaze.com/blog/the-3-2-1-backup-strategy/

Based on what I have read in your posts so far, I suggest developing your backup strategy BEFORE building any servers. You will be much better off if your data is protected before you start messing around.

Hardware is in my first post, I'm already got MB, CPU, HBA card and RAM, waiting to buy a chassis, PSU and the hard disks. Please point me where I mentioned gaming hardware or gaming at all. My backup strategy is clear: i want my first pool synced with my second pool and I will have an external device (portable) for another backup. I didn't ask for a lecture on backups, i was just looking for suggestions on hardware and ways to implement that e.g. how many hard disk in a pool, ways to backup first pool on second, how many parity disks, etc or suggestion on how to optimize it HDs wise. I understand that my post was a long one and not asking to read it all but if you want to reply at least take 5 mins to read it.

Since I have an half size rack I was also looking for chassis options, but I already found the solution myself that would be another couple of jbod half size cases. Anyways thanks for your reply to questions I didn't ask and thanks everybody I promise i will not bother you anymore on this so friendly forum.

 

[HoneyBadger]

Another idea would be to buy a couple of 4U half-size rack chassis, put the server in one of them and use the others for a DIY jbod of n disks, each used as a pool. It should not be difficult using a cheap motherboard and a SAS expander. Need to investigate.

This might be a good option, but make sure to compare the total cost of the piecemeal solution against a refurbished enterprise SAS JBOD such as the Lenovo SA120, EMC KTN-STL3, Dell MD1200, or HP D2600. Depending on your regional market and eBay prices, it may be cheaper to grab a used example of those; although they will be full-depth which might negate the benefits.

 

[Herr_Merlin]

The EMC expansion Shelves should fit into an half sized rack as they are freaking short

 

[HoneyBadger]

The EMC expansion Shelves should fit into an half sized rack as they are freaking short

Good point; for some reason I forgot they were around half the depth of the others.

 

[Herr_Merlin]

35cm deep are those EMC shelves of an CX-3-20f we still have packed away in our storage. So that should fit with cables etc.

 

[Glorious1]

I'd like to setup two pools and sync them maybe once a day. The idea is to have the first pool as a primary NAS that backups on the second pool like it was a secondary NAS (I'll have an external for a third backup):

3) Is this possible in Freenas?

This is what I do. Just set up daily snapshots on the main pool, and replication to the second pool. In the replication task, the Remote Hostname will be localhost.