Prospective build

[colmconn]

I built my first Freenas-based NAS close to four years and I'm doing now what, in hindsight, I wish I had done then: ask for advice before building it. My MB died a couple of weeks ago and I was starting to run low on disk space. So, now I'm considering a rebuild and expansion.

My initial build was with the following hardware:
MB: Asrock Rack C2750 crowned with a 3D printed adapter and a Noctua NF-B9 to cool the otherwise hot CPU
HDD: Initially, 3x3TB WD Reds with 3 more added a year later and a 4TB Seagate (shucked from a USB enclosure) thrown in as a hot spare when I no longer needed it for other purposes. The 6 WD Reds are in a RAIDZ2 with the 4TB hot spare.
Boot SDD: 2x120GB SanDisk SSD (mirrored; previously used as a dedicated pool for jails, VMs, and the system dataset)
Jails SSD: 2x 240GB SanDisk SSD. Mirrored in a pool dedicated to jails, VMs and the system dataset.
RAM: 2x16GB ECC Crucial Kits from the MB QVL
PSU: Silverstone ST45SF 450W
Case: Silverstone DS380
UPS: Cypberpower 1350VA

At the time I bought the the case, I liked how I could put so many drives in such a small space. I quickly learned that due to how the air flowed in the case it would swiftly raise the HDDs to eye watering temperatures with the door closed. This lead me to add a 3D printed baffle next to the large gap between the 2 HDD intake fans and the HDD cage. I also removed the filter and added grills to help with air ingress and added some duck tape to cover up small gaps that the baffle didn't obstruct. This solved the HDD incineration problem. I now detest the case with a passion. It's too damn small, is difficult to stuff everything into and cable management is the stuff of nightmares. Even the smallest amount of work in it generally involved removing the cooling crown from the CPU and more than likely the drive cage.

Prompted by the failure of my MB and the need for more HDD space, I'm now considering a rebuild. I'm proposing to keep the following from the list above: The motherboard (a new warranty replacement one is currently on route to me from Asrock and should be here next Thursday), the RAM, the HDDs and SSDs.

This time, I'm proposing a build inspired by Stux's Norco build. I'm thinking of using a Norco RPC-3216 16 bay rack mount case as I intend to add at least another 6 HDDs to the main pool in the next few months. I was looking at the Supermicro CSE-836 cases on ebay but they, by and large, come with dual 1000W PSU that my research tells are noisy. That would be fine in a data center, but not for my use case which is at home in my office. However, since I want to keep the MB, I need a HBA.
(I don't currently need the horse power of a Xeon though the case would afford replacing the MB with, for example, a Supermicro X10 and Xeon later should I need/choose). Since the MB only has a single PCIe 2 8x slot I'm thinking of something like a LSI SAS9201-16i married with 4x SFF-8087 1.6FT/0.49m cables. I may change out all of the fans in the case for Noctua fans but before deciding to do so, I'll see if the fans supplied with the case are obnoxiously loud. The only calculation I've not done yet is one for power. I suspect that something like a Corsair TX750M will suffice to get sufficient molex plugs for the drives and maybe leave room for upgrading the MB and CPU in the future.

So to summarize, the new build I'm thinking of is:
MB: Asrock Rack C2750 with cooling crown
RAM: 2x16GB ECC Crucial Kits from the MB QVL
HBA: LSI SAS9201-16i
Cables: 4x SFF-8087 1.6FT/0.49m and 2x SATA for mirrored SSDs to be attached to the SATA ports on the MB
HDD: 6x3TB + 1x 4TB hot spare in RAIDZ2 to be augmented with a further 6x WD Reds (probably 4TB).
Boot SDD: 2x120GB SanDisk SSD (mirrored)
Jails SSD: 2x 240GB SanDisk SSD. Mirrored in a pool dedicated to jails, VMs and the system dataset.
Possible PSU: Corsair TX750M

Before I go and pull the trigger on this new build, I'd very much appreciate any advice you may have.

 

[Yorick]

> 4x3TB + 1x 4TB hot spare in RAIDZ2

Done that way because it's an existing pool? It sounds like you're going to do two pools, given that the vdevs are different sizes. Arguably, in that case, raidz3 is preferable over a hot spare: If you're building the vdev from scratch. See also https://serverfault.com/questions/883065/zfs-hot-spares-versus-more-parity

 

[Chris Moore]

This time, I'm proposing a build inspired by Stux's Norco build. I'm thinking of using a Norco RPC-3216 16 bay rack mount case as I intend to add at least another 6 HDDs to the main pool in the next few months.

If you are considering a 16 bay Norco chassis, they run about $320 new with no power supply. These Supermicro chassis are less expensive and include redundant power supplies. A better deal if you ask me.
https://www.ebay.com/itm/Supermicro...addy-SAS836A-REV3-2-2x-1000w-PSU/264185334924

I was looking at the Supermicro CSE-836 cases on ebay but they, by and large, come with dual 1000W PSU that my research tells are noisy. That would be fine in a data center, but not for my use case which is at home in my office.

I have two of those in my office and the sound is not enough to bother me. Meaning that they are not very loud, but many people are more sensitive to sound than I am. It is less than the oscillating fan that I run in the same room just to circulate air.

 

[colmconn]

> 4x3TB + 1x 4TB hot spare in RAIDZ2

Done that way because it's an existing pool? It sounds like you're going to do two pools, given that the vdevs are different sizes. Arguably, in that case, raidz3 is preferable over a hot spare: If you're building the vdev from scratch. See also https://serverfault.com/questions/883065/zfs-hot-spares-versus-more-parity

That was a typo, now corrected. It’s was pre-existing 6x4TB Z2 pool. I had nothing better to do with the 4TB drive so I threw it in as spare

 

[Chris Moore]

That was a typo, now corrected. It’s was pre-existing 6x4TB Z2 pool. I had nothing better to do with the 4TB drive so I threw it in as spare

It would be better to keep it as a cold-spare so it is not sitting there accumulating power on hours. If you have a failure, then put it in the pool, unless the system is in a location where nobody can attend to it within a day or two of a drive fault. You should also be running monitoring scripts to track the health of the system:

Github repository for FreeNAS scripts, including disk burnin
https://forums.freenas.org/index.ph...for-freenas-scripts-including-disk-burnin.28/

 

[colmconn]

If you are considering a 16 bay Norco chassis, they run about $320 new with no power supply. These Supermicro chassis are less expensive and include redundant power supplies. A better deal if you ask me.
https://www.ebay.com/itm/Supermicro...addy-SAS836A-REV3-2-2x-1000w-PSU/264185334924

I have two of those in my office and the sound is not enough to bother me. Meaning that they are not very loud, but many people are more sensitive to sound than I am. It is less than the oscillating fan that I run in the same room just to circulate air.

Do you happen to know if there are any limitations, in terms of bandwith, imposed by that backplane? That is does it 6Gb/s drives at that speed rather than downgrading them to 3Gb/s. The latter would only happen if the back-plane included SAS1 expander chip which I'm almost certain this does not have.

Also, it's SFF-8077 cabled rather than needing break-forward SFF-8077-SATA cables, right? Try as I might, I can't figure it out from the supermicro docs.

 

[Chris Moore]

Do you happen to know if there are any limitations, in terms of bandwith, imposed by that backplane? That is does it 6Gb/s drives at that speed rather than downgrading them to 3Gb/s. The latter would only happen if the back-plane included SAS1 expander chip which I'm almost certain this does not have.

If we are talking about the link to a Supermicro 16 bay chassis that I posted, it does not have any limitation imposed.

A better deal if you ask me.
https://www.ebay.com/itm/Supermicro...addy-SAS836A-REV3-2-2x-1000w-PSU/264185334924

The seller description says it is a SAS836TQ backplane. That means it is connected with forward breakout cables.
Looks like this on the back:

You have options with that, you can use a single SAS HBA to control it by adding your own SAS expander card or you can use two SAS HBA cards. I would go the expander route. The seller didn't show the inside of the chassis and they don't say if it is already cabled or not. I would hold off on buying cables until I got the chassis and look to see what is there.

Here is a collection of videos that you should look at to tell you more about SAS cabling and SAS expanders:
https://forums.freenas.org/index.php?resources/informational-videos-mostly-about-sas-hardware.105/

Particularly, you probably want to look at this one:

Explaining the IBM SAS-2 expander and how to do 24xHDD setup with only 2-port SAS controller
https://www.youtube.com/watch?v=qccpopxc_Uo

 

[Chris Moore]

HBA: LSI SAS9201-16i

I just noticed this. You can get a SAS HBA and a SAS expander for less than they are asking for this 16 lane card. It is really not needed, even if you are running an all SSD pool, you would want to go with a card that has PCI-E 3.0 instead of this older tech PCI-E 2.0 card. In this card the older PCI-E interface becomes the bottleneck to performance.

 

[colmconn]

If we are talking about the link to a Supermicro 16 bay chassis that I posted, it does not have any limitation imposed.

The seller description says it is a SAS836TQ backplane. That means it is connected with forward breakout cables.
Looks like this on the back:
View attachment 28327
You have options with that, you can use a single SAS HBA to control it by adding your own SAS expander card or you can use two SAS HBA cards. I would go the expander route. The seller didn't show the inside of the chassis and they don't say if it is already cabled or not. I would hold off on buying cables until I got the chassis and look to see what is there.

Here is a collection of videos that you should look at to tell you more about SAS cabling and SAS expanders:
https://forums.freenas.org/index.php?resources/informational-videos-mostly-about-sas-hardware.105/

Particularly, you probably want to look at this one:

Explaining the IBM SAS-2 expander and how to do 24xHDD setup with only 2-port SAS controller
https://www.youtube.com/watch?v=qccpopxc_Uo

Thanks for clarifying the bandwith issue. The ad says its got a SAS836A in the title and a SAS836TQ in the body, hence my confusion. Unfortunately, I couldn't use an expander card as the C2750 MB only has one PCIe 2 slot. Useful video, nevertheless.

 

[Chris Moore]

Thanks for clarifying the bandwith issue. The ad says its got a SAS836A in the title and a SAS836TQ in the body, hence my confusion. Unfortunately, I couldn't use an expander card as the C2750 MB only has one PCIe 2 slot. Useful video, nevertheless.

The expander card only uses the PCI-E slot for power, not data. There are many ways to provide power.

 

[Bozon]

I built my first Freenas-based NAS close to four years and I'm doing now what, in hindsight, I wish I had done then: ask for advice before building it. My MB died a couple of weeks ago and I was starting to run low on disk space. So, now I'm considering a rebuild and expansion.

My initial build was with the following hardware:
MB: Asrock Rack C2750 crowned with a 3D printed adapter and a Noctua NF-B9 to cool the otherwise hot CPU
HDD: Initially, 3x3TB WD Reds with 3 more added a year later and a 4TB Seagate (shucked from a USB enclosure) thrown in as a hot spare when I no longer needed it for other purposes. The 6 WD Reds are in a RAIDZ2 with the 4TB hot spare.
Boot SDD: 2x120GB SanDisk SSD (mirrored; previously used as a dedicated pool for jails, VMs, and the system dataset)
Jails SSD: 2x 240GB SanDisk SSD. Mirrored in a pool dedicated to jails, VMs and the system dataset.
RAM: 2x16GB ECC Crucial Kits from the MB QVL
PSU: Silverstone ST45SF 450W
Case: Silverstone DS380
UPS: Cypberpower 1350VA

At the time I bought the the case, I liked how I could put so many drives in such a small space. I quickly learned that due to how the air flowed in the case it would swiftly raise the HDDs to eye watering temperatures with the door closed. This lead me to add a 3D printed baffle next to the large gap between the 2 HDD intake fans and the HDD cage. I also removed the filter and added grills to help with air ingress and added some duck tape to cover up small gaps that the baffle didn't obstruct. This solved the HDD incineration problem. I now detest the case with a passion. It's too damn small, is difficult to stuff everything into and cable management is the stuff of nightmares. Even the smallest amount of work in it generally involved removing the cooling crown from the CPU and more than likely the drive cage.

Prompted by the failure of my MB and the need for more HDD space, I'm now considering a rebuild. I'm proposing to keep the following from the list above: The motherboard (a new warranty replacement one is currently on route to me from Asrock and should be here next Thursday), the RAM, the HDDs and SSDs.

This time, I'm proposing a build inspired by Stux's Norco build. I'm thinking of using a Norco RPC-3216 16 bay rack mount case as I intend to add at least another 6 HDDs to the main pool in the next few months. I was looking at the Supermicro CSE-836 cases on ebay but they, by and large, come with dual 1000W PSU that my research tells are noisy. That would be fine in a data center, but not for my use case which is at home in my office. However, since I want to keep the MB, I need a HBA.
(I don't currently need the horse power of a Xeon though the case would afford replacing the MB with, for example, a Supermicro X10 and Xeon later should I need/choose). Since the MB only has a single PCIe 2 8x slot I'm thinking of something like a LSI SAS9201-16i married with 4x SFF-8087 1.6FT/0.49m cables. I may change out all of the fans in the case for Noctua fans but before deciding to do so, I'll see if the fans supplied with the case are obnoxiously loud. The only calculation I've not done yet is one for power. I suspect that something like a Corsair TX750M will suffice to get sufficient molex plugs for the drives and maybe leave room for upgrading the MB and CPU in the future.

So to summarize, the new build I'm thinking of is:
MB: Asrock Rack C2750 with cooling crown
RAM: 2x16GB ECC Crucial Kits from the MB QVL
HBA: LSI SAS9201-16i
Cables: 4x SFF-8087 1.6FT/0.49m and 2x SATA for mirrored SSDs to be attached to the SATA ports on the MB
HDD: 6x3TB + 1x 4TB hot spare in RAIDZ2 to be augmented with a further 6x WD Reds (probably 4TB).
Boot SDD: 2x120GB SanDisk SSD (mirrored)
Jails SSD: 2x 240GB SanDisk SSD. Mirrored in a pool dedicated to jails, VMs and the system dataset.
Possible PSU: Corsair TX750M

Before I go and pull the trigger on this new build, I'd very much appreciate any advice you may have.

Just a quick question, did you contact Asrock to get a replacement board? Did it die because of the well know issue with intel atom chips?

Thanks.

 

[colmconn]

Just a quick question, did you contact Asrock to get a replacement board? Did it die because of the well know issue with intel atom chips?

Thanks.

Yes. I RMAed it. I mailed it the week before last and they sent an email on Friday night (Feb 08) to say a new one is on the way. It should be here on Thursday. They never stated what the exact failure mode was. In the grand scheme of things I don't care so long as its being replaced free of charge. Having said that, the failure mode was consistent with the failure of the atom chip itself. The BMC continued to work. If yours has failed, contact them, you've nothing to loose. I would advise ruling out PSU issues before you contact them so you can include details of such in the email to William. This, in my experience, will help expedite the RMA process.

 

[colmconn]

Update time.

I rebuilt the machine and it's been doing it's thing for about the last 2 weeks flawlessly.

The following is the parts list, some old, some new, and some old but new to me:
MB: Asrock Rack C2750 crowned with a 3D printed adapter and a Noctua NF-B9 to cool the CPU. The CPU on this MB doesn't appear to run as hot as my original board. I guess the coupling between the CPU and the cooler is much better this time round.
HDD: 6x3TB WD Reds in a RAIDZ2 with 6x4TB in a RAIDZ2 vdev. Stripped.
Boot SDD: 2x120GB SanDisk SSD; mirrored.
Jails SSD: 2x 240GB SanDisk SSD. Mirrored in a pool dedicated to jails, VMs and the system dataset.
HBA: 1x LSI SAS9201-16i flashed to the P20 IT firmware. In this case the flashing could only be done from the EFI shell rather than through FreeDOS. I have extensive notes on how to do this in case any one wants them.
Drive cables: 4x Mini SAS 36Pin (SFF-8087) Male to 4 SATA 7Pin Female Cable, Mini SAS Host/Controller to 4 SATA Target/Backplane
2.5-to3.5 inch drive adapters: 4 of these to fit the SSDs in the caddy slots of the case. The adapters work very well and are well built.
RAM: 2x16GB ECC Crucial Kits from the MB QVL
PSU: 2x980W Supermicro PSU that came with the case below
Case: Supermicro CSE-836 Chassis w/ 16x 3.5" Caddy
UPS: Cypberpower 1350VA

Other stuff that I discovered after the fact that were necessary
1x Motherboard power extension: The ATX power connector in the case was too short to reach the diminutive motherboard
1x bag of screws to secure the HDDs and SSDs in the caddys (Supermicro part #: MCP-410-00005-0N)
1x Supermicro 11.81-Inch 16-Pin Front Panel Split Cable (CBL-0068L) to connect the front panel to the headers on the motherboard.
2x fan splitter cables. There are more fans than there are headers on the MB.

I'm currently only using only one of the PSUs that came with the case. (They're noisy.) Together with a Cable modem, UBNT USG3, 8-port UBNT PoE switch, an UBNT UAP-AC-Lite and the UPS, the whole lot only draws about 118-120W when idle.

I've not messed with the fan profiles in the IPMI yet the drives are currently hovering around the 30C-ish mark.

Some pics are below:

 

[Bozon]

... HBA: 1x LSI SAS9201-16i flashed to the P20 IT firmware. In this case the flashing could only be done from the EFI shell rather than through FreeDOS. I have extensive notes on how to do this in case any one wants them. ...

That would be a good thing to post in the resource section of this forum.