Xeon Gold 6144. My first NAS

[TFAiSO]

Hi,

I have just acquired a few pieces to build my first FreeNAS!
Parts:

2x: CPU: Intel Xeon Gold 6144 (3.5GHz, 8 cores, 16 threads, Turbo 4.2GHz)
1x MB: Supermicro X11DPH-T-O
1x Chassis: SuperChassis CSE-847BE1C4-R1K23LPB (36 drive bays)
16x RAM: Samsung M393A4K40BB1-CRC (DDR4 32 GB ECC )
13x HDD: Seagate Enterprise Capacity 3.5 HDD ST10000NM0016 (10TB, SATA3)
...and some cables, coolers, etc.
UPDATE:
1 x SAS PCIe-card: Lenovo/IBM M1215 (with two internal mini-SAS HD connectors SFF-8643)
1 x Switch: Cisco SG350XG-24T (10 GBase-T)

I will let you know how it goes. I will get the last piece tomorrow, the CPU’s.

The intention is to store movies, handle 5 simultaneous transcodes (some HVEC) and to store backups. I plan to use Z2 parity. Software: PMS, Radarr, Crashplan.

What’s missing is a good online UPS.

 

[MrToddsFriends]

2x: CPU: Intel Xeon Gold 6144 (3.5GHz, 8 cores, 16 threads, Turbo 4.2GHz)
1x MB: Supermicro X11DPH-T-O
16x RAM: Samsung M393A4K40BB1-CRC (DDR4 32 GB ECC )

Why 16 DIMMs? LGA 3647 CPUs have 6 memory channels so 12 DIMMs would have been a natural choice for 2 CPUs. Of course the expected performance hit caused by a mixed 1DPC/2DPC configuration most possibly won't be perceptible in NAS utilization.

Edit:
And, of course, please let us know if there a difficulties during putting into service of this beast. Welcome to the forum, BTW.

 

[Ericloewe]

I'm speechless at the amount of overkill presented here...

 

[SweetAndLow]

Use case and workflow? Pool layout?

 

[TFAiSO]

Use case and workflow? Pool layout?

I had no idea about 12 RAM banks vs. 16 RAM banks. I think I'm stuck with the 16 DIMM's I have now...

The use case is to be able to transcode as many HEVC streams as possible. That codec is quite processing intensive and I didn't want to break out the processing part to another machine.

There's nothing that's really 'future proof' but 36 bays should take a while to fill up, hence that chassis.

I haven't set on the final pool layout yet. I'm a newbie to ZFS. I'm thinking in the terms of the below.
10 drives in total:
vdevA raidZ3: drive 1-6 "useful", drive 7,8,9 parity drives. drive 10: disconnected spare.

I just realised I forgot to get two Sandisk USB's for boot drive. This case do not have a DOM port :/

"there's no better kill than overkill" :)

 

[MrToddsFriends]

I had no idea about 12 RAM banks vs. 16 RAM banks. I think I'm stuck with the 16 DIMM's I have now...

Page 37 of your motherboard manual has some details. As I wrote it should be no problem to use all your 16 DIMMs when used with FreeNAS.

 

[BigDave]

I just realised I forgot to get two Sandisk USB's for boot drive. This case do not have a DOM port :/

"there's no better kill than overkill" :)

Sorry Sir, but thumb drives are NOT overkill.

 

[Arwen]

...
I just realised I forgot to get two Sandisk USB's for boot drive. This case do not have a DOM port :/
...

Yes, if you have 10 data disks, you don't have a free SATA port for a DOM.

This system board DOES come with 2 SATA ports that would supply power to a SATA DOM, (or 2).

 

[BigDave]

There are also two M.2 slots on that board as well.

THAT at least gives you some overkill potential

 

[Chris Moore]

...and some cables, coolers, etc.

Since you have 36 bays, you may as well get the SAS controller for the system now. What are you thinking of or would you like a suggestion?

 

[Chris Moore]

Yes, if you have 10 data disks, you don't have a free SATA port for a DOM.

The chassis has expander backplanes, he will need to get a SAS controller to run them and the onboard SATA will not be used, except for boot drives.
The chassis has two rear mounted 2.5" hotswap bays that could be used for boot drives, so SATA DOMs are not strictly needed. It might be better to use some 32GB SSDs like I think @BigDave recommended before

 

[Chris Moore]

2x: CPU: Intel Xeon Gold 6144 (3.5GHz, 8 cores, 16 threads, Turbo 4.2GHz)

$2925.00 each... I could have used that money to buy a car.

 

[BigDave]

The chassis has two rear mounted 2.5" hotswap bays that could be used for boot drives, so SATA DOMs are not strictly needed. It might be better to use some 32GB SSDs like I think @BigDave recommended before

If the OP is spending 7K on processors, he should get:
SM961 NVME 128GB Samsung to run his operating system, I would definitely take advantage of those slots!

 

[TFAiSO]

Hi guys,
First of all a big thanks for your replies.
The 961... hmm I’m thinking of using the NVME ports for cache-drives later on, if needed. I’ll see where this box will take me first.
After I posted about the thumb drives I remembered USB disks are not always that good.
I’m thinking of an LSI/Avago/Broadcom 9305-24i. The drivers should be mature enough by now and I have seen forum members using them without hassle. Any better suggestion? I’ll put two 2.5” SSD’s for boot drives in the hot swap bays.

I will use 10Gbit networking. Two Chelsio will talk between a Cisco.

 

[Stux]

A single Optane 32MB NVMe M.2 card would be a good boot device.

Still leaves another M.2 for cache use.

 

[Stux]

BTW, your chassis uses dual sas expander backplanes, and will require a sas controller... which will leave you with plenty of free SATA ports.

I didn't see a built in SAS controller on that motherboard, but perhaps you should consider a model which does have one

 

[danb35]

I’m thinking of an LSI/Avago/Broadcom 9305-24i.

A -24i is (like much of your build) overkill, but it wouldn't even do anything for you. A -8i card would be fine--run one SAS cable to the front backplane, and another to the rear one.

 

[wblock]

Optane is overkill for a boot device, but better than USB. Also, can FreeNAS boot from an Optane device? I don't know, never had one to test. I'd suggest a mirror of standard, cheap SSDs. Some people say using 120G or even 64G drives is a waste, but I disagree. That extra space is useful for boot environments and wear leveling.

 

[Ericloewe]

Also, can FreeNAS boot from an Optane device?

Sure, it's just plain NVMe (slowish NVMe). The Optane part is Intel marketing for "Chipset-enabled virtual hybrid hard drive" and comes with a bunch of strings attached - but you can ignore all that and use it as a regular SSD.

I'd suggest a mirror of standard, cheap SSDs.

It's really a no-brainer, if the chassis even has two hot swap bays in the back for precisely that use.

 

[TFAiSO]

Sure, it's just plain NVMe (slowish NVMe). The Optane part is Intel marketing for "Chipset-enabled virtual hybrid hard drive" and comes with a bunch of strings attached - but you can ignore all that and use it as a regular SSD.


It's really a no-brainer, if the chassis even has two hot swap bays in the back for precisely that use.

Hiya,

Your replies make sense. I have settled on using the two hotswap bays for 2 basic SSD drives. I'm thinking of the Intel® SSD E 5400s Series ,they are about 50 bucks each. I haven't ordered them yet. I have also left the idea of using USB or NVME as boot devices. With 80GB I intend to set them up to handle wear leveling.

-8i is fine if I intend to only use 2,3 drives or so (up to 8 drives). I was thinking using the -24i so I can hook up more SATA or SAS drives later on. My intention is to use those drives as a second pool, vdevB. There is no SAS controller on the MB so I would need a such card to utilize all the bay possibilities.