First NAS build - Supermicro-X11SSL-CF

[Platy]

Hi,

I'm planning to build my first NAS build, it would be great if you could give me some feedback:

• MOTHERBOARD: Supermicro-X11SSL-CF (Amazon)
• CPU: Xeon E3-1230 v6 (Amazon)
• CPU COOLER: Noctua NH-U14S (Amazon)
• RAM: Kingston Technology ValueRAM 16GB DDR4 2400MHZ ECC Module 16GB DDR4 2400MHz (Amazon)
• BOOT: Samsung MZ-V7E500BW SSD 970 EVO 500GB NVMe (Amazon)
• HD: 2x Western Digital WD Red HDD 6000 GB, SATA III, 6000 Mb/s, 5400 RPM, 64 MB (Amazon)
• CASE: Fractal Design FD-CA-DEF-R5-BK Define R5 Case (Amazon)
• PSU: Be quiet! Pure Power 11 500W (Amazon)

Data:
I have around 2TB "important data" to backup from 6 devices. The clients (Notebook, Tablet, Mobilephone, Workstation, IOT) will save the data on the NAS box (sftp, smb, nextcloud) which i will backup daily to cloud (not yet decided if zfs send or borgbackup+rsync+wasabi).

The 2TB don't include media, so I expect to add more storage very fast (at the moment I'm favoring mirror vdevs instead of raidz2, faster resilvering, easier to upgrade). I want to stream the media (I guess at the beginning around 1-2TB) through Plex to 6 devices (appletv,firestick,rasperrypi..) there will be probably never be more than 2 transcoded streams at the same time.

Connectivity:
Unfortunately most devices are connected only through AC WiFI and the other GbE so thats the limiting factor. In the future I want to connect my workstation directly through 10GbE to the NAS.

Work:
The build is way too overkill for only streaming, however I would like to play around with VM's (Plex, Hass, ELK Stack, Mailserver, nextcloud, some backtesting...).


Thank you very much!

 

[KevDog]

I have the Mobo and it works great -- haven't had any problems. Needed some fan splitters however since the MOBO only has FAN1-4 and FANA.
In terms of boot drive - honestly you don't need much space. I'm personally booting of a 64 Gb SATA-DOM drive which is mirrored to a second SATA-DOM. When I was building my system, others told me that 64Gb was overkill. I thought I new better. I just looked --- I'm using less than 1 Gb of space on the boot drive. Oh well. I know many use thumb drives. Others a small hard drive. I'm using SATA-DOM modules since the mobo has the headers for 2 drives
Storage space - So basically 2 - 6Tb drives but with only 6Tb of usable space? I'd probably add more storage. VM's start to take up a lot of space. It just depends on your needs.

 

[Platy]

Hi KevDog,

thank you very much for your feedback. Thanks about the fan splitter did not know that, in your opinion should this one work? Amazon. I guess you are right regarding the boot drive size, maybe I should use two smaller mirrored SSD drives. Excuse my ignorance how are the supermicro SAT DOM's cost/speedwise vs SSD's?

I'm a bit unsure about the CPU, if it makes sense to buy a cheap Pentium G4400 and buy the Xeon in a few months hoping the prices go down or do think the CPU prices will be kinda stable? As i guess that the first few months probably I wont use heavily VM's but will be more busy to setup everything. [edit] found this thread: https://forums.freenas.org/index.php?threads/x11ssl-cf-boot-time-bios-and-kaby-lake-g4560.55475/. Does this mean I need a "non kaby lake" processor to update the bios and only then the cpu works?

Storage, yeah its not much but without my "media demand" it should be more enough hoping that in a few months/maybe 1 year the prices for 8TB drives go down and I will add a 2x8TB mirror.

Thank you!

 

[LimeCrusher]

MOTHERBOARD: Supermicro-X11SSL-CF (Amazon)

That board can accommodate 14 drives. Do you really see yourself adding more than the 8 drives the Fractal Design Define R5 can accommodate? If not, you may want to consider a cheaper option like the X11SSM-F or her "crippled-M.2" sister the X11SSH-F.

BOOT: Samsung MZ-V7E500BW SSD 970 EVO 500GB NVMe (Amazon)

@KevDog is right. 500GB sounds overoverkill.
Plus, it's a M.2 SSD. The X11SSL-CF does not possess a M.2 port.

I'm a bit unsure about the CPU, if it makes sense to buy a cheap Pentium G4400 and buy the Xeon in a few months hoping the prices go down or do think the CPU prices will be kinda stable? As i guess that the first few months probably I won't use heavily VM's but will be more busy to setup everything.

Yes, a Pentium kinda makes sense to start messing with FreeNAS. You may realize that this CPU will be fine for longer than you expected. Some people here have a relatively heavy family use of Plex with Pentiums.
I would personally go for a G4560 (make sure the BIOS is updated) because of price and availability in my region.

HD: 2x Western Digital WD Red HDD 6000 GB, SATA III, 6000 Mb/s, 5400 RPM, 64 MB (Amazon)

Storage, yeah its not much but without my "media demand" it should be more enough hoping that in a few months/maybe 1 year the prices for 8TB drives go down and I will add a 2x8TB mirror.

There are different strategies for storage. If you're about to buy 2x6TB disks, you can probably rather go for 6 x 2 or 3TB disks and set up a 6 disks RaidZ2 vdev straight up. That's approximately 8 or 12TB of storage. Really up to you to determine what your needs are.

 

[KevDog]

@Platy

That type of fan splitter should work -- it just depends how many fans you want to place within the case and where you want to position them.

In terms of the motherboard/CPU incompatibility, I can't really comment. I have a Xeon and have never had any problem.

"Experimenting" with FreeNAS in my opinion is kind of pricey. For me when I experiment with things, I usually buy cheaper hardware. If I don't like it -- then great -- I've protected myself against the downside of spending too much money. On the flipside however -- if I like the system -- I usually discover that the system is greatly underpowered and now I have to spend more money to replace components I've already bought which ends up costing even more money. Where you fall on this risk profile I'm not sure. I wanted my FreeNAS first and foremost to be a backup solution to the other devices in my house. Because of this, use of a RaidZ2 setup for me was imperative since I didn't want to loose any data. If your needs are different, such as wanting the device to be a Plex/Kodi media server first, then maybe a RaidZ2 setup isn't as important to you.

One last thing -- a 500W power supply is kind of on the low side. As you've described your setup above it seems right not it will probably be enough. If your adding drives to the system in the future and other components you may need to come up with a rough wattage calculation of how much you'll need at peak load.

 

[Bozon]

I have the Mobo and it works great -- haven't had any problems. Needed some fan splitters however since the MOBO only has FAN1-4 and FANA.
In terms of boot drive - honestly you don't need much space. I'm personally booting of a 64 Gb SATA-DOM drive which is mirrored to a second SATA-DOM. When I was building my system, others told me that 64Gb was overkill. I thought I new better. I just looked --- I'm using less than 1 Gb of space on the boot drive. Oh well. I know many use thumb drives. Others a small hard drive. I'm using SATA-DOM modules since the mobo has the headers for 2 drives
Storage space - So basically 2 - 6Tb drives but with only 6Tb of usable space? I'd probably add more storage. VM's start to take up a lot of space. It just depends on your needs.

How hard was it to flash the SAS Controller on the board? I have skimmed the notes on it, but I wasn't sure from the article if it was one of those easy to explain, but something always goes wrong processes; or actually something that is easy to explain, and always goes well. I can't actually imagine it being the latter because of my strict belief in Murphy's law.

 

[KevDog]

I was really confused at first how to do this since its not very well documented. I actually contacted Supermicro by email and they sent me the files however they needed to be installed on a bootable DOS USB stick. I usually use a Mac which compounds the difficulty but I basically had Windows 10 installed in a VM and made the stick with rufus. You could do this directly from a Windows computer which would have been a lot easier. On the drive I copied the files -- which in my case sas3flh.exe. The SAS address is on the Mobo or box -- I don't remember. I had a hard time locating it. The actual flash process was very uneventful. The creation of the bootable USB drive, the download of the SAS flash utilities and finding the Controller SAS Address were the difficult processes

 

[Bozon]

I was really confused at first how to do this since its not very well documented. I actually contacted Supermicro by email and they sent me the files however they needed to be installed on a bootable DOS USB stick. I usually use a Mac which compounds the difficulty but I basically had Windows 10 installed in a VM and made the stick with rufus. You could do this directly from a Windows computer which would have been a lot easier. On the drive I copied the files -- which in my case sas3flh.exe. The SAS address is on the Mobo or box -- I don't remember. I had a hard time locating it. The actual flash process was very uneventful. The creation of the bootable USB drive, the download of the SAS flash utilities and finding the Controller SAS Address were the difficult processes

Thanks, for the info, so it sounds like Murphy's law is in full affect here.

Ugh, I only run OS X, or FreeNAS at home. I would have Linux running at home, but I don't have really any time to play with it. I don't even have a windows VM, so I may have to figure something out there.

[QUOTE}The SAS address is on the Mobo or box -- I don't remember.[/QUOTE]

The document that I read mentioned being able to pull the address out of the BIOS screens, if my memory isn't playing tricks on me.

Thanks.

 

[Ericloewe]

CPU COOLER: Noctua NH-U14S (Amazon)

That's massive overkill. The NH-D9L with extra fan in my workstation can keep my Xeon E5-1650 v3, whose TDP is around twice that of a Xeon E3, pretty cool while still being pretty damn quiet.

 

[Platy]

That board can accommodate 14 drives. Do you really see yourself adding more than the 8 drives the Fractal Design Define R5 can accommodate? If not, you may want to consider a cheaper option like the X11SSM-F or her "crippled-M.2" sister the X11SSH-F.

Strangely all the onlineshops (including shipping cost) I have access to sell the X11SSL-CF cheaper than the SSM-F or SSH-F, probably some strange dynamics about the inventory, otherwise I would go for the SSM-F.

@KevDog is right. 500GB sounds overoverkill.
Plus, it's a M.2 SSD. The X11SSL-CF does not possess a M.2 port.

Thank you! I mixed it up with the X11SSL-nF. Will buy a cheap Samsung MZ-76E250B 860 EVO, 250 GB, 2.5" SATA III.

Yes, a Pentium kinda makes sense to start messing with FreeNAS. You may realize that this CPU will be fine for longer than you expected. Some people here have a relatively heavy family use of Plex with Pentiums.
I would personally go for a G4560 (make sure the BIOS is updated) because of price and availability in my region.

You are probably right, but will probably still go for the Xeon (for fun and because if I take in consideration that i have to change again the cpu and will probably need the power in a few months and at the moment I have no setup where I can reuse the G4560..)

There are different strategies for storage. If you're about to buy 2x6TB disks, you can probably rather go for 6 x 2 or 3TB disks and set up a 6 disks RaidZ2 vdev straight up. That's approximately 8 or 12TB of storage. Really up to you to determine what your needs are.

I agree that a raidz2 setup would be "recommended", but for now i like the "advantages" of a mirrored dev setup. [1] I found a lot of stores that sell the 4TB drives a lot cheaper (~111EURO) per TB vs 6TB drives, which made me change my idea (about the size not mirrored dev).

That's massive overkill. The NH-D9L with extra fan in my workstation can keep my Xeon E5-1650 v3, whose TDP is around twice that of a Xeon E3, pretty cool while still being pretty damn quiet.

The price difference between the NH-D9L and the NH-U14S is only a few euros, so if there is no disadvantage I think it makes still sense to go for the "big" one ?(looking at some reviews the NH-U14S performances as expected a little bit better)

@KevDog

One last thing -- a 500W power supply is kind of on the low side. As you've described your setup above it seems right not it will probably be enough. If your adding drives to the system in the future and other components you may need to come up with a rough wattage calculation of how much you'll need at peak load.

[2]

2) For an E3-1230v3 (32-98W board+CPU, 12W memory):
7-8 Drives: 430W peak, 148W idle -> SeaSonic G-550

I guess it would be better to have some "space" and take a 700W one.

[1] http://jrs-s.net/2015/02/06/zfs-you-should-use-mirror-vdevs-not-raidz/
[2] https://forums.freenas.org/index.php?threads/proper-power-supply-sizing-guidance.38811/


Build update - 15 Jan 2019

• MOTHERBOARD: Supermicro-X11SSL-CF (Amazon)
• CPU: Xeon E3-1230 v6 (Amazon)
• CPU COOLER: Noctua NH-U14S (Amazon)
• RAM: Kingston Technology ValueRAM 16GB DDR4 2400MHZ ECC Module 16GB DDR4 2400MHz (Amazon)
• BOOT: Samsung MZ-76E250B 860 EVO, 250 GB, 2.5" SATA III (Amazon) + caddy
• HD: 4x Western Digital WD Red HDD 4000 GB, SATA III (2x2 mirror vdevs)
• CASE: Fractal Design FD-CA-DEF-R5-BK Define R5 Case (Amazon)
• PSU: Be quiet! Pure Power 11 700W CM(Amazon)

I hope I didn't forgot to reply to somebody.

Thank you all again for your great help, it was great idea to post here you help me to avoid expensive mistakes :)

[edit]fixed formation and modified the HD section[/edit]

 

[Ericloewe]

The price difference between the NH-D9L and the NH-U14S is only a few euros, so if there is no disadvantage I think it makes still sense to go for the "big" one ?(looking at some reviews the NH-U14S performances as expected a little bit better)

It's also much heavier, which means a higher risk of damaging your motherboard when moving the server. And I guess the compact size of the D9L is an advantage in tighter chassis.

 

[Platy]

It's also much heavier, which means a higher risk of damaging your motherboard when moving the server. And I guess the compact size of the D9L is an advantage in tighter chassis.

Thank you, this makes definitely sense, will buy the D9L.

 

[LimeCrusher]

Strangely all the onlineshops (including shipping cost) I have access to sell the X11SSL-CF cheaper than the SSM-F or SSH-F, probably some strange dynamics about the inventory, otherwise I would go for the SSM-F.

200€ is pretty good indeed. I personally check price and availability in that online shop and it is more expensive. But the X11SSM-F or X11SSH-F are cheaper though.

You are probably right, but will probably still go for the Xeon (for fun and because if I take in consideration that i have to change again the cpu and will probably need the power in a few months and at the moment I have no setup where I can reuse the G4560..)

You can always resell. just hate to buy stuff that's overkill or that I don't fully need so I am rather conservative in terms of hardware. But if it's for fun, there's nothing I can do :D

Good luck with building and have fun!

 

[Platy]

Hi,

the system is alive :)



A few personal comments that could be useful for people that want to go for a similar build.

-The installed bios version was 2.1a, so no problem the xeon processor being recognized (If not you need to update your bios by first booting a "default" supported cpu or buy a IPMI license to upgrade through the IPMI webinterface). In any case IPMI is awesome, I dont know how I survived till today, for sure it will be a mandatory requirement for further motherboard buys.

-Noctua NH-D9L: I was messing around 30 minutes mounting the fan and trying to "move it lower" and going crazy until I found out (as you can see in the picture) that its a manufacturing error, its not possible to install it as they claim (probably the performance is the same and its only an esthetic issue, however in any case unacceptable). Watch this review video: https://www.youtube.com/watch?v=Q6dmU6PJqzA&t=884s (around minute 10-15). In any case the cooling power is great.

-Fan "issues": The noctua cpu fan (the only one with PWM) kept spinning all the time up and down. Quick answer:

set the correct fan tresholds using the IPMI tool:

I'm using the default fractal installed Fractal Design Dynamic GP14 case fans (2x) they have no PWM, attached to FAN3, FAN4)

I attached the Noctua fan to "FANA", in this case you need to set the fan mode to "standard" if you attach it to another FANX you should go with "optimal" mode.

The noctua cpu fan (noctua nf-a9 pwm) has max rpm 2200 (some say 2000) and min rpm 300.

Code:

ipmitool sensor thresh "FANA" upper 1900 2000 2200
ipmitool sensor thresh "FANA" lower 100 200 300

Set the mode to standard

Code:

ipmitool raw 0x30 0x45 0x1 0x0

Result (running badblocks on the 2 harddrives since 12 hours and using a CPU stress tool for a few hours):

Code:

Sun Jan 27 14:44:24 CET 2019
CPU Temp,62,degrees C,ok
PCH Temp,45,degrees C,ok
System Temp,38,degrees C,ok
Peripheral Temp,35,degrees C,ok
VcpuVRM Temp,52,degrees C,ok
DIMMA1 Temp,,,ns
DIMMA2 Temp,,,ns
DIMMB1 Temp,,,ns
DIMMB2 Temp,27,degrees C,ok
FAN1,,,ns
FAN2,,,ns
FAN3,1700,RPM,ok
FAN4,1700,RPM,unc
FANA,1000,RPM,ok
ada0  Current Temperature:                    26 Celsius
ada1  Current Temperature:                    27 Celsius
ada2  Current Temperature:                    27 Celsius

The CPU temperature never goes above 62 celsius and the disks are around 26-28, looks good to me :).

I got this information and more from this incredible good threads:

https://forums.freenas.org/index.ph...nge-ipmi-sensor-thresholds-using-ipmitool.35/
https://forums.freenas.org/index.php?threads/thermal-and-accoustical-design-validation.28364/
https://forums.freenas.org/index.php?resources/fan-scripts-for-supermicro-boards-using-pid-logic.24/

 

[LimeCrusher]

Thanks for your feedback.

I'm using the default fractal installed Fractal Design Dynamic GP14 case fans (2x) they have no PWM, attached to FAN3, FAN4)

I attached the Noctua fan to "FANA", in this case you need to set the fan mode to "standard" if you attach it to another FANX you should go with "optimal" mode.

My Supermicro X9SCM-F manual specifies that FAN1-4 should be used for CPU cooling while FANA should be used for peripherals. This is what @GrumpyBear recommends in his guide. What does your manual says? What do you get with those non-PWM case fans then? Are they spinning full speed all the time?

The CPU temperature never goes above 62 celsius and the disks are around 26-28, looks good to me :).

Those disks temperatures are really good indeed.

 

[Yorick]

Not sure whether this has already been purchased. My comment would be that an X11SSH-F with a Xeon E3-1225 v6 may be a better choice than a X11SSL-CF with a Xeon E3-1230 v6.

Rationale: C236 + 1225 v6 means hw transcode support for Plex "in the near future". Might as well, there's nothing to lose.
Rationale Numero Duo: M.2 PCIe x 2 means a SATA port can remain free and you can still boot from SSD rather than USB, increasing reliability. Corsair has a PCIe x 2 M.2 SSD that would fit well here.

 

[Platy]

Thanks for your feedback.

My Supermicro X9SCM-F manual specifies that FAN1-4 should be used for CPU cooling while FANA should be used for peripherals. This is what @GrumpyBear recommends in his guide. What does your manual says? What do you get with those non-PWM case fans then? Are they spinning full speed all the time?


Those disks temperatures are really good indeed.

Hi!

actually there should be no difference if you attach it to FANA or FAN1-4, as GrumpyBear states:

So we see that in "Optimal" mode FAN-1 through -4 increase as the CPU temperature increases under load and FAN-A remains at a low speed.

In Standard mode all the fans including FAN-A now are controlled by the CPU temperature.

so it is important to choose the right "fanmode", as far as i remember the default setting is optimal probably for that reason supermicro suggests to attach the cpu fan to FAN1-4.

Yes the case fans are non-PWM and run constantly at 1700rpm, I'm going to replace them soon with 3 PWM fans, not yet decided if Noctua NF-A15 PWM or Phanteks PH-F140MP or Be-Quiet Silentwings 3. The Silentwings should be the most silent, but I think the air intake is worse than the noctua, have to read more reviews.

@Yorick

yes already bought everything :) The X11SSL-CF was at the time I bought it much cheaper, I got it for 180EUR and the X11SSL-CF was 220-235 EUR. Did not know about the possible plex hw transcode support, but I have never more than 2 transcoded streams at the same time so the cpu should handle it easily. thanks for your comments!

 

[LimeCrusher]

actually there should be no difference if you attach it to FANA or FAN1-4, as GrumpyBear states:

so it is important to choose the right "fanmode", as far as i remember the default setting is optimal probably for that reason supermicro suggests to attach the cpu fan to FAN1-4.

My bad. I had read that guide a month ago, forgot the particulars and just checked the conclusion as I read your post. Interesting to know this is possible.

Yes the case fans are non-PWM and run constantly at 1700rpm, I'm going to replace them soon with 3 PWM fans, not yet decided if Noctua NF-A15 PWM or Phanteks PH-F140MP or Be-Quiet Silentwings 3. The Silentwings should be the most silent, but I think the air intake is worse than the noctua, have to read more reviews.

Which might be why you get so low HDD temperatures. Is the whole thing noisy?
I would be interested if you have good fan reviews and comparison to share. Those thing are a mystery to me.

 

[KevDog]

My bad. I had read that guide a month ago, forgot the particulars and just checked the conclusion as I read your post. Interesting to know this is possible.


Which might be why you get so low HDD temperatures. Is the whole thing noisy?
I would be interested if you have good fan reviews and comparison to share. Those thing are a mystery to me.

What case do you have and how many fans? I've got 6 total fans in my Fractal Node case -- all PWM - 4x 120mm and 2 x 140 mm.
(Enermax PWM Series TB Silence 120 mm Ultra Quiet Twister Bearing Cooling Case Fan and Noctura SSO Bearing Fan NF-P14s redux). When fans are at full speed the system is actually pretty loud as you would expect -- I don't have a dB meter so unfortunately I can't actually equate what lout means. I currently however trying to configure the system with some fan control logic as discussed in this post: https://forums.freenas.org/index.php?threads/pid-fan-controller-perl-script.50908/. There are 3 different fan controller scripts floating around on the forums -- Stux, Kevin Horton and Glorious1. Glorious1 script seems to be the most involved. I'm currently testing Kevin Horton's script -- he's pretty good about providing support.

 

[ullbeking]

C236 + 1225 v6 means hw transcode support for Plex "in the near future".

By "hw transcode" are you talking about offloading the computation to the GPU? Yes, this can work, but if you are in my situation, FYI, then I'm buildling a media server from a NAS and using entirely CPU computation. The decoded stream then gets sent to an RPi or some other AV interface that can get the signal out of the network and into the display; on the other hand, perhaps there's a way to get the decoded signal directly out of the network into the display, depending on its capabilities. I know a lot of you might not agree, but I'm just sharing my experience, and I have reasons for them.