Newbie FreeNAS Build

[thebill]

Hey all, this will be my first FreeNAS build and first foray into ZFS (though not my first system build by a long shot and not my first rodeo with RAID based systems). I'd love some input on hardware selection and am grateful for any advice that may be given.

The quick hardware guide was followed pretty closely for this build.

Design Intent: This NAS is intended to serve two functions: file serving to both windows and linux users and transcoding of up to 2 simultaneous 1080p video streams via DLNA. Initially the system is planned to use 4 disks in a RAIDZ2 configuration and then later on (12-18 months down the road), expand the pool to up to 8 disks in total once the new RAIDZ expansion feature migrates into a stable build of FreeNAS. 4 disks in a RAIDZ2 configuration will provide ample storage to meet the needs of the next year or so.

Not everything will be bought from NewEgg, it's just what was conveniently up when typing this out for linking to particular part numbers.

With that, here is the hardware list:

CPU: Intel Core i3-8300
Motherboard: SUPERMICRO MBD-X11SCH-LN4F-O
Memory: 2x Samsung DDR4-2666 32GB/2Gx8 ECC (64 GB total)
Boot Disk: Vaseky M.2 2280 SATA 60GB
Storage Disks: 4x WD Red 8TB
Case: Fractal Design R5
Power Supply: CORSAIR TX-M Series TX550M
Cooling: Cooler Master i30 CPU Cooler

If you see any errors, incompatibilities, or places where either money could be saved or performance increased for equal cost, please let me know!

Cheers and thanks in advance for any advice!

 

[PhiloEpisteme]

Motherboard: SUPERMICRO MBD-X11SCH-LN4F-O

Do you need the 4 1G connections?

Boot Disk: Vaseky M.2 2280 SATA 60GB

IMO this is a waste of that M.2 slot. If I were you I'd boot from SATADOM and save those M.2 slots for SLOG devices that actually make use of the speed of that port. If you don't plan on using SLOGs or you're not sure you'd like to use an M.2 SLOG you may want to look at a board without them.

Overall the build looks good. The CPU has a passmark of 8000 or so and the 1080p streams look to want a 4000 minimum so you're likely okay there.

 

[thebill]

Do you need the 4 1G connections?


IMO this is a waste of that M.2 slot. If I were you I'd boot from SATADOM and save those M.2 slots for SLOG devices that actually make use of the speed of that port. If you don't plan on using SLOGs or you're not sure you'd like to use an M.2 SLOG you may want to look at a board without them.

I need 2 1G connections, the 4 is slight overkill for my application but the board otherwise looked good and met the specs I was looking for. If you have any recommendations on an alternate board ATX (or smaller) with 2 1G connections I'd be happy to take a look at it.

Good thought on using the m.2 for SLOG - I may end up doing exactly what you recommend and move the boot disk to a SATADOM and use that or a slightly larger m.2 drive for SLOG.

 

[PhiloEpisteme]

The SLOG devices need to have PLP so keep that in mind. Also, if you decide to mirror the SLOG you'll want to be sure that the two ports are long enough to accommodate the M.2 devices with PLP, many of those are longer than the standard consumer M.2 drive.

As for specific boards, what are your memory requirements? Do you need one that supports more than 64GB?

 

[thebill]

As for specific boards, what are your memory requirements? Do you need one that supports more than 64GB?

Nope, 64 GB is sufficient. There is a slim chance in 2-3 years that that may change but it’s fairly unlikely. I liked that that motherboard leaves open the option of adding more but it’s not a hard requirement.

BTW, thanks for the advice so far. It’s much appreciated!

 

[thebill]

The SLOG devices need to have PLP so keep that in mind. Also, if you decide to mirror the SLOG you'll want to be sure that the two ports are long enough to accommodate the M.2 devices with PLP, many of those are longer than the standard consumer M.2 drive.

Two other quick questions:

1) This NAS will be running on a UPS with sufficient wattage to power it for around 45 min in the event of a power failure (more than long enough for a graceful shutdown). Is PLP really necessary on an SLOG drive in this use case?

2) The memory I spec'd is rated DDR4-2666 but the processor only supports up to DDR4-2400. I presume this isn't an issue as the processor will just clock the ram at a bit slower speed than what its designed to - is that correct?

 

[PhiloEpisteme]

1) This NAS will be running on a UPS with sufficient wattage to power it for around 45 min in the event of a power failure (more than long enough for a graceful shutdown). Is PLP really necessary on an SLOG drive in this use case?

I would argue that you should always get a device with PLP because otherwise you're half defeating the point of the SLOG which is to provide permanent storage functionality for sync writes to guarantee integrity of sync writes. You could consider just turning sync writes off rather than use a SLOG without PLP. You'll get better performance. I'm not 100% sure exactly how much more risk you put your data in if you do this (data in ram vs data in transition in your SLOG.)

2) The memory I spec'd is rated DDR4-2666 but the processor only supports up to DDR4-2400. I presume this isn't an issue as the processor will just clock the ram at a bit slower speed than what its designed to - is that correct?

That is likely fine; so long as either the board or the memory guarantees compatibility you should be fine.

 

[anmnz]

You could consider just turning sync writes off rather than use a SLOG without PLP. You'll get better performance. I'm not 100% sure exactly how much more risk you put your data in if you do this (data in ram vs data in transition in your SLOG.)

It's exactly the same. If you lose power and you have either sync disabled or a SLOG that doesn't preserve data (no PLP), then you will still have the data that was in the last transaction group that was successfully flushed to disk (which typically happens every 5 seconds or so), but nothing that was written since then. If you have sync enabled and a working ZIL (in the pool or as a SLOG) then you will additionally have the data that was in any sync writes that completed before power loss.

 

[PhiloEpisteme]

If you lose power and you have either sync disabled or a SLOG that doesn't preserve data (no PLP), then you will still have the data that was in the last transaction group that was successfully flushed to disk (which typically happens every 5 seconds or so), but nothing that was written since then.

I guess I don't follow 100%. My mental model is as follows; feel free to correct me. If you have sync disabled the system reports to the client that it has the data as soon as it is in RAM; if the power is lost from then and until the ~5s until it gets flushed to disk you lose that data. With a non-PLP slog it is a bit different. The client is told that the system has the data when it goes from ram into the SLOG but even once in the SLOG the data takes a bit of time from when the SLOG first gets it and when it is written to permanent storage; this slight bit of time is the risk and it is less than 5s. Thinking of it like this; the SLOG without PLP still presents risk, but the time that data is in that risky state is less than with sync writes off entirely, no?

 

[anmnz]

the data takes a bit of time from when the SLOG first gets it and when it is written to permanent storage; this slight bit of time is the risk and it is less than 5s.

But only trivially less; it is that 5s (give or take) minus only however long it took to actually write the data to the SLOG. If your SLOG is fast (surely the case we are discussing here) that write will take very little time j deed.

I apologise if I'm misreading but -- are you thinking that sync writes go "via" the SLOG to the pool? They don't. The ZIL is write-only in normal operation. Sync and async writes follow exactly the same path to permanent storage; it's just that sync writes have this extra step of writing data to the ZIL before the client is notified that the write is complete.

 

[PhiloEpisteme]

I apologise if I'm misreading but -- are you thinking that sync writes go "via" the SLOG to the pool? They don't. The ZIL is write-only in normal operation. Sync and async writes follow exactly the same path to permanent storage; it's just that sync writes have this extra step of writing data to the ZIL before the client is notified that the write is complete.

No worries. I was not saying that sync writes go via the SLOG but if I was unclear it is important to make it clear, thank you for that.

But only trivially less; it is that 5s (give or take) minus only however long it took to actually write the data to the SLOG. If your SLOG is fast (surely the case we are discussing here) that write will take very little time j deed.

Yup, this is the time I'm not sure about. With sync=off you're risking 5s of data every time, yes? What I don't know is how long does it typically take between when the system reports to the client that it has the data and when the SLOG has actually put it all the way into its permanent store. I imagine it is much less then 5s.

Anyway, we are off track here. Sorry for distracting from the main point; I just find it interesting. :)

 

[anmnz]

Ah right I get where you are coming from now. I think there is also the problem of partial writes: if my database/filesystem/etc. makes a sync write because it really really wants to know that chunk of data is intact and permanently stored, and the write is acked back to it, but then half that chunk of data disappears due to a power failure... that could be nasty.

Agreed this is off topic though. :)

 

[thebill]

That is likely fine; so long as either the board or the memory guarantees compatibility you should be fine.

That memory was chosen because it was one of the tested sticks listed on SuperMicro's site for that particular motherboard model. They didn't specify what processor it was tested with but I presume they're guaranteeing comparability if they're listing it on their site.

Thanks to everyone for the advice!

 

[1kokies]

Design Intent: This NAS is intended to serve two functions: file serving to both windows and linux users and transcoding of up to 2 simultaneous 1080p video streams via DLNA.

just want to relate my experience on trans coding. i have about 15TiB of movies and used Freenas to install Plex as a server to watch movies in the household. Trans coding takes lots of CPU resources (on top of file transfers, scrubs, rsync etc) and since then i have moved the Plex server out of Freenas server.

The SLOG need to take advice from both Guru above

 

[thebill]

just want to relate my experience on trans coding. i have about 15TiB of movies and used Freenas to install Plex as a server to watch movies in the household. Trans coding takes lots of CPU resources (on top of file transfers, scrubs, rsync etc) and since then i have moved the Plex server out of Freenas server.

The SLOG need to take advice from both Guru above

I appreciate it. What CPU are/were you running and how many simultaneous streams were you transcoding?

We have a couple of servers with plenty of CPU rescources to spare if the NAS can’t handle it but it’d be nice to be able to keep it all in the same box (which is the intent).

 

[1kokies]

I appreciate it. What CPU are/were you running and how many simultaneous streams were you transcoding?

my lab box was an AMD Phenom II 965 and then move on to a XEON E2609V3. Colloquially the CPU ran close to 100%, i was just having 1 stream, to my hall connected with a TV Box via cable of course, using Plex. To qualify further the movies are blu ray size means it's 10-15GB per movie in mkv (hence the stream speed is high & not 5mbps) the more the higher the trans code needed. That's when i decided to move it out. Movies that are 2-5GB will have no problems however if you watch them on a sub 60" screen there will be a difference. These are 2k movies ya, 4k is off subject. For those movies that i rip myself from blu ray disks to mkv have bigger files, the TV Box sometimes lag and will display "server not powerful enough.......". i have 1500 titles to date, so sharing ya.

We have a couple of servers with plenty of CPU rescources to spare if the NAS can’t handle it but it’d be nice to be able to keep it all in the same box (which is the intent).

in my experience it was ideal to have 1 box but when i put it in production i can see the Freenas hardware struggling. And they are serving as database so i did not want to risk it. May be practical to VM or physical box for Plex server only. If you use other than TV Box as client (a spare PC perhaps, the trans coding could be lesser).

Last word, i did this for years, i started as i had too many blu ray disks physically and had to spend months ripping it to Freenas. At the end of the day i have never been able to watch them all for the 2nd time, i had to always grow my storage, always worry about disk failures, need to have more raid cards, higher electricity costs and worst is try to find the case that can fit so many disks. Your choice of using 8TB is a wise one, lesser hdd=lesser problems. 4x8TB at raiz2 (assume only not suggest, since you are also keeping data) gives about 13 TiB so it depends ya.

It was exciting when i started, now i would rather archive to single disks or Storage Spaces in a HTPC or delete the movies.

 

[thebill]

IMO this is a waste of that M.2 slot. If I were you I'd boot from SATADOM and save those M.2 slots for SLOG devices that actually make use of the speed of that port. If you don't plan on using SLOGs or you're not sure you'd like to use an M.2 SLOG you may want to look at a board without them.

Philo: I've been considering a SLOG more over the last 24 hours for the M.2 port and moving the boot disk over to a mirrored pair of used Intel SSD's - do you have any recommendations on a PCI-E M.2 disk with PLP? I'm having no luck finding anything that can really take advantage of the PCI-E speeds on the M.2 port on that motherboard that has PLP.

I'd like to keep the two hardware PCI-E slots open for future HBA and/or 10G networking expansions, if necessary.

 

[PhiloEpisteme]

Philo: I've been considering a SLOG more over the last 24 hours for the M.2 port and moving the boot disk over to a mirrored pair of used Intel SSD's - do you have any recommendations on a PCI-E M.2 disk with PLP? I'm having no luck finding anything that can really take advantage of the PCI-E speeds on the M.2 port on that motherboard that has PLP.

I do; but it isn't cheap. I'm looking at the 200GB Intel Optane SSD DC P4801X. They are quite pricy. The 200G has a better write speed than the 100G even those both are larger than a SLOG needs to be. Also keep in mind that often folks choose to mirror them. My long-term goal is to pick up two of those devices and mirror them as a SLOG. There are cheaper SATA options via the Samsung 883 DCT as well. Whatever you choose definitely compare the write speeds across the different sizes; sometimes the size of the device affects its write speed.