High-performing NAS/Media Distribution Server

[CK224]

Hey all,

I am trying to build a high-performing local NAS/Media-Distribution server and thought that FreeNAS looks like a good fit. Since I am new to this, I went through the Forum and the Hardware Guide and tried to collect as much information and designed the Server accordingly.

I hope the hardware selection makes sense and I haven’t overlooked anything essential. Any comment / input is highly appreciated - thanks in advance!

Aim/Requirements/Outline:

• Media-Server for content distribution over Ethernet & WiFi(up to 10x 4K streams + up to 20x HD streams & connected to a 10GB-Ethernet Network)
• Small form factor (as small as possible)

Chassis:

• Supermicro SuperChassis E300:
• https://www.amazon.com/Supermicro-C...=1542476425&sr=8-1&keywords=SuperChassis+E300

Motherboard

• Supermicro X10SDV-4C+-TP4F:
• https://www.amazon.com/Supermicro-Socket-Motherboard-X10SDV-4C-TP4F/dp/B01FYD9TSG/ref=sr_1_1?ie=UTF8&qid=1542476700&sr=8-1&keywords=X10SDV-4C+-TP4F

CPU

• Intel Xeon D-1518 Processor
• (refer to Motherboard)

RAM

• 32GB (4x 8GB):
• Samsung DDR4-2133 8GB/1Gx72 ECC/REG CL15 Server Memory (M393A1G40DB0-CPB)
• https://www.amazon.com/Samsung-DDR4-2133-Server-Memory-M393A1G40DB0-CPB/dp/B00NTBF8KE/ref=sr_1_1?ie=UTF8&qid=1542476821&sr=8-1&keywords=DDR4-2133+8GB/1Gx72+ECC/REG+CL15+Server+Memory

Power Supply

• MCP-250-10122-0N
• https://www.newegg.com/Product/Prod...e=MCP-250-10122-0N-_-9SIA5EM59D6745-_-Product

Additional SATA/SAS connectivity

• N/A

Boot Devices

• 1x Samsung 970 EVO NVMe M.2 SSD 250GB (MZ-V7E250BW)
• https://www.amazon.com/Samsung-970-EVO-500GB-MZ-V7E500BW/dp/B07BN5FJZQ/ref=sr_1_3?ie=UTF8&qid=1542477305&sr=8-3&keywords=970+EVO+NVMe+M.2+SSD&th=1

Networking

• 2x 10GBE (refer to Motherboard)

Storage

• 2x Samsung 860 EVO 4TB 2.5 Inch SATA III Internal SSD (MZ-76E4T0B/AM)
• https://www.amazon.com/dp/B07864XY8B/ref=twister_B079P94LLX?_encoding=UTF8&th=1

• Since space in the chassis is limited and I want to use both SSDs as data-storage, is it possible to not use any Raid/ZFS set-up (the data can be restored from another server in case the SSDs fail)?

Thanks again for your feedback.

 

[Chris Moore]

10 x 4k streams and 20 HD streams. I think that might be a little ambitious for a low-power system like this. What software for the streaming? Plex?

Sent from my SAMSUNG-SGH-I537 using Tapatalk

 

[kdragon75]

Nope.

 

[JohnK]

Nope.

I like a short and sweet response. .

I don’t have any of the proposed hardware and can’t comment on experience, but if the usecase is just reading and no transcoding, I would stick in 16Gb ram modules x 4 and it might work as the processor shouldn’t work too hard. But with that said, you do lose all the things that makes FreeNas good.

 

[Chris Moore]

Media-Server for content distribution over Ethernet & WiFi(up to 10x 4K streams + up to 20x HD streams & connected to a 10GB-Ethernet Network)

Why have you not responded in any way?

2x Samsung 860 EVO 4TB 2.5 Inch SATA III Internal SSD (MZ-76E4T0B/AM)

The number of vdevs is more of an issue with your system than anything. Even with SSDs, I would say that it would be better to have eight 1TB drives than to have two 4TB drives. More vdevs gives more IOPS which is going to make a difference when you are trying to serve 30 clients at once.
The issue of transcoding video is another question and you really need to get more detailed about how you are thinking to use this system before that can be addressed.

 

[CK224]

Thanks for the feedbacks, really appreciated!

10 x 4k streams and 20 HD streams. I think that might be a little ambitious for a low-power system like this. What software for the streaming? Plex?

I haven't decided on streaming software yet, depends more on the performance to make it "possible", but Plex is an option. Do you have a higher-powered system in mind or you think it's just not feasible with the form-factor?

I like a short and sweet response. .
I don’t have any of the proposed hardware and can’t comment on experience, but if the usecase is just reading and no transcoding, I would stick in 16Gb ram modules x 4 and it might work as the processor shouldn’t work too hard. But with that said, you do lose all the things that makes FreeNas good.

Upgrading the RAM should not be a problem, so thanks for this input. Sorry for the lack of knowledge, but why do you lose the good things of FreeNAS if you use this configuration?

Why have you not responded in any way?
The number of vdevs is more of an issue with your system than anything. Even with SSDs, I would say that it would be better to have eight 1TB drives than to have two 4TB drives. More vdevs gives more IOPS which is going to make a difference when you are trying to serve 30 clients at once.
The issue of transcoding video is another question and you really need to get more detailed about how you are thinking to use this system before that can be addressed.

Noted, thanks. The problem with having more SSDs is that they don't fit into the chassis I would like to have. I based the 2x 4TB SSDs on their reading speeds (550 MB/s) which I thought should be enough for the streams (1x 4k I estimated 25MB/s and the HDs with 6.25 MB/s). But I will have a deeper look into IOPS and vdev.
The system set-up was based on the assumption of no-transcoding - I assumed already that if you put transcoding on top of the "just streaming" that it would be hard to realize. Therefore the content would be available in 4K and HD.
What other further inputs you would need?

Hope this clarifies a bit - thanks again so far!

 

[kdragon75]

So price does not matter, only size? It doesn't matter how much money you throw at it, you can only get so small before you just hit a wall... Also if you don't care about data integrity, uptime, or anything other than performance, you can use ZFS to stipe the disks together.

it's just not feasible with the form-factor?

Bingo.

Upgrading the RAM should not be a problem, so thanks for this input. Sorry for the lack of knowledge, but why do you lose the good things of FreeNAS if you use this configuration?

Its not the RAM, its striping the disks. FreeNAS is all about keeping data intact and online no matter what. If you stipe the disks, you lose all of that.

The problem with having more SSDs is that they don't fit into the chassis I would like to have.

See my point above...

I based the 2x 4TB SSDs on their reading speeds (550 MB/s) which I thought should be enough for the streams

That's based on best case scenario an not real world performance

The system set-up was based on the assumption of no-transcoding

Unless your pre encoding all of the media and have well defined media players that's not going to work for long. As pointed out we don't know enough about your plans to say if this is evan sain.

I assumed already that if you put transcoding on top of the "just streaming" that it would be hard to realize. Therefore the content would be available in 4K and HD.

This makes no sense. Just because its stored and played in 4k does not mean you don't need to transcode.

What other further inputs you would need?

You rational for way you need the ABSOLUTE SMALLEST CASE EVER. Your just making it harder on yourself. Not to mention more expensive. But hey, some people shi.. excrete gold.

 

[CK224]

Hey kdragon75, thanks for your inputs!
All fair points and I try to rationalize my approach a bit in my answers below. But in general, the use-case is a an environment where space is limited and therefore I tried to start with the smallest design possible. So in the end it's to find the best balance between performance and size for the set-up. I know the space limitation sounds weird, but it is what it is (height is more critical than footprint - so staying slim and bigger would be ok-ish)

So price does not matter, only size? It doesn't matter how much money you throw at it, you can only get so small before you just hit a wall... Also if you don't care about data integrity, uptime, or anything other than performance, you can use ZFS to stipe the disks together.

In the end the price/cost is not the most important factor, as stated above, size is more critical. As you mention hitting the wall, with the current set-up what would you think is a reasonable assumption for streams (aka "hitting the wall")?
The next size up is basically having a 1U-Rack size, which is too big...

Its not the RAM, its striping the disks. FreeNAS is all about keeping data intact and online no matter what. If you stipe the disks, you lose all of that.

Understood - as said, not the main aim of the design and therefore a risk to take...

That's based on best case scenario an not real world performance

Any experience what real world performance would look like? 10% less?

Unless your pre encoding all of the media and have well defined media players that's not going to work for long. As pointed out we don't know enough about your plans to say if this is evan sain.
This makes no sense. Just because its stored and played in 4k does not mean you don't need to transcode.

Probably not very rational I give you that, but yes the content would be pre-encoded and the media-players used would need to support the content. In the end the aim is to mainly stream H.264 & H.265 encoded content. So overall the goal is to get this huge amount of streams out to 4K TVs and HD-Tablets.

You rational for way you need the ABSOLUTE SMALLEST CASE EVER. Your just making it harder on yourself. Not to mention more expensive. But hey, some people shi.. excrete gold.

I wish But sometimes you have to work with the limitations you have - that's why I try to use as a first step the "absolute smallest case ever"...
If not possible to get the hoped for performance an alternate could be:

• divide the 4K and the HD-content and create 2 servers and/or live with the amount of streams which come out of this server and duplicate it to double the streams
• or go a bit bigger in size, but not preferred

Cheers!

 

[Ender117]

For your form factor and IOPS, maybe m2 nvme ssd can do the job. If you are sure no transcode is required this sounds like the way to go.
If transcode comes into play I would install windows server and get a quadro p2000 or above to do GPU transcoding. ASIC is much faster than CPU but quality suffers

 

[Chris Moore]

First, lets not get in a rush and make decisions too quickly because it is potentially a good bit of money you are taking about spending.

divide the 4K and the HD-content and create 2 servers

This confuses me though. If you are so space constrained that you can't have a larger chassis, how is it you can have two? Just why is there a space constraint? The server can be located anywhere that a network connection can be provided, even if new cable needs to be run.

I haven't decided on streaming software yet,

This is an important decision. I think you need to understand a few things about video streaming, but that is a bit out-of-scope for our forum. Basically, even a 1080p file will be transcoded when it is streamed to a tablet because of the screen size. It is more about the number of pixels that need to be provided on the target device and the size of the screen makes a big difference.
Here is some reading on the matter of 4k...
https://www.cnet.com/news/why-ultra-hd-4k-tvs-are-still-stupid/

But sometimes you have to work with the limitations you have

It might help to understand what the limitations are and why they are limitations. Sometimes people impose artificial limitations on themselves where there is another option that they have not considered. There are two main reasons we are making these statements. First, it is very processor intensive to transcode. To transcode a 1080p stream (for example) the folks over at Plex suggest a CPU with a PassMark score of 2000 or better, so you would theoretically need a score of 16,000; multiplied by the number of 4k streams (if they are transcoding) but the amount of transcoding required varies depending on the source media and the destination player so that estimated number is based on the maximum. This is also assuming you are referring to H264 video; H265 takes a lot more power, so consider at least double the score needed if it is H265.
So, with that number (16000) in mind, we look at the PassMark scores of tested CPUs:
https://www.cpubenchmark.net/high_end_cpus.html

The top performing processor on the list (at the time of my post) is a Intel Xeon Platinum 8173M @ 2.00GHz with a PassMark - CPU Mark score of 28,860.
The thing that tells us is this, even the best CPU currently tested would (potentially) only support two 4k streams simultaneously.
It is important to realize what you are asking for before you go out and decide you can do it.
The organization I work for had (for several years) a large presentation system running in a museum-like environment where we ran about 18 screens that were each presenting a video on continuous loop.
This was before the day of 4k and we had a mix of 720p and 1080p videos but generally we found that the most a single computer could handle was eight screens. Computers are more powerful now, but 4k is a lot more demanding.
My suggestion would be that you only provide 1080p content. With 1080p as your only media type, you might actually be able to do this.

Second reason we are talking to you about it is the cost. It will be very expensive to buy enough CPU power to transcode that much 4k. We are trying to save you some money, but if you don't care about the money, it can be worked out.

 

[Chris Moore]

PS. It is worth note that the AMD Ryzen Threadripper 2950X has a PassMark score of 25,930 and the list price for that CPU (today) is around $900, where the only slightly better performing Xeon processor is almost $4000.

 

[kdragon75]

In the end the price/cost is not the most important factor, as stated above, size is more critical. As you mention hitting the wall, with the current set-up what would you think is a reasonable assumption for streams (aka "hitting the wall")?
The next size up is basically having a 1U-Rack size, which is too big...

Perhaps if we understood more about your design constraints we could come up with something. Is this going in a small yacht? Are you hiding it in your jobs datacenter?

Any experience what real world performance would look like? 10% less?

I would anticipate 30% less but that still depends on how full it is, how often content get replaced and how many people are watching the same thing at the same time.

I wish :) But sometimes you have to work with the limitations you have

We don't even know what the limitations are. though with such "limited" space cooling and power may also be limiting factors as well. Again we just don't know enough to help.

• or go a bit bigger in size, but not preferred

So this is not a HARD requirement? Your making this harder than it needs to be.

 

[CK224]

Thanks Chris and kdragon75, for your continuous support and inputs.

Based on your feedback, I realized maybe it makes sense to take a step back and divide the discussion into 2 parts:

• Technology required to achieve the streams (10x 4K and 20x HD-Streams)
• Space constraints

In general the set-up is aimed for a gaming gallery project in an open attic space, with no space to add racks. Therefore the idea was to put the server(s) above the ceiling panels between the structural beams (therefore the limited space). But looking at the transcoding requirements, even without any space restriction achieving the amount of streams is “challenging”, if I understand correct:

Technical:

• Up to 10x 4K TVs with Nvidia Shields
  Input: H.264 & H.265 4K files – therefore the idea was with the Shields that there is no 4K transcoding needed. (Yes, I have considered that the shields have local storage, but the requirement is to have all media available to all TVs. Hence the need for the central server).

• The HD streams, would stream to Tablets or other personnel devices.
  1. With the numbers from Plex (Transcoding a 1080p) file requires as mentioned a PassMark score of 2000. So with 20x Streams that would mean 40.000 – so no processor on the market today is able to achieve that, right?
  2. So the question would be how to achieve 20x streams, what would be the best approach?

• Avoiding the huge amount of transcoding? (the tablets selected would need to play 1080p and H.264/5 content)?
• Having more servers? (refer to space constraints)
• Pre-encoding the content further to 1080p and 720p?
• ?

--> As mentioned by Chris, I guess this is a bit out of scope for this forum, but any input is appreciated. But if required/preferred I can take it over to PLEX or elsewhere :)

Space Constraint:
As mentioned above the goal was to put the server above ceiling panels:
[refer to attachment]
But if 1 server is not sufficient, it would be possible to add others in different similar sized locations.

Hope this makes more sense.

 

[Reader]

You may want to look at GPU HW acceleration to achieve those large numbers of 1080p streams, and force direct play for the 4k streams.

The following shows some configurations with the P2000:

Source: http://slothtechtv.com/2018/09/hard...e-p2000-and-a-2u-supermicro-dual-x5670-server

I have a p2000 along with a e5-1650v3 and I've had 12-15 concurrent streams with no noticeable issues.

 

[rivey]

In general the set-up is aimed for a gaming gallery project in an open attic space, with no space to add racks. Therefore the idea was to put the server(s) above the ceiling panels between the structural beams (therefore the limited space). But looking at the transcoding requirements, even without any space restriction achieving the amount of streams is “challenging”, if I understand correct:

Space Constraint:
As mentioned above the goal was to put the server above ceiling panels:
[refer to attachment]
But if 1 server is not sufficient, it would be possible to add others in different similar sized locations.

Hope this makes more sense.

If I am understanding correctly after reading this and looking at the attachment, you want to put the server in the overhead space that is not climate controlled?
If that is the case, you are asking for trouble. Heat is the enemy of any electronics. I am a low voltage contractor and on the rare occasion that I have been asked to install an alarm panel (as an example) in an attic to save space, I explain to the client that they are then shortening the life span of the electronics. Isn't there a way to put the server in a cabinet or other area? I hope that I am wrong in the assumption of non climate controlled space.

 

[kdragon75]

Can't you just load the videos on flash drives and put them in the TVs? This seems like gross overkill to just play a few videos on a loop.I see your playing to tablet too. If you need this ondemand, transcode everything to 1080p. Unless you playing this on 50"+ 4k TVs or your standing 4ft away and looking for pixels, you won't see the difference anyway.
So you telling us that there will be no space for something like this:
http://www.fractal-design.com/home/product/cases/node-series/node-304-black
And it NEEDS to be in the attic space? Where is the 10gbe switch? What can't it go there? Is that above the ceiling too? I hope it doesn't cook.

 

[Ender117]

Thanks Chris and kdragon75, for your continuous support and inputs.

Based on your feedback, I realized maybe it makes sense to take a step back and divide the discussion into 2 parts:

• Technology required to achieve the streams (10x 4K and 20x HD-Streams)
• Space constraints

In general the set-up is aimed for a gaming gallery project in an open attic space, with no space to add racks. Therefore the idea was to put the server(s) above the ceiling panels between the structural beams (therefore the limited space). But looking at the transcoding requirements, even without any space restriction achieving the amount of streams is “challenging”, if I understand correct:

Technical:

• Up to 10x 4K TVs with Nvidia Shields
  Input: H.264 & H.265 4K files – therefore the idea was with the Shields that there is no 4K transcoding needed. (Yes, I have considered that the shields have local storage, but the requirement is to have all media available to all TVs. Hence the need for the central server).

• The HD streams, would stream to Tablets or other personnel devices.
  1. With the numbers from Plex (Transcoding a 1080p) file requires as mentioned a PassMark score of 2000. So with 20x Streams that would mean 40.000 – so no processor on the market today is able to achieve that, right?
  2. So the question would be how to achieve 20x streams, what would be the best approach?

• Avoiding the huge amount of transcoding? (the tablets selected would need to play 1080p and H.264/5 content)?
• Having more servers? (refer to space constraints)
• Pre-encoding the content further to 1080p and 720p?
• ?

--> As mentioned by Chris, I guess this is a bit out of scope for this forum, but any input is appreciated. But if required/preferred I can take it over to PLEX or elsewhere :)

Space Constraint:
As mentioned above the goal was to put the server above ceiling panels:
[refer to attachment]
But if 1 server is not sufficient, it would be possible to add others in different similar sized locations.

Hope this makes more sense.

As you already realize that transcode that amount of streams requires fairly large amount of CPU power. While a dual or quad CPU system should be able to do it I believe it's out of the scope of the project.

Now if you are willing to consider hardware transcode, take a look of these systems:
https://www.supermicro.com/products/system/1U/5019/SYS-5019S-TN4.cfm
https://www.supermicro.com/products/system/Mini-ITX/1019/SYS-1019S-MP.cfm
It's rated to do 18 H264 1080p@30fps in mini-itx format. You will need get 2 of these and somehow distribute the load.

Or you can opt for Nvidia, get a beefy Quadro like GP100:
https://developer.nvidia.com/video-encode-decode-gpu-support-matrix
https://developer.download.nvidia.c...ZikCRg2EcxT9h2IFxzwjUHlhrn8-Rdxkm6U4l18zYXA1A

None of these can be done on FreeNAS though (the plex plugin is software transcode only)


As others pointed out, do look at the heat issue.

 

[Ender117]

I have a p2000 along with a e5-1650v3 and I've had 12-15 concurrent streams with no noticeable issues.

I am interested in your builds. I got myself a P2000 exactly for this

 

[Reader]

I am interested in your builds. I got myself a P2000 exactly for this

My Plex server is currently running in Ubuntu not FreeNAS unfortunately, so I can't give stats for an apples-apples comparison though I'd imagine it's the same. I am moving to FreeNAS with the CPU and GPU soon, and had planned on putting up a build report with tests for concurrent streams when that is done. Apparently FreeBSD doesn't support hardware acceleration :(

 

[CK224]

Thanks All, for the further inputs, gives me valuable new directions to look into.
As it currently seems, the hardware transcoding could be a solution - with the result to go away from FreeNAS if I understand Ender117's comment correctly:

None of these can be done on FreeNAS though (the plex plugin is software transcode only)
As others pointed out, do look at the heat issue.

The points with the heat / installation methods are well appreciated and noted - I am trying to find a workable solution for the environment, but will focus first on the technology to achieve the streams and then see in which form factor it can be put into.