BUILD Building a server for FreeNas primary for Plex

[AltecBX]

It easily saturates my 1G network. I've got some 10Gb Chelsio NIC's on order from ebay. Then it will be an interesting test. I haven't tried w/o my ssd's. They are there as a SLOG and L2ARC, not as a separate pool.

Sweet!

What I meant are the drives from my workstation.
I'm getting an average of 90-95MB/s transfer rate to my FreeNAS from my HDD in my workstation. I'm getting 105-115MB/s transfer rate to my FreeNAS from my SSD.

 

[depasseg]

I'm getting 108-115 between my laptop and FreeNAS.

 

[AltecBX]

So there you go @ggoldfingerd, if you don't have or plan a 10Gbs network, I don't think a SAS3 will benefit you that much. It might improve some transfer speed internal but nothing drastic that you'll notice day to day as a file/media/backup/plex server.
I know FreeNAS will get this issue resolved. As for now I don't know the date, but since this configuration plays nice with every other OS, I don't see FreeNAS not fixing this issue with their drivers in the near future.

 

[ggoldfingerd]

So there you go @ggoldfingerd, if you don't have or plan a 10Gbs network, I don't think a SAS3 will benefit you that much. It might improve some transfer speed internal but nothing drastic that you'll notice day to day as a file/media/backup/plex server.
I know FreeNAS will get this issue resolved. As for now I don't know the date, but since this configuration plays nice with every other OS, I don't see FreeNAS not fixing this issue with their drivers in the near future.

Thanks. How is the noise on your system?

 

[depasseg]

Here are a couple videos of mine for comparison, including my SM 5028R.
https://forums.freenas.org/index.ph...ni-itx-cases-quieter.25635/page-2#post-165599

 

[ggoldfingerd]

Here are a couple videos of mine for comparison, including my SM 5028R.
https://forums.freenas.org/index.ph...ni-itx-cases-quieter.25635/page-2#post-165599

Good info. The SM 5028R is pretty quite. If I went with the SM 5028R, I would also want the SAS NAS drives you purchased for the SAS 6Gb/s. That would drive my costs up even more! Might be a waste for my 1 Gbs LAN anyway. Do you happen to know what kind of wattage your SM 5028R is drawing? You got a good deal on the SAS drives BTW.

Cost is still a factor for me. I can afford it, but that cuts into savings. I want to purchase a home at some point! I was really planning to go with SC836E26-R1200 chassis and WD Red SATA drives. In the future, I can upgrade to a SAS 6 Gb/s backplane / chassis with SAS drives. Going with the used chassis will bring the overall cost down about $1000 and I will get about 10 TB more storage assuming I went with the SAS drives with a new chassis. Tough decisions!

 

[depasseg]

Under normal conditions it's reporting 218 watts. It reports a max of 282 watts.

decisions, decisions!

 

[AltecBX]

Here is my server noise. I was testing a LACP connection earlier and finally got the server on the Wall Chassis.
It's running 2 Plex connection right now.

View My Video

 

[ggoldfingerd]

Here is my server noise. I was testing a LACP connection earlier and finally got the server on the Wall Chassis.
It's running 2 Plex connection right now.

View My Video

Thanks, what about power usage?

 

[AltecBX]

Here's my power consumption from the IPMI.

 

[TXAG26]

ps - For anyone with this board or similar from SuperMicro. They recommend me to leave the memory exactly how i had it. A1, A2, B1, B2. basically on one side of the CPU. This way it gives me the best performance and it operates in Memory Interleaving. Any other way, it won't do Memory Interleaving.

In that case, if you don't mind, it'd be nice to know if any weird problems arise when slots A1, B1, C1 and D1 are populated, with the rest empty, and vice-versa.

Was anyone ever able to do any testing with 4 sticks of ram in the X10SRH-CLN4F to see which slots actually need to be populated to take advantage of quad-channel ram?

To recap, previous manuals for boards in the X10 generation stated to populate DIMMS as follows:
DIMM-A1, DIMM-B1, DIMM-C1, DIMM-D1; DIMM-A2, DIMM-B2, DIMM-C2, DIMM-D2.

The quick-guide for the X10SRH-CLN4F and a phone tech with SM stated to populate DIMMS as follows (for "Balanced" performance, whatever that means):
DIMM-A1, DIMM-A2, DIMM-B1, DIMM-B2; DIMM-C1, DIMM-C2, DIMM-D1, DIMM-D2.

This board has FOUR channels (A, B, C, D) and TWO Banks (1 & 2). I have a sneaking suspicion that to get both quad-channel performance (A,B,C,D) AND memory interleave (1 & 2), one stick in each DIMM Bank (e.g. Bank 1), and two sticks in each DIMM channel (E.g. Ch. A) are required. E.g. each of the 8 memory slots require a DIMM for optimal performance.

Can anyone confirm? I would think this could be confirmed with MemTest since it indicates various speeds and memory bandwidth.

Quick-Reference Charts from two different SuperServers that contain the X10SRH-CLN4F board:
http://www.supermicro.com/QuickRefs/superserver/2U/QRG-1607.pdf
http://www.supermicro.com/QuickRefs/superserver/4U/QRG-1609.pdf

 

[Ericloewe]

You can't get quad channel performance if two channels are empty. That's certain.

I really can't imagine a valid reason for recommending setting it up as dual channel.

Another certainty is that the channels are grouped as two on each side of the processor, so ABCD are in fact the channels.

A processor bug is unlikely, since other boards recommend the obvious approach of one per channel - and it's working fine on my desktop (Xeon E5-1650v3, four 4GB Crucial ECC RDIMMs, ASRock X99-WS)

 

[jgreco]

Using the second bank may also limit peak memory speeds.

 

[Ericloewe]

Using the second bank may also limit peak memory speeds.

Don't think it's the case with Haswell-EP, since the memory clocks are lowish to begin with.
Definitely the case in many DDR3-based systems.

 

[jgreco]

Don't think it's the case with Haswell-EP, since the memory clocks are lowish to begin with.

I call wrong, sorry.

http://www.supermicro.com/manuals/motherboard/C612/MNL-1570.pdf

Page 2-15, population of DIMM1 slots in either RDIMM or LRDIMM configurations results in the elimination of DDR4-2133, reducing to DDR4-1866. We've verified this in shop. This isn't necessarily applicable to every Haswell-EP board. LRDIMM fares somewhat better sometimes.

Definitely the case in many DDR3-based systems.

Yes, sometimes appallingly so.

http://www.supermicro.com/manuals/motherboard/C606_602/MNL-1333.pdf

See page 2-13. Fully populating drops speeds from DDR3-1866 to DDR3-1066 or even DDR3-800.

 

[Ericloewe]

I call wrong, sorry.

http://www.supermicro.com/manuals/motherboard/C612/MNL-1570.pdf

Page 2-15, population of DIMM1 slots in either RDIMM or LRDIMM configurations results in the elimination of DDR4-2133, reducing to DDR4-1866. We've verified this in shop. This isn't necessarily applicable to every Haswell-EP board. LRDIMM fares somewhat better sometimes.



Yes, sometimes appallingly so.

http://www.supermicro.com/manuals/motherboard/C606_602/MNL-1333.pdf

See page 2-13. Fully populating drops speeds from DDR3-1866 to DDR3-1066 or even DDR3-800.

Well, that's unfortunate. I have to say I'm disappointed that LRDIMMs don't manage DDR4-2133, given the hype they got a while back.

 

[TXAG26]

The Supermicro X10SRH-CLN4F board came in today and I got it assembled with an E5-2618L v3 and 4x 16GB Samsung ECC DDR4 RDIMMS (DDR4 running at 1866 MHz, as expected and supported by the E5-2618L).

Reluctantly, I installed the ram as stated by Supermicro (DIMMA1, DIMMB1, DIMMA2, DIMMB2), as pictured below.

While running Memtest86, I noticed that the DDR4 ram appears to have about 12.5 GB/s of bandwidth. Is this correct? According to ARK, the E5-2600 v3's have 59 GB/s, which I assume is for a dual cpu system. 59/2 = 29.5 GB/s. According to Memtest86, I'm seeing less than half of the 29.5 GB/s expected bandwidth (I'm only seeing 12.5 GB/s if I'm reading this correctly).

Has anyone else with an E5-2600 v3 seen what their memory bandwidth displays in Memtest86? If possible, I'd also be interested to know what memory bandwidths those of you with 8 sticks of DDR4 have in Memtest86.

A quick and dirty way to boot Memtest86 is to download the memtest86 iso and use the default settings in Rufus to convert and write it to a USB stick (Rufus converts it and also makes the USB stick bootable).

PS - I would swap the DDR4 sticks around, but the slots on my board are extremely tight and I needed to have the board sitting on a hard flat surface to get the memory seated all the way. Judging by how tight they were, the board would flex past my comfort zone if I were to insert DDR4 while it was screwed into the case on the stand-offs.

http://ark.intel.com/products/83351/Intel-Xeon-Processor-E5-2618L-v3-20M-Cache-2_30-GHz

 

[marbus90]

You must've got one hell of a deal for that CPU...

You have a Quadchannel capable system.

You want to use Quadchannel with your 4 DIMMs.

So, I see 4 channels listed as DIMMA, DIMMB, DIMMC and DIMMD.

Use DIMMA1, DIMMB1, DIMMC1, DIMMD1.

 

[DataKeeper]

I did the same thing. Change the ram modules and place them in DIMMA1, DIMMB1, DIMMC1, DIMMD1. It was @marbus90 who caught my mistake as well :D

 

[Ericloewe]

It's Supermicro's mistake, mostly. I'd like to talk with the guy who wrote the DIMM slot population charts.