My new 48 bay build.

[Chris Moore]

I do hold some reservations to high density cases as this simply because if it's placed in an uncontrolled environment (no a/c) the thermodynamics are always going to be working against the components. I read how your CPU temps spiked (could be a bad fan or 2) but the first row of disk get the coolest air (suppose 25C). It absorbs some heat and moves to second row probably with 27-30C (non scientific calculations) and to the 3rd row even warmer and then the CPU and HBAs which run hot too are receiving air close to 40C.

Each row of drives does get warmer moving toward the rear of the case. As all sides of the case are smooth and solid, there would be no path for the air to move if all fans were blowing in. It might be a benefit if the fan direction were reversed bringing cool air in from the rear of the case and out the front, but that would make it harder to filter for dust. As it is, I have added dust filters to the front intake which helps reduce the amount of dust that accumulates inside the chassis. I received it January of 2018, so it has been running for around a year and a half now and all the drives are within tolerable ranges and system board components are well within spec. The CPU temp problems I had initially were corrected by a better CPU heatsync with a larger fan. Overall, I am very pleased with the system and I wish I had bought a second when I had the chance, it isn't particularly loud either. The 24 bay Supermicro system I have is a lot louder, I need to investigate getting some of the quiet power supplies for it.

 

[kabutomz]

Is that eight 10Gb ports?

This one is 24 ports...
https://www.ebay.com/itm/Brocade-BR-VDX6720-24-F-6720-Layer-2-Ethernet-Switch-24x-10-Gb-SFP-Uplink-JMW/232231406925

Do some research on it first and make sure it will work for your purpose. I would expect it to be fine and give you plenty of expand-ability for the future.

Chris, we got the Quanta LB6mSwitch and we got 20 of those 4TB SAS drives you recommended. I have two questions:

1) How should I configure those 80TB, i mean we would like to have some RAID.
2) Do we need one SSD for FREENAS or should we stay with USB and use one 120/SSD for Cache/Logs?

Thanks

 

[kabutomz]

Sorry for the late entry. I guess I'm a bit late to the party. Just been busy with personal and professional life.
Congrats @Chris Moore.
I do hold some reservations to high density cases as this simply because if it's placed in an uncontrolled environment (no a/c) the thermodynamics are always going to be working against the components. I read how your CPU temps spiked (could be a bad fan or 2) but the first row of disk get the coolest air (suppose 25C). It absorbs some heat and moves to second row probably with 27-30C (non scientific calculations) and to the 3rd row even warmer and then the CPU and HBAs which run hot too are receiving air close to 40C.
what I'm about to say is my belief but anyone with a background in physics will agree I'd suggest either
1. Putting all fans to pull/pull mode. that ways you're sucking cooler air from all sides for a more even cooling.
2. Making a few mods. (Remember hot air rises). Before each row of hard disks perforate the case on the bottom side and after each row perforate the top of the case. Though I understand cases are made keeping the airflow in mind, I know a guy (yes the same one everyone knows) who used a similar case and in the second row put a thermocol (non conductive material) at a 45 degree angle to put the air out of the top and then on the bottom got an inlet which is separated by the same thermocol piece. As per him dividing the case into 3 sections (1st row HDD/2nd row HDD/ Other components) has the machine run much cooler.

Just saying. Don't punch holes in the beautiful case coz a guy on the forums said so. :p

We are thinking about using this solution for our server room:

https://stmfab.net/?gclid=EAIaIQobChMI857U9Ze34wIVDfDACh1LMAzHEAAYAiAAEgKnpvD_BwE

 

[Chris Moore]

80TB

First, you can't think of the raw capacity of the drives. I never look at 20 x 4TB drives and think, wow, that's 80TB, because it isn't. It never was.
A 4TB drive only has 3.6TB of usable space to start, and the idea that it is 4TB is marketing hype.

How should I configure

Depends on how you plan to use it.

How should I configure those 80TB, i mean we would like to have some RAID.

I would want to have a few drives for cold spares, so the thing I would do is use 18 of the drives in three vdevs (6 drives each) at RAIDz2.
That dedicates six drives to redundancy (2 in each vdev) so that after redundancy and checksum data, and allowing for not filling to pool over 80%, you would have about 37TB of usable space. It is a total of 46TB, but you need to keep utilization below 80% to keep the performance good.

Do we need one SSD for FREENAS

I use SSDs for boot drives on the systems I manage for work. They are just a more reliable solution and I don't want to worry about a system fault taking the storage down. There is such a thing as a USB DOM (Disk on Module) that is built more robust, but a regular USB Memory Stick is just not up to the task of being the boot media for a server. I avoid them.

We are thinking about using this solution for our server room:

I will need to look at that from home. The video is being blocked by our network here at work.

 

[Chris Moore]

I ordered a new chassis. A closeout special. I normally don't buy new, but I made an exception..
It arrived today. View attachment 22613

http://www.chenbro.com/en-global/products/RackmountChassis/4U_Chassis/NR40700

I have been using this for about 18 months now and I am just as happy as can be with it. I needed to figure out the cooling situation a little, but after that it has been smooth sailing. It even has locate LEDs on the drive bays to make finding a drive easy when I need to replace a faulty drive.

There is a vendor selling them on eBay right now, used:
https://www.ebay.com/itm/Supermicro-4U-48-Bay-FREENAS-Storage-Server-2x-Xeon-Low-Power-L5640-6-Core-32GB/142950808625

They have an older X8 generation system board, which I would replace with something newer, but if you are looking to mount more drives and you don't care about the extra heat and power for the older board, they look like a decent deal. If you are going to replace the system board, maybe you can make a deal with the seller...

NOTE: I have no affiliation, I just thought it was a deal...

 

[Chris Moore]

Edit: Don't use these guys. See the next message from @eptesicus
These might be even better deals, but you would need to 3D print your own drive caddies because they don't include any:
https://www.ebay.com/itm/Chenbro-NR40700-3-5-4u-48-Bay-Chassis-w-Slide-Rails/383089889304

https://www.ebay.com/itm/Chenbro-NR40700-3-5-4u-48-Bay-Chassis-E5-2603-XEON-32-Gb-RAM-REG-ECC/383068308295

 

[eptesicus]

These might be even better deals, but you would need to 3D print your own drive caddies because they don't include any:
https://www.ebay.com/itm/Chenbro-NR40700-3-5-4u-48-Bay-Chassis-w-Slide-Rails/383089889304

https://www.ebay.com/itm/Chenbro-NR40700-3-5-4u-48-Bay-Chassis-E5-2603-XEON-32-Gb-RAM-REG-ECC/383068308295

I've been watching this thread for awhile, and ended up buying 2 of the NR40700 chassis a couple months ago.

I would have gotten mine sooner, but I had a bad experience getting the servers you linked from the 'findingdeals' seller on ebay. I wanted to chime in and share my experience so that someone doesn't have to go through it as well.

I would not order these from them, as they are either incompetent when it comes to shipping servers, or they are banking on them getting damaged, collecting insurance from UPS, and then reselling the parts from these (they had fans, caddies, backplanes, SSD bays, etc listed a couple weeks after my experience). When mine arrived damaged (UPS ground even though me and the seller agreed on freight) with junk pieces of styrofoam thrown in the box (no thought given), they drug their feet shipping me replacements. In my correspondence with them, they advised that I keep the rails kits, caddies, and I asked if I could keep the PSUs and fans. They agreed. They continued to say the replacement units would be shipping out, but excuses always came up when I asked for updates... When they finally received the servers I shipped back to them from UPS, they sent me messages claiming that I was not owed a refund and I would not be getting my replacement servers until everything I removed from them was shipped back (even after showing them the messages that they told me to keep them and use them in the replacements). I stood firm as our correspondence confirmed that I was to keep the salvageable parts from the units, and I was not going to ship those parts back at MY expense.

They ended up refunding me $30 combined after 5-6 weeks going back and forth, and eBay finally stepped in and got me my full refund from them without hesitation, because of the messages they sent me telling me to hold onto the caddies, PSUs, rails, and fans.

On the upside, I ordered one of the NOS NR40700s on ebay without caddies (from kammkr), and was able to use the caddies I got from the findingdeals fiasco, but it was not worth the time and constant aggravation in dealing with them.

 

[Chris Moore]

On the upside, I ordered one of the NOS NR40700s on ebay without caddies (from kammkr), and was able to use the caddies I got from the findingdeals fiasco, but it was not worth the time and constant aggravation in dealing with them.

Wow. Sorry for the trouble you had with that.

 

[eptesicus]

Wow. Sorry for the trouble you had with that.

Me too. But all is good now with the two chassis I have. Have you toyed around with different cards with these dual-backplane biggins? I'm running Dell H310 cards (flashed to IT mode) in each of my systems, one port to each backplane, but I'm getting about 200MBps transfer between my two systems, and have redundant 10GbE links on each. NAS01 is running 2x Z1 8x10TB vdevs, and NAS02 is running 5x Z1 8x4TB vdevs. I'm understanding that Z1 is a no-no anymore, but right now I didn't want to sacrifice the usable space, especially since NAS02 is a backup of NAS01.

Do you think my H310 HBA is a performance bottleneck? Would I see any gains on an HBA with a SAS3008 chip in this system?

 

[Chris Moore]

I'm running Dell H310 cards

I like this card, because the Dell card is PCIe 2.0:

https://www.ebay.com/itm/HP-H220-6Gbps-SAS-PCI-E-3-0-HBA-LSI-9207-8i-P20-IT-Mode-for-ZFS-FreeNAS-unRAID/162862201664

 

[eptesicus]

I like this card, because the Dell card is PCIe 2.0:

https://www.ebay.com/itm/HP-H220-6Gbps-SAS-PCI-E-3-0-HBA-LSI-9207-8i-P20-IT-Mode-for-ZFS-FreeNAS-unRAID/162862201664

Thanks, Chris. Likely no performance gains in going with an H240 seeing as it's 12Gbps?

 

[Chris Moore]

Thanks, Chris. Likely no performance gains in going with an H240 seeing as it's 12Gbps?

The mechanical nature of the drives limits the performance. I did the math on it once and figured that with the fastest mechanical drives available, you would need around 72 of drives before you would come close to the data throughput to need a 12Gb SAS controller. There are situations for it, for example, I have a system at work that is running 124 drives, but it is just not needed in most circumstances. Not with mechanical drives. SSDs are a whole different story.

 

[eptesicus]

The mechanical nature of the drives limits the performance. I did the math on it once and figured that with the fastest mechanical drives available, you would need around 72 of drives before you would come close to the data throughput to need a 12Gb SAS controller. There are situations for it, for example, I have a system at work that is running 124 drives, but it is just not needed in most circumstances. Not with mechanical drives. SSDs are a whole different story.

Thanks for the insight. I figured there wouldn't be a gain, especially given my use of shucked Easystores, but I appreciate the clarity.

 

[Chris Moore]

I'm getting about 200MBps transfer between my two systems,

There are a lot of other things that can impact your performance besides the pool. You might want to do some testing of the pool performance internal to each system along with some network performance testing to determine where your restrictions are. I found that my system was impacted by the virtual network adapter that the Plex Jail uses. With no jails installed, the transfer speed was almost twice what it is with a jail even if the jail is just sitting idle. I have been considering the option of putting Plex on a separate system and just using FreeNAS for storage. The virtual adapter driver that gets installed to handle jails is the problem in my system.

 

[eptesicus]

There are a lot of other things that can impact your performance besides the pool. You might want to do some testing of the pool performance internal to each system along with some network performance testing to determine where your restrictions are. I found that my system was impacted by the virtual network adapter that the Plex Jail uses. With no jails installed, the transfer speed was almost twice what it is with a jail even if the jail is just sitting idle. I have been considering the option of putting Plex on a separate system and just using FreeNAS for storage. The virtual adapter driver that gets installed to handle jails is the problem in my system.

I run all of my services on some clustered Cisco UCS servers, so my FreeNAS devices are just for serving up storage via SMB. I ran iperf tests between the systems, and they were all good in getting about 95% of my 10Gb connectivity (both NAS' are connected via 2x 10Gb NICs to a Ubiquiti Edgeswitch ES-16-XG). I haven't done it in a while, so I can't remember my results, but I should also setup a RAMdisk on each NAS and try to do a transfer to see what sort of data speeds I can get. Also, when I do a sync of the systems for right now, I'm doing 'scp' with the aes-128-cbc cipher, as that's been the best performing for me.

 

[David Henrickson]

Well....lets re-initiate this thread, as I have been staring at it, due to having the exact same system. Bought the deal like @Chris Moore did. However, mine did not come with the HDD trays. Yes, you have stated that they are only for alignment, but I really want it looking nice and not having to worry about the drive not secure by having some wiggle room. Does anyone have a line on these caddys (plastic liners)? I know Chris has stated to 3D print them....but like a hairline % of people have those and I happen to be someone who does not. Yes...48 caddys, but someone has to know how to obtain them for this...er....1 year-old build (I'm taking my time, leave me alone, haha). I haven't even picked out a motherboard yet, as I know I want ECC and now potentially a Ryzen setup due to the cost and benefit of it all. Suggestions? Ideas?

 

[Chris Moore]

Does anyone have a line on these caddys (plastic liners)?

I contacted the company that made them around a year ago. They only had 17 of them back then and they wanted a lot for them.
I hear there is a 3d printable version if you can find the file for it. If you have the ability to do 3d modeling, I could take photos of mine and careful measurements.

Suggestions? Ideas?

I like the system board I have, if you can find one.