BUILD SuperMicro X10SRL-F + 3 846 Chassis + 72 Disks

[marbus90]

and they call this stable...

We're just betatesters for TrueOS. :)

 

[Apollo]

It sure was nice and quiet, but I later found out that my disk drives were being cooked! I only discovered this as I removed drives that failed SMART test. Is there a way to monitor disk drive temps from the GUI?

Here's a shot of the X10 chassis guts in it's operational state. Note USB stick on on-board USB 3.0 connector and IBM 1015 installed. This shot also shows the 4 wire mid plane fans all wired up, but as I mentioned above, with optimal fan speed, the disk drives run way hot! You can also see that I'm not running any rear panel fans at this point. CPU and RAM temps are higher than before, but I think that is all related to mid plane fan speed. I've switched all midplane fans to full speed for now to cool down the HDs.

No wonder your drives are cooking. Looking at your SAS expander boards, they are right between your HDD's and fans. Once the chasis is sealed, very little amount of air will be able to circulate from the drives to the fans. Also, the air vents on sides are preventing the air from going through the drives because the air is trying to find the path of least resistance to the fans.
It seems whoever designed the chasis made a huge oversight unless the intent was to only cool the processor.
Terrible design I tell you.
This is the reason why those fans are running full blast.
If I follow the "Air flow direction" on your CPU fan, I must assume the air flow is from from to back, so yes, the fans are mostly there to keep CPU and motherboard components cool, regardless of the HDD.

Is there a way you can improve HDD air flow distribution? The problems are the air vents on the sides and the expander boards.

 

[pclausen]

So this is a SuperMicro 846 chassis with their SAS2 expander board that comes standard with the case. Here's a shot of my spare BPN-SAS2-846EL1 expander board showing the 24 female SAS/Power connectors. There sure doesn't appear to be a whole lot of airflow possible.

My current production system, which uses the same chassis and sits in my basement, is currently showing disk temps ranging between 23 and 35 Celsius.

I agree that blocking off those side vents would be a good 1st step. In fact I'm going down to the basement to do that right now on unit there to see what it does to the temps. That box is running full blast since it only has 3 wire fans.

 

[marbus90]

No wonder your drives are cooking. Looking at your SAS expander boards, they are right between your HDD's and fans. Once the chasis is sealed, very little amount of air will be able to circulate from the drives to the fans. Also, the air vents on sides are preventing the air from going through the drives because the air is trying to find the path of least resistance to the fans.
It seems whoever designed the chasis made a huge oversight unless the intent was to only cool the processor.
Terrible design I tell you.
This is the reason why those fans are running full blast.
If I follow the "Air flow direction" on your CPU fan, I must assume the air flow is from from to back, so yes, the fans are mostly there to keep CPU and motherboard components cool, regardless of the HDD.

Is there a way you can improve HDD air flow distribution? The problems are the air vents on the sides and the expander boards.

These aren't "SAS Expander boards". These are backplanes. Used for plugging HDDs in. Normal chassis require each HDD individually cabled. It's a server rackmount chassis, not built to be silent or have low air troughput. Servers need a really high air flow. These chassis are made to run 24x 15krpm HDDs in there, which are using ~15W each instead of the 5-7w of consumer drives.

BTW, a single SFF-8087 cable is enough to the expander - but since you have 3x M1015, you can easily run two SFF-8087/8088 cables per chassis, ideally doubling your troughput.

 

[pclausen]

Blocking off the side vents shaved 3 degrees on the hottest drives. So the range is now 23 to 32. Not a bad tip, but as marbus mentioned, these are a pretty solid design from the get go.

Marbus, yes, I am actually dual-linked in my current main system. However, with just 'regular' SATA drives, I'm not really able to saturate the 24 Gbps bandwidth of even a single link. If I was running 15k drives instead of 7.2k drives, or SDDs, it would be a different matter. That said, I still plan to run dual link to all backplanes because I can and already have the cables. :)

 

[Ericloewe]

I look at that PCB and think "Holy crap, did was any thought given to aerodynamics or did they just plan to solve the cooling problem with 10k RPM fans and everything else be damned?"

Hell, an extra layer on that thing and they could've given the drives larger openings. The airflow through the small holes probably accounts for as much noise as the fans themselves.

 

[Apollo]

It's like adding a car exhaust pipe to a fighter jet and trying to go supersonic.
I would think those backplanes would have a better flow if air is coming form the front of the drives and escaping to the sides.
The reason why those fan are running like hell is to have an insane backpressure.
If you could ditch those backplane or find a way to relocate them and use cables to extend to the drive, you will most certainly improve you air flow. You should also be able to run much quieter too.

 

[DataKeeper]

I have that same BPN-SAS2-846EL1 expander. I have 12 REDs in running through a badblocks test currently at 18 hours and just completed the Reading and comparing test and staring the next pattern test.
The following are the temps of my REDs
da0=26c ; da1=25c ; da2=23c ; da3=26c ; da4=25c ; da5=23c ; da6=27c ; da7=28c ; da8=27c ; da9=26c ; da10=26c ; da11=25c

Internal temps: CPU=37c ; PCH=33c ; System=21c ; Ram=29c

Air Temperature in unfinished, fully below grade, basement is 66F.

I'm running all 5 fans plus 2 CPU fans in HeavyIO Mode since my drives are stacked in the left side bays (backplane IDs 0-11). The 3 center fans are running at 4000, 4100 & 4500. The CPU fans at 900 and the rears at 3500.

Currently pulling 216W.

There isn't enough draw through the front to hold a sheet of paper. The side holes on the left front will not hold the paper however the ones a few inches back will hold the paper.

What are your other temps like?

 

[mjws00]

Lol. The fskn 846 design is solid. It will cool 24 hot 15k drives. Plus 2 x8 gen Xeon space heater CPU's at full load and the bloody controllers as well.

Our idle slow spinning drives and modern idling Xenon's are a walk in the park. Temps do NOT need to be low 20's. A little smart baffling if you aren't using every bay, and you can run nearly silent and still be sub 35 degrees.

 

[marbus90]

It's like adding a car exhaust pipe to a fighter jet and trying to go supersonic.
I would think those backplanes would have a better flow if air is coming form the front of the drives and escaping to the sides.
The reason why those fan are running like hell is to have an insane backpressure.
If you could ditch those backplane or find a way to relocate them and use cables to extend to the drive, you will most certainly improve you air flow. You should also be able to run much quieter too.

DITCH THE BACKPLANE?

You could just say you want to reinstate witchhunt.

I think I have a fish somewhere...

 

[pclausen]

The following are the temps of my REDs
da0=26c ; da1=25c ; da2=23c ; da3=26c ; da4=25c ; da5=23c ; da6=27c ; da7=28c ; da8=27c ; da9=26c ; da10=26c ; da11=25c

How are you reading your drive temps?

Internal temps: CPU=37c ; PCH=33c ; System=21c ; Ram=29c

Air Temperature in unfinished, fully below grade, basement is 66F.

I'm running all 5 fans plus 2 CPU fans in HeavyIO Mode since my drives are stacked in the left side bays (backplane IDs 0-11). The 3 center fans are running at 4000, 4100 & 4500. The CPU fans at 900 and the rears at 3500.

Currently pulling 216W.

What are your other temps like?

I too am running HeavyIO mode right now. Seem to be a good compromise between noise and temperature. I don't have the 2 rear fans installed at the moment, and the FreeNAS chassis is sitting upstairs right now in my office, so probably around 70 degrees ambient or so.

Anyway, my temps are:

CPU 37C
PCH 36C
System 26C
Perf 31 C
VCPU/RM 34C
VMEMABRM 23 C
VMEMCDRM 28 C
DIMMA1 34 C
DIMMC1 34C

FAN1 4100 RPM (right midplane)
FAN3 5800 RPM (CPU)
FAN4 4000 RPM (middle midplane)
FANA 4700 RPM (left midplane)

PSU1 TEMP1 25 C
PSU1 TEMP2 33C
PSU1 FAN1 8427 RPM
PSU1 FAN2 7218 RPM

Current power consumption (with 10 2TB 7200 RPM and 10 1 TB 7200 RPM "old school" drives is 234W.

 

[pclausen]

Just for grins I enabled HDD standby and set it to 5 mins and also set Advanced Power Management to "1". I also shut down the one jail I had running (Emby) but "something" is still hitting all the drives as I'm seeing all the blue access LEDs blip in unison every 10 seconds or so.

 

[SweetAndLow]

It's every 8 second and that's from the .system dataset. It writes logs and statistics to the pool keeping disks active. Do a search for more info it's allover

 

[DataKeeper]

Quick and dirty.. Create a file named "custom_smart_temps" and put this in it..

Code:

smartctl -a /dev/da0 | grip -i temperature
smartctl -a /dev/da1 | grip -i temperature
smartctl -a /dev/da2 | grip -i temperature

Continue for each device. Type "chmod 700 custom_smart_temperature" which makes it read/writeable/executable only by root. Then run it like so..
[root@FileServ#] ./custom_smart_temps

Again, not pretty but it does the job quickly.

Corrected the typo and don't include the numbers ;)

 

[danb35]

s/grip/grep/

 

[pclausen]

It's every 8 second and that's from the .system dataset. It writes logs and statistics to the pool keeping disks active. Do a search for more info it's allover

Ah I see. I guess this is something that was introduced from 9.3. I don't plan to allow the drives to spin down, but was just curious to see what my power consumption would be if they were.

 

[pclausen]

Quick and dirty.. Create a file named "custom_smart_temps" and put this in it..

Code:

smartctl -o /dev/da0 | grip -i temperature
smartctl -o /dev/da1 | grip -i temperature
smartctl -o /dev/da2 | grip -i temperature

Continue for each device. Type "chmod 700 custom_smart_temperature" which makes it read/writeable/executable only by root. Then run it like so..
[root@FileServ#] ./custom_smart_temps

Again, not pretty but it does the job quickly.

Hmm, I wasn't able to make that work, but entering this at the command line works:

smartctl -a /dev/da0 | grep -i temperature

I tried sticking that in a file with "1. " in front of it, but that fails with a "1. " not found.

Anyway, here's VERY crude report of disk temps: :)

da19 is 32 C. :)

It would be cool if a future release would include a column for temp when going to "View Disks" under Storage.

 

[DataKeeper]

Yeah ok so I had a typo there.. It's -a and not -o. Not sure how that happened. Also don't include the numbers. Just the same line you enter on the command line.

# smartctl -a /dev/da0 | grip -i temperature
# smartctl -a /dev/da1 | grip -i temperature

Don't include the # :) Do that in a file for each device then type the chmod command and run the program as above.

 

[pclausen]

I went ahead and soldered an extension to the one rear fans so that it would reach one of the front fan connectors on the mobo. Also added a few wire-ties to clean things up a little.

I then dragged the NAS box down in the basement and sat it on top of the current X7 based production server (running Windows 8.1 64 bit).

So now I'm just waiting on the 3rd 846 to get here and once I get everything copied over and am happy with the setup, then comes the fun part of lifting them up into that open space in the upper left rack.

 

[DataKeeper]

Ya got a power station in back there somewhere? :D