So, here's the final results and links for the hardware used:
Case:
Norco-4224 case
Installed hardware:
CPU - E5606-2.13Ghz with powerd off
Motherboard - Some Gigabyte 1366 board - Forgot the exact model
RAM- 20GB ECC DDR3
RAID - Areca 1280ML 24-port in JBOD mode
The temperatures I'll be comparing will be the CPU, RAID controller on-chip(as read from areca-cli) and the hard drive temperatures as read from smartctl. I have had issues in the past where the RAID card overheated(very loud audible alarm) because of insufficient cooling(forgot to plug in my PCI slot fan.. whoops!) so I am very conscious of my RAID controller's temperatures now.
Summer hit and this was the first summer I've kept the server running. Since I have no central A/C the computer naturally got warmer, but hard drive temps went over 40C, then some were hitting 45C. Based on my reading of the Google white paper entitled "
Failure Trends in a Large Disk Drive Population" Figure 5(page 6) for longevity of hard drive life it is preferred to keep temperatures between 30C and 40C. Below 30C and above 40C and your hard drive failure rates go up, sometimes rapidly. At 3 years the annual failure rate(referred to as AFR for the remainder of this discussion) for a hard drive between 40-45C is over 10% and approaches 15% at as you pass 45C+. Based on this I want the hard drive temperatures to stay 30-40C for the longest lifespan. If you enjoy replacing drives at a higher frequency because you enjoy the exercise of resilvering a zpool, then you are definitely in the minority.
When I bought the Norco 4224 case I installed 24 drives. All are WD Green drives that are now 2-3 years old and a combination of 2TB and 3TB drives. There are 2 zpools of drives, with the associated drives bought at the same time.
When I bought my case I immediately dismissed the 4x80mm fan backplate and replaced it with the
upgrade 3x120mm fan backplate and
3xGELID 120mm fans that are effectively bearingless. The case was also outfitted with 2x80mm Gelid fans in the back of the case for exhaust air. The choice of fans was a personal one based on prior experience with these fans and their unusually long lifespan. I prefer fans that I install and never look at again. The case comes with 4x80mm
Delta(model AFB0812H) fans for comparison.
Unrelated sidenote(if you don't have WD Green drives you can skip this section): Because I bought WD Green drives I have used the wdidle3.exe tool to change the Intellipark technology(head parking timeout) from the default of 5-8 seconds to 300 seconds. A video showing how to do this from DOS(the tool is not available in any other OS) can be found at
View: https://www.youtube.com/watch?v=gqY5zlCXQmU
. Keep in mind that the latest version of the tool is 1.05 and hasn't been updated for a few years. If you choose not to use this tool you can cause premature failure of your hard drives(and a small performance penalty) due to excessive load cycling of the hard drives. They are rated for 250,000 to 350,000 cycles. In a desktop environment this can take several years, but in a RAID environment you can allegedly hit the rated lifetime in a matter of months. There have been rumors that some people have had RMAs denied due to very high load cycling values in the SMART data on the drive, so be warned. It's a foregone conclusion that if you rack up 200k cycles in 3 months then you weren't using the hard drive within its designed function and your warranty may not be honored. Because I'm using WD Green drives I don't consider it an option to buy cooler drives in order to achieve the 30-40C temperatures, but I can assure you that using high performance drives will result in higher temperatures(duh) and may require you to take actions. A friend that had 16 7200 RPM Seagate drives had major temperature issues with the drives idle(sometimes hitting 50C) with the 80mm fans that the case comes with. So keep that in mind when considering if you want to upgrade your fans. I'd wager that if you choose to use anything except "green" drives you
will need better cooling than the case can provide with the pre-installed hardware.
Earlier this summer I found that I started having hard drive failure. Up until this point I had one failure in February and none in the previous 3 years. Within about 2 weeks I had 4 more drives fail(thank goodness for redundancy). Looking at SMART data I determined that my hard drives were getting far too hot and I needed to do something about it. The information below is based on that information.
I have 5 sets of data for you: The 120mm backplate with the GELID fans, the 80mm backplate with the Delta fans that came with the case (AFB0812H) and Supermicro FAN-0126L4(NIDEC # V80E12BHA5-57). As some extra datapoints I have paired the Supermicro fans a PCI fan controller an provided temperatures at minimum fan speed and 50%.
I chose to use a fan controller for several reason: I believe these fans are far too powerful and will cool the hard drives below 30C, they are extremely loud(and I mean excruciatingly loud), and I prefer to minimized excessive vibrations of the case and therefor the hard drives. The fans at full speed are so loud I can hear them on my ground floor with the basement door closed. Going down in to the basement I feel like I need earplugs in my ears. Note that in the below values I have normalized the temperatures to 20°C(68°F) since the basement stayed between 67F and 70F. I verified that no scrub was in progress or any other heavy loading tasks for at least 2 hours before obtaining temperatures. All temperatures are in °C unless otherwise specified.
A few notes before I jump to the temperatures:
1. Running a zpool scrub(or resilver) will result in a measurable increase in temperatures. Because of this you ideally want to stay in the lower end of the 30-40°C range I'm attempting to hit.
2. A single fan failure with multiple fans in parallel can result in losing most of your cooling capacity because the air will take the path of least resistance. Using fans that are reliable is a major plus. I don't consider the Delta fans included to be particularly reliable as many people have complained that the fans died after less than 3 months of use.
3. The Supermicro fans are thicker than the Delta fans. It will be a tight fit, but they will fit.
4. The Supermicro fans I purchased are the 4 pin PWM fans. I chose not to use the PWM at this time, but I bought the PWM fans in case I have a need for PWM in the future. PWM controlled fans will work in a VR fan controller, but their speed control will be more logarithmic than linear. I purchased my fans from
ebay for $12 each plus shipping. Total cost was just over $50 for all 4 fans on my doorstep.
5. The 80mm Norco fan mounting plate has slots for the fans to slide in and out. The fans have 4 pin connectors, but only 3 leads are attached. The connectors on the Delta fans are the industry standard while the Supermicro have slightly different dimensions and not as long. The fan will easily slide into the fan slot if you remove the bracket from the Delta fans and install them on the Supermicro fans. Despite the deeper Supermicro fans the fan will fit, but you will not be able to use the Supermicro connector with the backplate connector. You will have to run your own power to each of the fans. Since I used my PCI fan controller I had no issues using a 1 foot 3-pin extension cable to connect the fans to the controller. Also you cannot simply replace the plastic connector on the 2 fans to make the Supermicro fan work with the Norco plate. If you wanted to do some splicing you could accomplish this task, but all 4 fans share the same molex connector so you'd need a fan controller capable of handling all of the fan load. Because of this I recommend you use the Norco fan slot hardware to mount the fan but use your own power source and appropriate 3-in fan extension cables.
Now onto the temperatures:
Temperatures with the 3x 120mm GELID fans as it was in my living room: (Estimated living room temperature was about 26C(78F) at the time)
Hard drives(min/avg/highest) : (35/40/43)
CPU: 43°C
RAID: 51°C
Temperatures in the basement with the 120mm fans:
Hard drives(min/avg/highest) : (35/40/43)
CPU: 42°C
RAID: 50°C
Notice that although the environment is much cooler, temperatures are pretty much the same.
Temperatures in the basement with the 80mm OEM(Delta) fans: (257watts at the wall for my system)
Hard drives(min/avg/highest) : (30/34/37)
CPU: 38°C
RAID: 45°C
Note that hard drive cooling seems pretty adequate for idle. When running a scrub the temperatures creep up and about 1/2 the drives go over 40°C.
Supermicro fans without a controller: (303watts at the wall for my system)
Hard drives(min/avg/highest) : (23/26/27)
CPU: 32°C
RAID: 37°C
WOW, talk about a "cool" server! The top and bottom row of hard drives are the warmest, with all 8 drives being 27°C. I never would have thought these temperatures possible with a high density case such as this.
Finally, I installed a fan controller(Sunbeam PCI fan controller PL-RS-PCI). These are discontinued and I can't find these for sale anywhere except Amazon, and they want $90 each! This is better than last week when they were asking
just $133 each. Naturally, any fan controller will do, but the PCI controllers are the best option I could find since there are no 3.5" or 5.25" bays with which to install most fan controllers.
Warning: The fan controllers are rated for 20W each and the fans are rated for 12W each. I used 2 controllers to control 2 fans each. Since I don't have any intentions of using the controller above about 1/2 way I'm not worried about damaging anything. I did try turning the fans up to 100% load and system load only went up to 288 watts and the fans weren't nearly as loud as they were without the controller. My guess is there is a regulator that limited power to 20w per controller since the unregulated 12v to the fans yielded a system load of 303 watts. Whether this was by design or because the controller was overloaded I don't know.
Couple of notes:
- On minimum fan speed the airflow feels roughly the same as the Delta fans when they were at full speed.
- On about 50% fan speed the airflow is fairly high. Far higher than the Delta fans could have ever achieved. About 50% is where the fan noise become high enough to begin to be a concern if I had this server installed in its closed rack in the back of my living room.
- Even at the minimum fan speed on the controller a sheet of heavy stock printer paper sticks to the case. This is a significant improvement over my "upgraded" 120mm fans which couldn't hold up a standard sheet of paper.
Supermicro fans at lowest setting on the fan controller: (I had an issue with this setting.. more information is below) (255 watts at the wall)
Hard drives(min/avg/highest) : (33/35/38)
CPU: 41C
RAID: 39C
Supermicro fans at 50% on the fan controller: (272 watts at the wall)
Hard drives(min/avg/highest) : (27/30/31)
CPU: 34C
RAID: 39C
At minimum fan speed I had some very odd problems with my server. I left the server to run for 5 hours idle to allow temperatures to stabilize. I went to a friend's house and when I came home I found out that I couldn't obtain SMART data for any of the drives on my RAID controller. The zpool showed all drives as online. I can't explain this condition because I had been running the temperature script for 3 weeks and never had this problem. After a reboot I got my temperatures for the hard drives. Within an hour 4 more drives wouldn't report SMART data from my Areca controller but the zpools still showed as online and my data was available. If 4 drives had actually been disconnected the zpool wouldn't have been able to function. Additionally, the areca-cli would not run. Attempting to run the program would simply result in a blinking cursor and the SSH session would become unresponsive. The system has an OCZ gold rated 850W PSU and the fans aren't electrically connected to any other hardware except potentially internally to the power supply. I checked the cabling is tight and nothing seems out of order. I really can't explain this behavior, but as soon as I increased the fan speed to 50% and rebooted I have yet to see a drive drop. It could be that at minimum fan speed there's some kind of electrical feedback from the fans or controller resulting in my hardware misbehaving. It's been about 5 hours since I changed the fan speed back to 50% and my intention is to keep the fans at this speed for the forseeable future(temps and noise are acceptable). I do have some upgrades for the FreeNAS hardware that are likely to happen in the next 2-3 weeks. If the issue reoccurs I will update the forum, but as I'm about to gut my system and do some upgrades we may never know the exact cause.
So let's talk about fan theory for a minute. There's a relationship between RPM, fan diameter, fan power consumption, noise, thickness of the fan, D/P(differential pressure) of the fan, and flow rate(there's more than this, but these are easily understood and usually available to the purchaser). In particular, D/P and flow rate matter the most for situations where high air resistance plays a part in cooling. As the fan diameter gets bigger you need the square of the depth of the fan to attain the same differential pressure assuming all other factors are the same. Unfortunately, all factors are not the same because a larger diameter fan running at the same RPMs will have its own centripetal force increase due to the increased mass of the fan blade related to the larger diameter. Fans that blow themselves apart when running at full speed generally aren't sold to the public so 120mm fans typically have a lower RPM than an 80mm fan of similar characteristics.
Put simply, bigger fans are great because they can move more air at a lower RPM with low air resistance(keyword: low air resistance). If you start dealing with situations where you have higher air resistance(such as a 24 disk 4U rack case) then you need to avoid 120mm fans because they will become excessively thick(remember that square of the thickness thing) and won't physically fit in a case. In my opinion, buying the "upgraded 120mm" fan backplate is no upgrade at all. I couldn't find any 120mm fans of any reasonable depth that had even 1/2 the D/P of the Supermicro fans I purchased. I had found 1 120mm fan that was something like 70mm thick and had 1/2 the D/P of the Supermicro fan. If you want better cooling than the 80mm fans that the case comes with, look at higher D/P fans such as the Supermicro fans I purchased.
I hope this helps out someone that is considering buying a 24U case and are concerned about the drive temperatures. Personally, i think my configuration is pretty ideal for "green" drives. If I were to go to 7200RPM(or higher) drives I'd highly recommend you buy better fans than the case comes with.. I think that the Delta fans were acceptable (but not ideal) for"green" drives, but if you want extra margin you need to get better fans.
This whole thing is brought to you by me... since I had 4 disks fail in short order due to hard drive temps. Yes, this was an expensive lesson learned since only 1 of the drives was in warranty still. But I'd rather spend $50 on fans than spend even more money on lots of RMAs, buying new drives when they are out of warranty, and resilvering disks regularly. I value my data and I don't see any reason to skip on cooling.
Anyone that wants to tell me that temperatures of a hard drive don't matter.. well, I'll keep my opinion of you to myself. :p