Long time listener, first time caller...

[iniazi]

Hello Everyone,

I just setup a 96TB-ish (usable) NAS for home storage and home-lab use. Here are the specs (i put the specs in my sig, but not sure if there is a policy for short sigs, and if so let me know and I'll remove it):

FreeNAS 9.3.1 Virtualized on ESXi 6.0
SuperMicro X9SRH-7TF (2x10GBase-T, LSI SAS2308--IT mode) Mobo
E5-1650v2 3.5ghz 6core Processor
64GB RAM (8x8gb DDR3 ECC Unbuffered)
2xM1015 (SAS2008) w/SAS9217-8i Firmware (IT)
22 x WD60EFRX 6TB WD Disks (96TB usable by using two 11 disk raidz3 vdevs)
2x120G SSDs (storage for VMs)
Norco RPC-4224
Corsair CS550M (550W) PSU
187W (idle)
410W (spinup)

It will be mainly used for:

1. Shared storage for the 4 PCs (including kids computers) and couple of laptops (documents, video editing etc.)
2. I usually do quite a bit of photography and 24megapixel dslr raws, gopro/camcorder storage etc. uses quite a bit of storage after maybe 5 to 10 heavy photo/video outtings
   
3. Media serving to 3 TVs (Kodi on FireTV, HTPC, XBOX360)
4. Running personal VMs for learning & experimenting/testing
5. Eventually an Emulator game box (have some Raspberry Pi + v2 & other extra PCs laying around)

I've physically built the box, installed ESXi & FreeNAS (32GB RAM assigned to FreeNAS), and created the main pool, plus a few datasets. I'm using two 120GB SSDs (I had them laying around) for VMs, including FreeNAS. I create a vmdk on each of the ESXi and FreeNAS is setup as mirrored. ESXi is booting off a USB. And ran a quick local benchmark of the pool and its writing at around 1.3GB/s. Next steps:

1. Figure out the CIFS shares/permissions -- more of what I want it do, rather than how to accomplish it--FreeNAS config seems fairly straightforward
   
2. Setup sabnzb, sickbeard etc.
   
3. Test the setup & copy my files over
4. Migrate my desktop's crashplan config over
5. Create some additional VMs (over iSCSI)
6. Maybe tune the OS/FS
   
7. I had some difficulty installing some additional packages under FreeBSD, so I need to figure that out; will clone the VM and play with package installer
   
8. Eventually get a 10GBe card for my main desktop

And a little about me: I'm a Systems Engineer/Administrator/DevOps (Insert other Buzzword 2.0 compliant moniker here) by trade, for about 20 years. Started off (first 6 years) on Solaris (2.6/2.7) mainly, and eventually went to Linux. So now I'm mainly a Linux guy. I've managed some thumper boxes off & on, but I'm relearning ZFS etc.

This is the first time setting up a discrete storage, for my home. Previously, I was just exporting shares from different HDs, and running VMs from my windows box on my home machine (i7-3930k six-core w/32gig ram & 20TB of storage with 8 3tb to 6tb drives). I've lost a few drives over the past couple of years. All that I've lost I didn't really care about (PVR'd tv shows, movie rips etc). But would be nice not to have to rip it again. Anything of importance, personal photos/videos, documents etc., is backed up to crashplan (which has saved my hide a few times).

Anyway, hopefully I'll be able to contribute, and maybe get answers here.

Thanks,
Imran

 

[DifferentStrokes]

Sounds like you will fit in here just fine. Welcome!

 

[Fuganater]

I was looking at that case before I decided to go with the Supermicro 846. How are the drive and system temps?

 

[Bidule0hm]

22 drives with a 550 W PSU is asking for problems, especially with a CS series Corsair PSU. If you do the math 22 * 1.75 = 38.5 A and that is 462 W on the 12 V rail, you need to add the drives consumption on the 5 V rail and all the others components of the NAS too.

I recommend at least a 650 W PSU and better quality than the Corsair CS series, like the SeaSonic X series for example ;)

 

[iniazi]

22 drives with a 550 W PSU is asking for problems, especially with a CS series Corsair PSU. If you do the math 22 * 1.75 = 38.5 A and that is 462 W on the 12 V rail, you need to add the drives consumption on the 5 V rail and all the others components of the NAS too.

I recommend at least a 650 W PSU and better quality than the Corsair CS series, like the SeaSonic X series for example ;)

Thanks for your concern. Since the System peaks only to 420W, I haven't worried too much about the PSU. the PSU says 43A on 12V (516W), 20A on 5V. Seems like its about right fit. I don't think the Motherboard/CPU are peaking on power draw on bootup (otherwise I'd have seen more than 420W?). Is it worth replacing? Although I may not have granular (or as high of frequency) of measures on the power draw on the wall (cheap P3 killawat). I looked at the power usage on a few bootups...

Looks like Microcenter & Frys don't carry Seasonic. I guess I'll have to look online...

 

[Bidule0hm]

Ah, I didn't saw that the server was already built. If it was me I'd change the PSU (and return it if you can or re-use it in a desktop PC for example) but it's your server, your data and your money. In pure theory this PSU should be enough (but with an E5, than many RAM, 2x M1015 and 2x SSDs I'm not even sure you're not over the 550 W) but there's at least two things to account for: first it's not recommended to load a PSU to its 100 % rating. Second it's a Corsair CS series PSU so it'll have crappy capacitors.

Actually I think the MB has a current peak at start-up too because the several embedded DC/DC converters have capacitors and I don't think they are kept charged, but I can be wrong on that one because I've not checked, it's a logic educated guess. But even is there's no peak there's still the normal consumption.

Yep, the problem with measuring power on the wall side of the PSU is that the PSU smooth the current peak a bit (if you know how they work you'll know why) and the meter will add integration over that so the power peak you see on the meter is not as high as it actually is.

You're not limited to SeaSonic of course. I suggested this brand because they are the best quality I know of and not too high priced.

 

[iniazi]

I was looking at that case before I decided to go with the Supermicro 846. How are the drive and system temps?

I only have 22 drives in there at the moment. The top row slot 1 and 4 drives for me are 37C, most other drives are 38C and 2 are 39C. Ambient temperature is 30C in this room. I'm using the three 120mm that came with the case, right behind the drives, and two 80mm that are on the back of the case. I plugged any backdrafting holes in the 120mm fan wall. Anyway, its kind of loud.

On the topic of loudness, I could replace the 80mm with quieter fans, but even when I disable them, the 120mm fan or the 60mm fan on the cpu cooler is loud. I replaced the stock supermicro fan on the cpu cooler (SNK-P0048AP4) with a 4k rpm 60mm fan, which is MUCH quieter (the old one had this loud whine), but its still quiet a bit of noise. Are there supermicro mobo compatible aftermarket coolers? their mounting holes on the 2011 slot don't look standard. Or are they? I had googled it when buying the motherboard, and I couldn't find any info... so I bought the supermicro one.

 

[Bidule0hm]

Is this the drive temps at idle? because if yes then it's too hot. Drives should never go beyond 40 °C. Cyberjock had a heat problem with his Norco case, see this thread for more info.

 

[iniazi]

Ah, I didn't saw that the server was already built. If it was me I'd change the PSU (and return it if you can or re-use it in a desktop PC for example) but it's your server, your data and your money. In pure theory this PSU should be enough (but with an E5, than many RAM, 2x M1015 and 2x SSDs I'm not even sure you're not over the 550 W) but there's at least two things to account for: first it's not recommended to load a PSU to its 100 % rating. Second it's a Corsair CS series PSU so it'll have crappy capacitors.

Yeah, I believe you. :) I'm not too crazy about Seasonic--had a bad one about 5 years ago (died after a few months), and I couldn't reach their support for warranty replacement (phone call, guy told me to email, and emails were unanswered), and I just threw the PSU away. Only other bad PSU I've had in 15 to 20 purchases of PSU was DOA. Newegg reviews don't look promising. But looking at other PSU reviews, its about the norm... Anyway, Amazon wanted $24 more, so ordred from newegg...

 

[iniazi]

Is this the drive temps at idle? because if yes then it's too hot. Drives should never go beyond 40 °C. Cyberjock had a heat problem with his Norco case, see this thread for more info.

Yeah at Idle. I read that thread a while ago when I was trying to figure out cooling. My server, sheet of normal and thicker (like double the thickness) paper sticks to the front of the server. And my ambient/room temperature is about 3C higher than Cyberjock's. I guess, maybe I'll purchase the corsair sp120 high performance edition ones (or look for other high static pressure and maybe little more cfm to boot). And might have to resort to other soundproofing to make it quieter..

 

[iniazi]

Ok, I've put in the Seasonic X 650 Gold. Its a nicer package than the corsair it replaced. I also realized that my desktop has a Seasonic 650 Platinum. I guess I had already forgiven Seasonic. :)
Now I need to figure out if replacing the 120mm fans that came with norco rpc-4224, would help. Maybe some high static pressure pwm controlled fans with little higher rpm max speed.

 

[Fuganater]

You may not know this but I saw a video explaining the drive bays on the 4224. Apparently each bay has a vent and they should be closed if there no drive there. Check to see if yours are all open or just the ones with HDDs are open.

If you alredy know this... ignore me. :)

 

[Bidule0hm]

@Fuganater the problem is that there's very little space between the drives so you need a high static air pressure to have a decent airflow and you can't have that with 120 mm fans (unless you want a fan who eat 50 W and is noisier than a 747...), see the link to cyberjock's thread I posted earlier ;)

 

[Fuganater]

I read that thread before but I TOTALLY missed where he talked about the trays in the first paragraph. But again, no one explicitly mentioned it here so I thought I would.

 

[Bidule0hm]

Yep, it's always a good idea to check for the obvious things first ;)

 

[iniazi]

I cut some EPE plastic foam in the shape of a harddrive to fit in to each of the two empty bays. I should probably do a smoke test to see if the flow is more hindered there than the adjacent harddrive bays.

But yeah, I'm just looking for fans that have more static pressure than whatever ships with the 4224 (and possibly ones I can ramp up the RPMs with as well). I think I'm going to wood enclosure to hide the server on top of a storage cabinet I have in my study. Which will probably dampen the sound a little (and I could add a little sound dampening design internal to it).

I replaced the heatsink/cooler with a Noctua NH-U9DX i4. Pretty nice. I guess I could have gotten the bigger one, but wasn't 100% sure if it would fit.