BUILD First Time Adventures in FreeNAS

[ran]

I figured I would document my first FreeNAS box build from pre-conception to implementation and through use. I hope that this will be a relevant resource for others (such as the really useful guides posted by others in the forums). I hope to update this first post with information and the final components used in my build.

Background: I started running OpenSolaris in 2008 or so utilizing ZFS to serve music, video, and pictures to various devices. The main goal, initially was to store my immense collection of pictures (i.e., my wife is a photo-taking-freak). Later I added VirtualBox and used a single mirrored pool to run and store virtual machines. Then added a third mirrored pool to store sensitive encrypted data.

Current storage system setup:
OS: 5.11 snv_151a
Mobo: Asus M2N-E
PSU: Corsair VX450W
CPU: AMD Athlon 64 X2 5000+ Brisbane black edition
MEM: 8GB Mushkin blue (non-ECC)
HDDs: 6 SATA II storage, 1 IDE system

An ancient video card, and a couple 120mm fans complete the build.

The ZFS Configuration is 3 mirrored pools.

1. 1TB (2 WD 1TB EARS HDDs)
2. 1TB (2 WD 1TB EARS HDDs)
3. 2TB (2 WD 2TB EARS HDDs)

I've had to replace only 3 failed drives in the last 6 years. Although I did upgrade from 500GB drives in the original setup.


No build is complete without an actual goal, or at least an illustration as to how the server will be used. I have a few needs:

• Serve photos, videos, music to DLNA enabled devices
• Allow upload of photos, videos, music and backups
• Not consume 1.21 GigaWatts of power
• Fit in a place that doesn't have a server rack
• Not make me want to kill myself or others because it's so loud
• Run a torrent client, like transmission
• Gigabit ethernet
• Enough CPU to do efficient encryption (can freebsd do ZFS encryption without a fuss?)

Desires:

• Run OpenVPN server in a virtualized environment (maybe jails? otherwise virtualbox)
• Hot swap drives without opening the case (never gonna happen, not enough space in my house for a rack, not if I want to keep my wife) Didn't get a hot swap. The chassis for that are a lot more expensive and I can't see swapping drives out that often.

FreeNAS wish/guess list:

• OS: FreeNAS 9.1
• Case: Antec P280 (not enough 3.5 drive bays :(), or maybe Fractal Define R4 Rosewill RSV-L4500 server chassis (15 drive spaces)
• Mobo: X10SL7-F (stole this idea from this guy's thread) or X10SLH-F (no SAS, need external card for this, like the IBM Server RAID M1015)
• PSU: Seasonic Platinum Fanless 520w, Seasonic X-650, Corsair AX850 (depending on need)
• CPU: E3-1285v3, 1275v3, 1270v3, 1280v3 (board has VGA; 80 is %30 more expensive)
• MEM: 32GB ECC (M391B1G73QH0-YK0 Samsung K4B4G0846Q-HYK0 - SuperMicro suggested)
• HDDS: 6 2TB drives to start (maybe 3TB? needs lots of research. WD Reds suggested.)
• Boot Device: 4GB thumb drive (decent, but need 2 for failsafe reasons, maybe 8GB)

FreeNAS Completed Components:

• Rosewill RSV-L4500 server chassis ($109, free shipping Amazon)

Random important stuff:

IBM ServeRAID M1015 is good if you need more ports and your board sucks. But it's pretty expensive unless you want to play eBay games. ($264.90 on Amazon) It also needs a flash to make it run like ZFS needs it to. There are other options for expansion like the $30 Dell SAS 5 PCIe cards. And, speaking of, does the M1015 or the Dell SAS 5 need the x16 slot? Or is x4 ok? I really don't even know what that means.

jgreco is an awesome poster but seems to push for server class hardware for your ZFS server. (How odd. ;)) Some of us can't live life the same way if we attempt to bring in a rack and add it to the home decor. :) There's some good information in this thread about suggestions for various pieces of hardware. It seems like SuperMicro, even after all these years, is still a good place to go. Bought my DLUAN board from them in a giant case like 16 years ago for the dual pentium II box I had. Oh the days of BeOS.

IPMI is kind of scary. Maybe I should stay away from that. It looked good right up until I saw there was a big glaring un-patched security hole. :(

Intel has a ton of CPUs now. E3 is probably where I need to be although they appear to all be limited to 32GB max memory. Is an E3 with graphics support slower/worse for this system than one without? Like E3-1275V2 (No Processor Graphics) vs E3-1275V3 (Processor Graphics). I need to figure that out.

TODO:

• Check on memory makers
• Check on other mobos
• Read more about E3s (sorry AMD, our affair is over) i* series isn't ECC. E5 dual proc systems, E3 single procs ... going E3.
• Can the raid controller on the supermicro board be flashed like the M1015? Yes Does it have to be? Probably for best performance.
• Find out about hard drives; maybe switch from WD, maybe pair up WDs and Seagates
• Find more cases!
• Find out if the SuperMicro boards come with all the cables you need
• Find a board without IPMI that fits the bill

Any suggestions, advice, criticisms, questions are welcome. I have only the barest idea of what I'm doing at this point.

*sweet, you can update posts, I'll be updating this as I proceed.

 

[Michael Wulff Nielsen]

Sounds to me like you have a solid build there.

The SuperMicro boards are popular because they run well and are well supported, so why spend your time digging around for more? Unless you (like me) live in a region where a SuperMicro board is almost "unobtanium".

Kingston Server ValueRam seems to be popular these days because it's good quality and very competitively priced.

As for the cpu you have the usual choice of the usual suspects: G2020, I3 4130(T) or some E3 based Xeon. It depends on your needs, especially transcoding with Plex takes quite a bit of power.

I don't know about the raid controller on the SuperMicro, but I suspect that it can be flashed.

WD Reds are popular around here because they work well and are considered reliable. In my opinion the 3TB drive is the sweet spot at the moment. A classic home combination is 6 x 3TB in RaidZ2.

I wouldn't worry about IPMI unless you are hanging the server directly to the internet without a firewall or anything.

 

[ran]

@Michael

I live in a place where we get deliveries, so I'm hoping that I can find a SuperMicro that will end up on my doorstep pretty easily. :) But I've not researched that part just yet; I take that for granted.

Thanks for the other bits of information. I appreciate the input!

As a slight aside, I had attempted to run Plex on my current configuration, and it was completely bogged down. I really blame Plex for this. My other DLNA server application runs ffmpeg without any problems and transcodes and streams to my DNLA clients without any lag at all. The DLNA server even streams music at the same time as the transcode, no problems. (Yeah, 100% CPU utilization just about, but that's what it's there for.) Base on that, I have to say that Plex is somehow inefficient. Since it cause so much lag for me, I didn't bother doing any research into why. I'm not even sure if it uses ffmpeg or some other application to perform transcoding. Whatever it uses, maybe I downloaded a pre-built package that wasn't optimized for my AMD processor. I just don't know. I guess I'll investigate that more when I get the FreeNAS built. I have multiple options to investigate, but I won't be able to actually tell a difference until I have the NAS up and running. I really like the interface for Plex, but I can live without it if I need to (as I do now).

 

[Yatti420]

Have you ran WDIDLE on the green drives? Oh and I love IPMI.. I'll probably never buya board without it now..

 

[ran]

Have you ran WDIDLE on the green drives? Oh and I love IPMI.. I'll probably never buya board without it now..

I have no idea what that is, so no, I have not.

At this point I'm trying to understand the SAS 2 drives cable needs. I thought they would be just sata cables. :(

 

[9C1 Newbee]

Oh and I love IPMI.. I'll probably never buya board without it now..

Me too! The server sits right next to me. But I don't have to worry about swapping keyboards, cd drives, or monitors. Very handy.

 

[9C1 Newbee]

I have no idea what that is, so no, I have not.

IMPI means you can use an internet browser(or a program) to connect to the motherboard. This allows you to use the monitor, keyboard, and mouse of the computer you are using. Rather than physically connecting a monitor, keyboard, and mouse to the ports on the server. In other words, I have a power cable and an Ethernet cable plugged into the server. Yet I can do a fresh software install from half way around the world if I wanted to.

 

[ran]

I should have been more specific. I have no idea what WDIDLE is. Unfortunately, I'm aware of what IMPI is, and that currently there's an unpatched security vulnerability in it. :(

 

[Yatti420]

Supermicro fixed alot of vulnerabilities in 3.15.. Im not sure about asrock etc but if you can find an IPMI webpage without using Java let me know.. You don't have to touch the server at all.. Even to boot media..

The WDIDLE3 tool allows you to change the intellipark setting (I think).. It defaults to 8 seconds on the greens which causes LCC to rack up very fast.. They need to be run on greens if you use the drives for media etc or you will notice poor performance and/or buffering etc while waiting for the drives as they constantly move the heads dealing with the 8 second timer..

 

[jgreco]

Unfortunately, I'm aware of what IMPI is, and that currently there's an unpatched security vulnerability in it. :(

Which begs the question: why are you running IPMI on an Internet-facing LAN segment?

 

[9C1 Newbee]

My server is safe behind my router. But that is good to know about the security issues.

 

[ran]

Which begs the question: why are you running IPMI on an Internet-facing LAN segment?

I wouldn't run it exposed to the internet but consider running software with known vulnerabilities bad practice and I try to avoid it if at all possible. But because the boards I want all have it, there's not much I can do about it. I'll have to look more into details about how the issues can be mitigated and I'll post back here once I do.

And since you're here, thanks for all your awesome posts! I gained a ton of information out of your insightful postings.

As an update, I've purchased the Rosewill RSV-L4500 server chassis because it was inexpensive, has a 15 drive capacity, and 6 120mm fans. I hope to not be disappointed with the need to fully disassemble to swap a drive but I just couldn't see spending another $200 or $400 more on something with less hot swap drives. In my experience with SATA drives I haven't had to replace all that many over the last few years, so I hope that trend continues.

I'm going to keep my old server up and running whilst I try and perfect the build, so I still have access to my data, and I'll try and play with raidz and mirroring to see if I can find the best performance. I've not read anything since my first setup years ago so I'll need to re-investigate that.

 

[scurrier]

Avoiding any IPMI "security issues" should be easy- you just leave the dedicated IPMP ethernet port on your motherboard unconnected.

 

[9C1 Newbee]

Avoiding any IPMI "security issues" should be easy- you just leave the dedicated IPMP ethernet port on your motherboard unconnected.

Thus defeating most of the benefit of IPMI. LOL

Correct me if I am wrong, but if you leave the IPMI behind the firewall and use VPN to connect, it's all good. Right?

 

[scurrier]

It wouldn't defeat "most of the benefit." It would defeat *all of the benefit.*

User "ran" was saying there's not much he could do about IPMI security issues because every board he wants has IPMI.

He could leave it unplugged, effectively making it like a non-IPMI motherboard.

 

[ran]

Leaving it unplugged removes the network layer attack surface; but also negates the use of it for remote management. I'll treat it like console cable access from yesteryear.

 

[DJABE]

Every vendor should support it's own hardware AND software. Period. Why so many of them fail to do so, I'm really unsure.
I've seen in practise vendor's requirement forcing usage of unsupported legacy Java runtime like NetApp and other servers, but as long as you're behind NAT/router i.e. inside a local area network there are no (big) security issues, except 'man in the middle' risk.
Completely different story if you need your server to be available from the outside world (Internet)! That's the point where we can define IPMI as insecure.

Background: I don't blame vendors for this. It's all Oracle's fault. Big time. They literally ruined what was once Java (3/3 of key people from Sun left "The Big Red" and now we see the result!). But I do blame vendors for not being up to date with supporting their own hardware AND software, in this case IPMI access. They should patch JRE version with BIOS/similiar patch and no security issue for customers! Yes, it's hard to keep up to date, but in today's hi-tech world you simply don't have a choice. Constant development and changes costs money, but it's the only right way. For the future every vendor should avoid using Java based solutions at all costs!

Perhaps ad-hoc workaround would be to forward IPMI port behind the NAT of a router (i.e. your.public.IP:9002 instead of default IPMI port). At least port scanners would be unable to see that you have an open IPMI service running.

 

[9C1 Newbee]

It wouldn't defeat "most of the benefit." It would defeat *all of the benefit.*

User "ran" was saying there's not much he could do about IPMI security issues because every board he wants has IPMI.

He could leave it unplugged, effectively making it like a non-IPMI motherboard.

Ah! Gotcha.

I didn't say "all" because you can still physically plug IPMI back in to perform a task and physically unplug it again. That might be better than swapping in a CD rom, monitor, mouse, etc.

 

[ran]

Server chassis arrived. Time to settle on the guts!

 

[ran]

Hmm, maybe I should start a new thread. If no one sees this I guess I will.

I am unclear on how to determine the size of the power supply I will need. Previously I just made sure that I was able to plug in all my drives. :) I'm pretty sure I have to make a better decisions about it this time. This system should have the following power-pulling devices:

• 6 120mm fans
• 2 80mm fans
• X10SL7-F mobo
• E3-1270v3 (and whatever fan comes with the stock CPU retail package)
• 1 USB stick
• 2 1GB ethernet links
• 32GB memory
• 15 2/3TB drives (to start I will have only a few drives that are 2TBs)

I was going to get the Corsair AX 850, but they don't make it anymore, or, rather, it's hard to come by and I assume since it's older they don't make it anymore. I thought about the Seasonic X-650 but I'm wondering if it will have enough power to push all 15 drives. I'm currently contemplating the Seasonic X-1050. But I don't know if that's too much. I looked around and tried to figure out how much the drives would need to spin up, but I couldn't find anything on western digitals website. I did find some other links that seem to show that each drive takes up about 15 watts of power. So that would mean it would take around 255 watts to spin up the drives. The 120mm fans should be something like 2 watts a piece. So that's another 12 watts.
I don't know what the mobo, memory, or CPU would take, nor any of the other little bits and bobs. I was hoping for a bit of advice on this. I'm sure someone on here will school me on what I need to learn to figure this part out.