BUILD 1U colocation considerations

[corv]

Hello folks,

I've been a happy FreeNAS user for a couple of years now. Originally a longtime Linux fan, I'm starting to lean more towards the BSDs. Once again it's time for me to start thinking about a new server.

I've checked the forums and found a wealth of new information (big thanks goes to Cyberjock) and yet there are still some questions I have before I start ordering any components.

I've been offered 1U (possibly 2U) of rack space and would like to consolidate a rented dedicated server and a couple of VPS into a server backed by ZFS.

I won't be able to drive to the server quickly so I'm looking for a reliable solution where components can be easily replaced by technicians at the facility if necessary. An efficient or low-power solution would also be preferable as long as performance isn't severely limited.

I've been looking at this Avoton SoC from Supermicro as one possible option:
http://www.supermicro.com/products/motherboard/Atom/X10/A1SAM-2750F.cfm
It's only marginally more expensive than the popular ASRock SoC and makes a more reliable impression.

Would it be underpowered for a couple of jails and a VM or two running several low-traffic websites (python, ruby, php, node.js, postgresql, mysql, mongodb), smtp & imap, vpn, rsync? I was planning to use 32GB ECC from the start.

Is an i3 a better choice or would a 4-core Avoton suffice?

Since the SoC has 6x SATA would it make more sense to run the jails and VMs on two mirrored SSDs and then use the remaining 4 SATA for HDDs with RAIDZ2? How difficult will it be to ensure that the mirrored SSDs running live OSes are properly backed up to the RAIDZ2 pool? Is there any TRIM support? Are the SSDs unnecessary and would it therefore be far more economical and simpler to use 6 HDDs with RAIDZ2?

Supermicro offers some cases with redundant PSUs but I've never used any. Since I've had bad experiences with defective PSUs I'd like to make a sensible choice even if it means spending some more money. Do the redundant PSUs need to be supported by the OS or motherboard? Do they kick in automatically when something is amiss?

As this is more than just a storage appliance, is there any reason to use vanilla FreeBSD instead of FreeNAS?

Any comments & criticism are highly appreciated.
Thanks for reading.

 

[L]

I really don't know how we lived before IPMI. It also gives you remote access to the hardware, if it is down and you need to reboot..

CPU is typically never an issue for zfs, though the webgui seems to use just a little more that raw freebsd. RAM RAM RAM...

I would definately recommend dual psu's if this is a remote location. I would also recommend disks and hardware that allow you to flash the LED on a drive that has failed so a technician can replace them. Most IPMI tools allow you to do that...

You don't need to worry about trim with freenas.

That's not all your questions but a couple...

 

[jgreco]

I really don't know how we lived before IPMI. It also gives you remote access to the hardware, if it is down and you need to reboot..

IPMI is nice but cannot be safely exposed to the Internet, so you need some firewalled OOB arrangement for that to be useful. We lived before that with KVMoIP.

CPU is typically never an issue for zfs, though the webgui seems to use just a little more that raw freebsd. RAM RAM RAM...

CPU is often an issue for ZFS. ZFS is big and piggy and ends up being substantially slower than a conventional fileserver, which we generally make up for by throwing much faster CPU's at the problem. In the mid-2000's I had some great Opteron 240EE's for datacenter use that could saturate dual gigE, but under ZFS the performance was pathetic.

I would definately recommend dual psu's if this is a remote location.

I would not necessarily recommend dual PSU's unless uptime is a key consideration. Dual PSU's especially in a 1U involves a lot of electronics and heat in a very small package, so individual modules are more likely to fail, and the PSU chassis itself can also fail, which becomes very difficult to replace on short notice (special order). The duals are also wicked expensive. Unless you plan to stock spares of the dual PSU components, buying two relatively less expensive nonredundant supplies and holding one as a spare is a more practical solution. This answer changes somewhat if you have redundant power circuits available at your facility (i.e. power from separate UPS's).

I would also recommend disks and hardware that allow you to flash the LED on a drive that has failed so a technician can replace them. Most IPMI tools allow you to do that...

I have no idea what IPMI tools allow you to flash the LED on a drive. Error LED's on a drive sled are typically handled on modern systems by the sideband/SGPIO/SES2 bus, though some of us have older kit that requires more hands-on approaches.

You don't need to worry about trim with freenas.

You certainly need to think about TRIM if you're using an SSD, but, fortunately, it is supported on FreeNAS and FreeBSD. However, you need to make sure it is set up appropriately.

That's not all your questions but a couple...

You can feel free to take many messages from the poster above and invert the answer to arrive at a more-correct answer.

If I sound annoyed, I am.

 

[jgreco]

Hello folks,

I've been a happy FreeNAS user for a couple of years now. Originally a longtime Linux fan, I'm starting to lean more towards the BSDs. Once again it's time for me to start thinking about a new server.

I've checked the forums and found a wealth of new information (big thanks goes to Cyberjock) and yet there are still some questions I have before I start ordering any components.

I've been offered 1U (possibly 2U) of rack space and would like to consolidate a rented dedicated server and a couple of VPS into a server backed by ZFS.

I won't be able to drive to the server quickly so I'm looking for a reliable solution where components can be easily replaced by technicians at the facility if necessary. An efficient or low-power solution would also be preferable as long as performance isn't severely limited.

I've been looking at this Avoton SoC from Supermicro as one possible option:
http://www.supermicro.com/products/motherboard/Atom/X10/A1SAM-2750F.cfm
It's only marginally more expensive than the popular ASRock SoC and makes a more reliable impression.

Would it be underpowered for a couple of jails and a VM or two running several low-traffic websites (python, ruby, php, node.js, postgresql, mysql, mongodb), smtp & imap, vpn, rsync? I was planning to use 32GB ECC from the start.

Is an i3 a better choice or would a 4-core Avoton suffice?

Since the SoC has 6x SATA would it make more sense to run the jails and VMs on two mirrored SSDs and then use the remaining 4 SATA for HDDs with RAIDZ2? How difficult will it be to ensure that the mirrored SSDs running live OSes are properly backed up to the RAIDZ2 pool? Is there any TRIM support? Are the SSDs unnecessary and would it therefore be far more economical and simpler to use 6 HDDs with RAIDZ2?

Supermicro offers some cases with redundant PSUs but I've never used any. Since I've had bad experiences with defective PSUs I'd like to make a sensible choice even if it means spending some more money. Do the redundant PSUs need to be supported by the OS or motherboard? Do they kick in automatically when something is amiss?

As this is more than just a storage appliance, is there any reason to use vanilla FreeBSD instead of FreeNAS?

Any comments & criticism are highly appreciated.
Thanks for reading.

So in answer to your message:

If you can get 2U of space, the best option could be to build two 1U servers and have them in some sort of redundancy, or even just have one sitting there as cold spare. I hate doing anything with magic hardware (defined as anything we don't have in shop or couldn't go to a local distributor for). Having a two week downtime because you had to special-order a part and it is on its way over from China sucks. Also: smart hands at a facility are generally not all that smart and can rapidly turn frustrating and expensive when the goal is doing something that involves ripping a server apart. Most of our gear lives 800 miles away from where I sit, so I know a bit about that.

The C2750 has a lot of oomph, more than half a low end Xeon. If by VPS you mean jails, then you're probably all good on that front. If you're planning to run actual VM's, that I don't know - I guess VirtualBox may be available as of 9.2.1.6 but I don't know how heavy it is on the system. Fortunately with 8 cores you can experiment without too much risk.

A C2750 with 32GB should be fine for moderate traffic. I don't suggest an i3 because most lack ECC, and I don't suggest the C2550 because FreeNAS will eat a few cores all by itself. The C2750 is known to have extra room beyond the demands of the system.

Two mirrored SSD's are probably a great idea for your jails/VM's. But be sure to buy a LOT more space than you need. With FreeNAS, the SSD pool will still be ZFS, and ZFS gets cranky and ill-mannered with VM data once it passes maybe 60% capacity, due to fragmentation. SSD makes that better but only by so much. Backing up the VM's could be as simple as an rsync cron job, but be willing to spend a little time on making sure it works right. TRIM support is there but probably needs to be manually enabled. VM storage on RAIDZ2 sucks awfully; RAIDZ2 is simply not good at that.

The redundant PSU's are automatic, and if you have a Supermicro motherboard, you may have a PMbus (power management bus) which provides additional information, usually including things like current watt draw, back to the motherboard. This data will be graphed by the IPMI as well. Obviously both the PSU and the MB have to support PMbus for that. Redundant power supplies are both active, it is not a failover or standby situation. This is nice in that it means each one is working half as hard under normal circumstances, but also means that when you need them the most, they're suddenly being stressed 100% more.

There is a compelling case here to go and use FreeBSD, and that's that FreeNAS isn't really designed for exposure to the Internet. If you plan to have a firewall appliance between your FreeNAS and the Internet, then that's great, otherwise, be aware that FreeNAS isn't designed for direct exposure. You can still use it for the intended application, but proper safe deployment becomes ... tricky. Not for beginners, really. FreeBSD can be secured against the Internet fairly easily but it is harder to harden an appliance like FreeNAS. With that said, most of what's been said here can be applied either way.

 

[Ericloewe]

A C2750 with 32GB should be fine for moderate traffic. I don't suggest an i3 because most lack ECC, and I don't suggest the C2550 because FreeNAS will eat a few cores all by itself. The C2750 is known to have extra room beyond the demands of the system.

All Desktop Ivy Bridge and Haswell i3s support ECC, so that's not a problem, given an appropriate Server motherboard. That said, it might be a bit slow in this case, given the web server duties, so a Xeon or C2750 are probably the better ideas.

 

[marbus90]

in case you want to pack many HDDs into 1U, lookie here:
http://www.asrockrack.com/general/productdetail.asp?Model=1U12LX-14S
also available without redundant PSU:
http://www.asrockrack.com/general/productdetail.asp?Model=1U12L2SW-14S
or if you want to stay low-power:
http://www.supermicro.com/products/system/1U/5018/SSG-5018A-AR12L.cfm

keep in mind that as soon as you want to upgrade the Avoton to 64GB RAM, it's about the same cost or cheaper to buy an E5-1620 v3 w/ 4x 16GB DDR4.

in case if you still want to use freenas: look if your hoster provides an additional private interface with VPN access. I'd use that with lan1, where usually also the IPMI is shared. lan2 would then be used for all WAN traffic.

 

[cyberjock]

CPU is typically never an issue for zfs, though the webgui seems to use just a little more that raw freebsd. RAM RAM RAM...

It is, but only if you aren't being silly and trying to use a 6+ year old CPU. IMO the Pentium G2020/G3220 is kind of "bottom of the barrel" and as long as you don't plan to do dedup (which is a MASSIVE CPU workload at large scale) people should be okay.

I would definately recommend dual psu's if this is a remote location. I would also recommend disks and hardware that allow you to flash the LED on a drive that has failed so a technician can replace them. Most IPMI tools allow you to do that...

I'm not a fan of dual PSUs, even more so in a 1U. As for flashing disks, I've never ever seen that for LED disks. In fact, flashing LEDs on a drive bay is a TrueNAS only feature because there are so many implementations and such that it's basically impossible to make everything work properly on FreeNAS.

 

[corv]

Many thanks to everyone that commented, especially to jgreco for the detailed replies. There's a wealth of information here for me to consider.

My summary from the comments thus far:

- IPMI is practically a must for a remote server but it needs to be secured properly.
- Dual PSUs are probably more trouble than they are worth in a 1U.
- C2750/C2758 has sufficient power for ZFS and light server duties.
- i3's are dual-core only but support ECC.
- An Atom SoC with 64GB of RAM is disproportionately expensive.
- Mirrored SSDs are great for Jails and VMs.
- RAIDZ2 is a bad option for VMs.
- Over-provision space to make sure that ZFS performs well.
- FreeNAS is more involved to secure than FreeBSD.
- It's important to pick components that are readily available.
- An identical 1U server can prevent significant downtime.

In light of this I might consider the ASRock C2750D4I since it's cheaper, easier to get a hold of and likely to be supported in the future.
How does the IPMI perform on the ASRock? Is the Marvell controller still problematic?

@marbus90, I'm unsure whether the LSI controllers are properly supported by FreeBSD. I won't need quite that many drives and I prefer hot-swap bays but thanks for the hint.

I like the idea of a spare server but doubling the price is quite a proposition when it's unclear if automatic failover is actually possible for most of the services running on the server. If anyone can point me towards a jail/container or VM based solution that can handle a host failure I'd be very interested.

Is RAIDZ2 on HDDs going to provide acceptable performance when only using Jails and not VMs?

Thanks again

 

[marbus90]

The LSI controllers are the go-to hardware for FreeNAS. The Marvel SATA controllers on the C2750D4I aren't playing nice with FreeNAS/BSD, so with that board you are still limited to 6 drives.

6 drives in 1U with front-side hot-swap require 2.5" drives. Other than that you'd need the 12drive solutions. the Supermicro one comes afair with a cable management arm and sliding rails, so the disks could be swapped without powering down the server. You would just need to pull the server from the rack.
2U would lead you to the 08/15 8-12drive chassis with a 2U PSU, which is pretty much standard stuff.

 

[corv]

The LSI controllers are the go-to hardware for FreeNAS. The Marvel SATA controllers on the C2750D4I aren't playing nice with FreeNAS/BSD, so with that board you are still limited to 6 drives.

6 drives in 1U with front-side hot-swap require 2.5" drives. Other than that you'd need the 12drive solutions. the Supermicro one comes afair with a cable management arm and sliding rails, so the disks could be swapped without powering down the server. You would just need to pull the server from the rack.
2U would lead you to the 08/15 8-12drive chassis with a 2U PSU, which is pretty much standard stuff.

I was eying this combination: http://www.supermicro.com/products/system/1U/5018/SYS-5018A-MHN4.cfm

I'm fine with 4 large HDDs and the SSDs are probably not absolutely necessary if they don't fit.

I've checked both LSI controllers (ASRock and Supermicro option) and it appears you are right, they are supported by FreeBSD.

 

[jgreco]

In light of this I might consider the ASRock C2750D4I since it's cheaper, easier to get a hold of and likely to be supported in the future.

I can probably get you access to try out the IPMI on a C2750 if it is an issue for you.

 

[corv]

I can probably get you access to try out the IPMI on a C2750 if it is an issue for you.

That's a very generous offer. I don't want to be a hassle. My concern is simply whether it's compatible with Firefox on OS X. I'm under the impression it's Java based? Some applets are finicky even if they are platform-agnostic...

Two remaining questions are:

1. Is there a Jail based solution that can handle a host failure and resume on a spare server? If not Jail based, VM or container?

I'm aware that redundancy can be added to smtp and dns services easily but some websites have sessions and state to preserve.

2. Is RAIDZ2 on HDDs going to provide acceptable performance when only using Jails and not VMs?

 

[jgreco]

Well, I have a C2750 sitting on the shop bench. I don't have Firefox on OS X so I can't speak to the compatibility there. I can set it up for you to play with it but I don't have time to do the playing myself right now.

As for your other questions, usually that's an architectural question as to what you're deploying and how. Running redundant? A web server with static web pages might need no more than two DNS A addresses. A complex website with databases might require some complex MySQL replication and a loadbalancer front end. Running as warm standby? That's more challenging in some ways. What's responsible for propagating the data from the failed server? Merely keeping a spare server racked and ready could mean that handling a catastrophic failure is a matter of pulling the disks from one unit and putting them into the backup unit. Anything more complicated than that requires some design and there are hundreds of options as to how you could do it.

Performance-wise, it is hard to predict what RAIDZ2 will do. If you are accessing files (as in like a jail) that is somewhat easier on ZFS than if you've got virtual disks (as in VirtualBox etc) because ZFS is handling the file allocation and is able to be smarter about it. The virtual disk situation (and/or large complex databases) kind of hurt ZFS because of the implications of Copy-on-Write. So pure jails on RAIDZ2 are better than VM's on RAIDZ2 by a good amount, but as to whether it is acceptable? Hate to say it, but you kinda gotta try it.

 

[corv]

Well, I have a C2750 sitting on the shop bench. I don't have Firefox on OS X so I can't speak to the compatibility there. I can set it up for you to play with it but I don't have time to do the playing myself right now.

If it's not too much to ask I'd have time to look at it tomorrow. Is it the ASRock one?

As for your other questions, usually that's an architectural question as to what you're deploying and how. Running redundant? A web server with static web pages might need no more than two DNS A addresses. A complex website with databases might require some complex MySQL replication and a loadbalancer front end. Running as warm standby? That's more challenging in some ways. What's responsible for propagating the data from the failed server? Merely keeping a spare server racked and ready could mean that handling a catastrophic failure is a matter of pulling the disks from one unit and putting them into the backup unit. Anything more complicated than that requires some design and there are hundreds of options as to how you could do it.

The key word here is complex. It would be very time consuming and difficult to make sure all of the databases are replicating correctly and even then it's not guaranteed that users won't be logged out or lose some data. I know that at least one of the applications can't be load balanced properly.

I was hoping it's possible to replicate the jails continuously to an identical machine and take over the failed server's ip once it's unreachable. Certainly not transparent to the end user but preferable to weeks of downtime. I'm not familiar enough with FreeBSD's ecosystem to know if there is some off-the-shelf solution.

Performance-wise, it is hard to predict what RAIDZ2 will do. If you are accessing files (as in like a jail) that is somewhat easier on ZFS than if you've got virtual disks (as in VirtualBox etc) because ZFS is handling the file allocation and is able to be smarter about it. The virtual disk situation (and/or large complex databases) kind of hurt ZFS because of the implications of Copy-on-Write. So pure jails on RAIDZ2 are better than VM's on RAIDZ2 by a good amount, but as to whether it is acceptable? Hate to say it, but you kinda gotta try it.

I would prefer using Jails to running VirtualBox simply because of the overhead involved and you've also illustrated why it would thrash the disk. Unfortunately some applications might require 64bit Linux and I'm assuming Linux Jails are still 32bit only?

 

[marbus90]

Linux jails in FreeNAS aren't supported come 9.3 and aren't exactly a dream. Das Debian-Nazi me tried to run some stuff via a Debian jail - no luck.

Weeks of downtime should be in the past millenia. That Supermicro is built from parts which should be readily available: 1U PSU, standard ATX mobo, standard ECC DIMMs... only the HDD-backplane would be an issue, but then again even that should be available quite fast.

Since you want/need Linux jails, you need another solution. Docker is probably a considerable idea. Also raid10 can deliver 4x the read speed of single disks, where raidz2 would be stuck at a factor of 1x per vdev. Of course raid10 isn't as safe or resilient as raidz2- but you still have a 2 out of 3 chance to survive a 2disk loss without having the pool killed. You should have backups anyway.

In terms of minimizing downtime you should prefer hosted servers. If something fails, they have the spare part usually at hand.

 

[corv]

Linux jails in FreeNAS aren't supported come 9.3 and aren't exactly a dream. Das Debian-Nazi me tried to run some stuff via a Debian jail - no luck.

I've done some research on visualization options in FreeBSD and it looks like Bhyve is a new contender on the block. It's tightly integrated with the OS and ZFS and can use zvols directly.

Weeks of downtime should be in the past millenia. That Supermicro is built from parts which should be readily available: 1U PSU, standard ATX mobo, standard ECC DIMMs... only the HDD-backplane would be an issue, but then again even that should be available quite fast.

If the colocation was in another city it wouldn't pose much of a problem but it's in another country! I wonder how much longer server components are going to last compared to consumer parts.

Since you want/need Linux jails, you need another solution. Docker is probably a considerable idea. Also raid10 can deliver 4x the read speed of single disks, where raidz2 would be stuck at a factor of 1x per vdev. Of course raid10 isn't as safe or resilient as raidz2- but you still have a 2 out of 3 chance to survive a 2disk loss without having the pool killed. You should have backups anyway.

I'm going to attempt to run everything with FreeBSD, a hypervisor running Linux is just a fallback option. Docker seems to be all the rage nowadays but I don't really see the value proposition over Jails. That being said it might be the best option on Linux. I'll make sure to have backups on another ZFS.

In terms of minimizing downtime you should prefer hosted servers. If something fails, they have the spare part usually at hand.

This is probably the main takeaway from your post. Although the single 1U rack is cheaper over the long run a single server is not going to live up to the reliability I need under my circumstances. That being said, I've found it is possible to make hot backups of jails in order to send them to a hot spare. It's not an elegant solution however and very uneconomical.

Unfortunately reasonably priced dedicated servers are not going to offer KVMoIP which poses a problem:
How am I going to unlock the encrypted root partition?

My debian box uses a two-stage boot process in which ssh listens for the passphrase but I've never made such a change to FreeBSD's boot sequence.

 

[jgreco]

I've done some research on visualization options in FreeBSD and it looks like Bhyve is a new contender on the block. It's tightly integrated with the OS and ZFS and can use zvols directly.

Which'll be nice for FreeNAS when it eventually becomes available for FreeBSD 10.

If the colocation was in another city it wouldn't pose much of a problem but it's in another country! I wonder how much longer server components are going to last compared to consumer parts.

"Server" parts don't differ significantly from high quality consumer parts, in terms of reliability. A quality consumer or prosumer board will be made out of long life components and may even carry ratings for harsh environments. The big deal is that both are intended to run 100% duty cycle for at least 5 years. This differs significantly from the cheap low end motherboards that are made for maybe a 30% duty cycle, the cheapest components that can be found (including Shenzhen back alley caps with the magic stolen electrolyte formula). Other than that, server boards differ from quality prosumer gear mainly in selection of components and features, and they're typically priced more reasonably than the "high end."

Anecdotally I've got a quartet of Supermicro P4SCi's out in Ashburn that are about to turn ten years old. One is overheating and cycling down the CPU clock speed as a result (failed fan), another has had a total IDE hard drive failure, the other two are dandy. The mainboards keep running and have shown no sign of problems (I don't consider fan failure to be a MB issue).

 

[corv]

Anecdotally I've got a quartet of Supermicro P4SCi's out in Ashburn that are about to turn ten years old. One is overheating and cycling down the CPU clock speed as a result (failed fan), another has had a total IDE hard drive failure, the other two are dandy. The mainboards keep running and have shown no sign of problems (I don't consider fan failure to be a MB issue).

That's amazing! I've had consumer components that weren't running continuously break far sooner than that. Come to think of it, on/off cycling is probably a major culprit.

I still need to figure out how to start networking and a small ssh daemon from an unencrypted boot partition that waits for user input to decrypt the root partition and continue booting.

 

[jgreco]

It's also a factor in my pendulum swing back to Intel systems, I kept having server grade AMD boards take various craps-in-my-rack which isn't real fun when the rack is 800 miles away and it isn't clear what fried. Remote hands are like "uh the lights are on". Niiice. I've actually got some burnt AMD Athlon's from the days of the Tyan S2462, which we had maybe as many as a dozen of. Let me see if I can get a picture, they're funny as hell.


Here, this one, it was just all gone. There were bits stuck to the heatsink.



But this, this, impressively, continued to run, but not well.

 

[corv]

It spontaneously combusted! I'm sure the Stuxnet team would like to know how to do that, haha.