So you want some hardware suggestions.

[cyberjock]

Moved Klox's post to his own thread where it belongs... it can be found at http://forums.freenas.org/threads/i-want-my-build-checked.15305/

 

[Alice Wonder]

First of all, thank you for writing this.
I still need to read all of the responses.

Some areas where I might have some disagreement -

It says to use server boards because ``it wasn't truly designed to run 24/7''

I've been building my own PCs since 1999 always using consumer boards. I run Linux on them and keep them powered on 24/7 and the board failing has never been a problem. I only use Asus boards.

Other points in the argument for a server board probably are corrent, notable the onboard NIC. Until my most recent build I always used a PCI card because it performed notably better than the onboard NIC. My latest desktop I am using the onboard NIC and performance is fine.

But since ECC does seem to really be necessary to safely run FreeNAS server board does seem to be the way to go, most consumer boards do not have ECC, so that point is probably moot.

The other issue I have is the suggestion to use thumb drives. Um, if you shouldn't use consumer boards because they aren't "designed" for 24/7 use, then why suggest thumb drives which are not designed to be primary boot devices?

I will be using an SSD, quality SSD drives are not that expensive now and it doesn't have to be just for running the operating system. For example, I do a lot with LaTeX and I assume that I can use the unused space on the SSD to run a revision control server for my various projects (then backing up the revision control system to the spin drives). As the NAS is going to be on 24/7 anyway, might as well utilize it for LAN services that need to be 24/7 but don't require a box with a head and keyboard. No need to run a second server, and that would allow me turn my Linux desktop off when I'm not using it saving energy (currently I run SVN on my Linux desktop so I can work from my Linux laptop and keep it all in sync)

It just seems to me that an SSD designed to be used for installing an operating system is going to be less problematic than a USB thumb drive that really is designed to be used for a sneakernet.

I was unable to find reliable failure rates of USB thumb drives, the data I found included junk brands that I wouldn't even use for a sneakernet so I don't know if my concern is valid. But since I plan to utilize the FreeBSD system for other things beyond just NAS it definitely makes sense to use an SSD over a thumb drive.

 

[cyberjock]

Well, if you read up on desktop boards, no manufacturer rates them for 24x7 usage. You might be able to get 24x7 usage out of them, but that's not what the manufacturer intended them be used for. Compare that to server boards that are often designed for 24x7 use at 100% loading the whole time and guess which one is likely to be a better choice? But the argument is kind of pointless as you said since ECC is pretty much required.

USB sticks aren't rated for any time period. Unlike motherboards they don't suffer from heat problems, draw large amounts of power, have regulators, etc. Far less things to go wrong compared to limited write cycles of the memory. The manufacturer rates them for total write cycles and that's pretty much it. Since FreeNAS doesn't do much writing, they should last a very very long time. FreeNAS isn't alone with this thought process either. ESXi, pfsense, and many other OSes use a USB stick in the same way.

Some people have had good luck with USB sticks(I have had zero failures in FreeNAS servers I've built). Technically, an SSD is more reliable, but they are also significantly more expensive. Do you really think you'll go through enough 4GB USB sticks in the coming years to make an SSD a smarter choice? Personally, I use USB and I don't even question the reliability. Others might use an SSD just to avoid the potential. The choice is yours to make, just don't be surprised if the SSD install doesn't go quite to plan. Plenty of new users had issues with SSD installation and later saw the light and went USB.

The only brands of USB recommended are name brands. Samsung, Corsair, Kingston, Mushkin, etc. Again, plenty of new users use no-name sticks and spend days(or sometimes weeks) before they finally spend the $8 for a real USB stick. It makes me wonder how they do anything in life when they'll spend that much time to save themselves $8.

 

[jgreco]

It says to use server boards because ``it wasn't truly designed to run 24/7''

I've been building my own PCs since 1999 always using consumer boards. I run Linux on them and keep them powered on 24/7 and the board failing has never been a problem. I only use Asus boards.

We were an Asus shop through much of the '90's. The products are better than average, and tend towards prosumer rather than consumer grade. However, many of them are still hobbled with useless hardware like audio, which adds to the energy consumption, hardware tradeoffs such as Realtek ethernets, which eats an extra slot when you need to "fix" it by adding in an Intel card, inadequate-for-server-use video that is designed for desktop rather than server use (ideally you want the cheapest, lowest-watt video solution), and no remote management (IPMI) capability. Further, the support for the "finer details" such as VT-d may be poor to craptacular to nonexistent, and oftentimes you'll run into situations like "That PCI-e slot only supports video cards" for no discernable reason. That's my short list of complaints about prosumer boards.

Do you want my list of complaints about consumer grade boards? I shouldn't, as I have better ways to spend my time. Capacitors sourced in a Shenzhen back alley, other marginal components, outsourced design, inability to support anything but current generation Windows users because they didn't retain the BIOS staff, etc. Consumer grade boards are designed as part of an ecosystem that emphasizes price before all else. I am not actually going to go on. I'm just going to assume you get the point.

The other issue I have is the suggestion to use thumb drives. Um, if you shouldn't use consumer boards because they aren't "designed" for 24/7 use, then why suggest thumb drives which are not designed to be primary boot devices?

I don't actually recall recommending that. It seems rather unlikely because I'm actually one of the minority around here who is disenchanted with thumb drives. I do advocate the "separate boot device" approach, though. It closely mirrors some of our own local design paradigms.

I will be using an SSD, quality SSD drives are not that expensive now and it doesn't have to be just for running the operating system. For example, I do a lot with LaTeX and I assume that I can use the unused space on the SSD to run a revision control server for my various projects (then backing up the revision control system to the spin drives). As the NAS is going to be on 24/7 anyway, might as well utilize it for LAN services that need to be 24/7 but don't require a box with a head and keyboard. No need to run a second server, and that would allow me turn my Linux desktop off when I'm not using it saving energy (currently I run SVN on my Linux desktop so I can work from my Linux laptop and keep it all in sync)

It just seems to me that an SSD designed to be used for installing an operating system is going to be less problematic than a USB thumb drive that really is designed to be used for a sneakernet.

I was unable to find reliable failure rates of USB thumb drives, the data I found included junk brands that I wouldn't even use for a sneakernet so I don't know if my concern is valid. But since I plan to utilize the FreeBSD system for other things beyond just NAS it definitely makes sense to use an SSD over a thumb drive.

The boot device cannot be (easily) used for your own purposes; it is meant to be dedicated to FreeNAS. It is one of the design decisions that the authors made.

 

[jgreco]

Oh and the cost differential between the prosumer and serve grade boards is usually next to nothing, with the server boards often being less expensive. It just seems like a no-brainer to me.

 

[John M. Długosz]

The other issue I have is the suggestion to use thumb drives. Um, if you shouldn't use consumer boards because they aren't "designed" for 24/7 use, then why suggest thumb drives which are not designed to be primary boot devices?

I suppose it's because the drive is not pounded on. It's only written to occasionally. Also, most trouble with thumb drives (other than really shoddy ones) are due to transport and handling, and if it's just sitting there in the port it doesn't get stressed or abused.

I will be using an SSD, quality SSD drives are not that expensive now and it doesn't have to be just for running the operating system. For example, I do a lot with LaTeX and I assume that I can use the unused space on the SSD to run a revision control server for my various projects (then backing up the revision control system to the spin drives). As the NAS is going to be on 24/7 anyway, might as well utilize it for LAN services that need to be 24/7 but don't require a box with a head and keyboard. No need to run a second server, and that would allow me turn my Linux desktop off when I'm not using it saving energy (currently I run SVN on my Linux desktop so I can work from my Linux laptop and keep it all in sync)

I think you can, but might take a little fiddling. When I made my boot drive, I noticed that it wrote 4 partitions, all marked as primary, with the rest unused. You cannot create more than 4 partitions, so the unused space can't be used unless you change around the others to use extended rather than primary partitions at least for the last. A partition editor will do that for you. But will FreeNAS still like it?

Perhaps installing FreeNAS the other way — not as embedded — will work just fine for that. I don't recall what it's called, and it's hardly ever used or discussed.

As for using the same hardware for all servers: For a home environment I like that idea too. The SVN server actually makes sense on a NAS even if you don't take that point of view. You can install that in a "Jail" I believe, if it has a FreeBSD implementation. I was contemplating a git server in a similar way.

Other things might not work so well, like a domain controller or LDAP. They need to be present already when the NAS software starts up!

 

[cyberjock]

Perhaps installing FreeNAS the other way — not as embedded — will work just fine for that. I don't recall what it's called, and it's hardly ever used or discussed.

There is no "other way". At least, not in the context you are describing. The ISO installer created 2GBs worth of partitions(4 total) and the remainder of the disk cannot be used. If you use the image writing option to install FreeNAS, you'll end up with a disk with the same 2GBs worth of partitions(4 total) and the remainder of the disk still cannot be used.

There's a reason why it's "hardly ever used or discussed". In fact, its not discussed at all. Because it doesn't exist.

Now if you wanted to compile your own copy of FreeNAS and change the partition size and do other crazy things that might cause problems later on with FreeNAS upgrades that's totally your prerogative and your tech support call. But expect that the second you star talking about your weird setup you'll be on your own to support it if any problems develop. Unfortunately, just like always, even if you change the partition size you'll still have problems with FreeNAS updates wiping out all of your customizations and software that you put on the FreeNAS partition. So unless you plan to make your changes with every FreeNAS update you should just stick to the jail for installing software.

 

[Alice Wonder]

Ouch. I hate to say it but that sounds like a design flaw. I guess I'm too use to logical volume management in Linux where it is easy to extend partitions in unused space as needed. Basically if it is a boot volume, a single small boot partition is created for the booting and the rest of the space is a second partitition controlled by the logical volume management and you can easily extend, shrink, mirror, etc. as needed w/o even needing to reboot the system. You can even have filesystems spanning multiple physical disks if you need. It's nice, but apparently not the way FreeNAS works. Ah well.

 

[jgreco]

Ouch. I hate to say it but that sounds like a design flaw.

No, it really isn't. It's a storage appliance, a NAS. It is not intended to be a general purpose UNIX server. The nasty paradigms that most UNIX admins learn in the course of administering servers are bad enough on a UNIX box. How many times have you logged onto a box, only to find various services spammed all over the filesystem? "Oh yeah I've got a web server on there to support webmail alongside my Postfix install and SpamAssassin, and I had to install this plethora of dependencies to get it to all work". Policy around here changed many years ago to compartmentalize the UNIX platform and to treat it as a platform for running applications. OS is stored on one disk. Applications on another disk (or disks), and mounted under a root mountpoint. The OS can be trivially upgraded by just wiping the OS disk and reloading. The application tree can be easily updated too because you can SEE what all the dependencies are; the source code is in /appname/src. Very clean. In comparison: I recently ran into a Web server that had at least three distinct Apache installs, two old and unused ones, because cleaning up can be kind of tricky when you don't even know where "make install" spammed stuff. And I couldn't just ask the guy who set it up, because he's DEAD.

FreeNAS takes that a step further. The whole idea of the thing is to be an appliance. It has no business writing its stuff on the storage disks. It is entirely compartmentalized and able to be independently upgraded, downgraded, whatever. It happens to be a little inconvenient on the PC architecture where options for small bootable mass storage devices are kind of poor. I find a nice hardware RAID controller like the BR10i and a pair of cheap small SSD's in RAID1 to be suitable.

It is an intelligent architectural decision. The concept will be familiar to many administrators who have worked with an appliance style device. The concept is completely foreign to many UNIX sysadmins though.

 

[Alice Wonder]

How many times have you logged onto a box, only to find various services spammed all over the filesystem?

I don't. I'm RHEL/CentOS and virtually all software installs via RPM where the rpmlint utility points out violation in the FHS guidelines. Those packages are fixed before installed.

Non RPM packaged software goes in /usr/local/product/version e.g. /usr/local/texlive/2012 or /usr/local/java/jdk1.7.0_40

Anything compiled from source is built with RPM in a clean mock build environment.

I guess it is just a philosophical difference, and is off topic, so I'll refrain from defending it.
I'll use FreeNAS as it is intended, rather than try to bend it. Maybe I'll like that philosophy better.

 

[vegaman]

I guess it is just a philosophical difference, and is off topic, so I'll refrain from defending it.
I'll use FreeNAS as it is intended, rather than try to bend it. Maybe I'll like that philosophy better.

I'm sure you will like it better like that It also has the advantage that if your USB stick (or any other device you boot from) dies it's not a big problem. You just replace it and restore your config.

On a similar topic I have started doing similar things on Linux servers myself. Because of the software being deployed that's being done through virtualenv - which is similar to Jails/Containers but for python environments. This makes administration much easier - you can move packages between servers, run different versions side by side and not have to worry about the mess of dependencies.

 

[jgreco]

How many times have you logged onto a box, only to find various services spammed all over the filesystem?

I don't. I'm RHEL/CentOS and virtually all software installs via RPM where the rpmlint utility points out violation in the FHS guidelines. Those packages are fixed before installed.

Yeah, well, that doesn't really do much to address package interdependencies, or that in many environments, RPM isn't up to the task of deploying intricate service platforms in a consistent and reproducible manner.

Because of the software being deployed that's being done through virtualenv - which is similar to Jails/Containers but for python environments. This makes administration much easier - you can move packages between servers, run different versions side by side and not have to worry about the mess of dependencies.

There's been a lot of interest in containers lately (Docker, etc). But really a lot of the concepts are much older. UNIX has had chroot for a long time. I think the main impediment was that they were uniformly hard to use, and that you basically had to design and build your own systems. We did that years ago but sometimes it's a royal ***** to maintain. Many people start with a "full container" (full jail environment) and then go about putting in extra stuff, but the real magic of a jail is an environment without /bin/sh hanging around. We start with an empty container and then design a script that puts just enough stuff in to let it work right. That's helpful for Internet-exposed services..

 

[Alice Wonder]

I'm sure you will like it better like that :) It also has the advantage that if your USB stick (or any other device you boot from) dies it's not a big problem. You just replace it and restore your config.

On a similar topic I have started doing similar things on Linux servers myself. Because of the software being deployed that's being done through virtualenv - which is similar to Jails/Containers but for python environments. This makes administration much easier - you can move packages between servers, run different versions side by side and not have to worry about the mess of dependencies.

Oh I like virtual machines, but I don't like the constraint of not being able to easily add something useful to me to an existing install, like adding SVN to a NAS server.

 

[cyberjock]

Oh I like virtual machines, but I don't like the constraint of not being able to easily add something useful to me to an existing install, like adding SVN to a NAS server.

That's why the jails exist. And they are kept separate from the OS to boot!(pun not intended) So now when the OS dies in a ball of fire you just reinstall and your jails will automagically work!

Aside from not being able to run things on the FreeNAS USB that you might have to(like hardware controlling stuff) I don't see a downside.

 

[Alice Wonder]

Data goes in /srv which gets backed up and can be separate, conf files go in /etc which gets backed up and can be separate, so restoring after an OS crash really isn't difficult. It's just a different way of doing things.

 

[jgreco]

So... a survivability strategy for when the OS needs replacement. Great. So we agree. On FreeNAS, your "/srv" is "/mnt/poolname". This is off-topic for this thread at this point, so if you wish to discuss further, please open a new thread.

 

[John M. Długosz]

There is no "other way". At least, not in the context you are describing.

I was thinking of “full” vs “embedded”. FreeNAS no longer supports the non-embedded installer?

 

[pirateghost]

I was thinking of “full” vs “embedded”. FreeNAS no longer supports the non-embedded installer?

That was a Pre-FreeNAS 8.x world, also known as FreeNAS .7 or now referred to by its new project name NAS4Free. I am pretty sure I have never seen that option in FreeNAS 8.x +

 

[underbara]

Hello, this is my first post. Nice to meet you all.
If I choose the recommended Supermicro board and I only need 4 hard disks, is an advantage connect them to the IBM Raid controller instead of the SATA integrated controller? As I read, I suppose that you should use this controller if you need more disks than can receipt the motherboard :)
Regards.

 

[jgreco]

Hello, this is my first post. Nice to meet you all.
If I choose the recommended Supermicro board and I only need 4 hard disks, is an advantage connect them to the IBM Raid controller instead of the SATA integrated controller?

No. 10 extra watts burned, too.