First NAS Build - Looking at Hardware

[Joe]

This will be my first build and I am looking for feedback on my hardware selection. We are in the process of moving and will need to set up our own storage, and thought NAS would be suitable. It's purpose will be a central location multiple users can access scientific data, so the file types will include video, images, office documents. The images and videos in particular are large files (1-2GB each). We are currently using 6TB of data, which will lead me to some other questions.

My primary concerns are reliability, maintenance/performance, as well as upgrade-ability if we need to add storage (I've only done some quick reading on setting up the pools and such from the n00b slideshow, so I know this may have to be thought out some more).

I'm also trying to read the documentation and forums as much as possible, trying to do as much research as I can, so please excuse me if I have not found the answer I'm looking for. I can also provide links to specific items below if needed.

• ADATA AV120 8GB USB Thumb Flash Drive - MFg:
• ASUS P8B-X LGA 1155 Intel C202 ATX Intel Xeon E3 Server Motherboard
• Intel Xeon E3-1230 3.2GHz 4 x 256KB L2 Cache 8MB L3 Cache LGA 1155 80W Quad-Core Server Processor BX80623E31230
• MASSCOOL 8W2002F1M4 90mm Long Life Bearing CPU Cooler for INTEL Socket LGA 1155/1156 Cooper base
• Kingston KVR1333D3E9SK2/16G 16GB (2 x 8GB) 240-Pin DDR3 SDRAM ECC Unbuffered DDR3 1333 Server Memory (16GB to start with, will most likely decide to max at 32GB at time of build or soon after).
• Antec Three Hundred Black Steel ATX Mid Tower Computer Case

As for HDD, it will be a mix of capacities of 1TB, 2TB, and/or 3TB. I haven't decided on the exact volumes yet. I am looking at the WD Red (WDX0EFRX) or Green (WDX0EZRX series) internal HDDs.
Some of my other questions/thoughts were:

1. If I can partition a HDD, e.g., the 2TB into (2) 1TB and have them as separate drives, similar to doing so in Windows (e.g., mapping a network drive). I would like to have each HDD dedicated (for example) to a type of media or type of file we use, i.e., instead of buying 2 1TB HDD, one HDD for images and one HDD for videos, purchase a 2TB and partition it for these two types of files. We also currently have individual folders, so I'd like to have a dedicated HDD for users.
2. I had originally planned a small partitioning one of the HDD for FreeNAS instead of using a USB drive. My only objection to using USB was that I did not want it to be easily pulled out (if it's plugged in externally), but I can just place the USB drive in the casing instead.

This is just for our server side, and would be duplicated for back ups.
If I can explain anything better I will be glad to do so, as well as learn. I'm sure I will have plenty of questions, and I appreciate all the help.

 

[joeschmuck]

Your system parts look great, can I have it when you're done ;).

As for the filing system, ZFS is not the same as what your use to in Windoze. You basically create a pool of drives and then create data sets and those can look like individual hard drives to the users. Look for a posting from Cyberjock, in his tagline is a link to a great explanation of how ZFS and pools work.

Here is a link where you can go to calculate out how much "usable" space you can expect based on the pool configuration. http://www.servethehome.com/raid-calculator/

You have 6TB of data now and if you plan to increase it then I would recommend your run six WD Red 3TB drives in a RAID-Z2 configuration. This will net you 10.9TB of usable space and utilize all you SATA connections on the MB. If you want more hard drives then you will need an additional controller but not any fancy RAID controller, ZFS will handle the RAID work.

 

[Joe]

Haha, I'll take that as a good sign. The parts look and seem to be compatible, I would just hope they all fit together lol. One thing I also wasn't sure of was if a server MoBo and server memory was appropriate. To be honest, I can't really tell the difference between NAS and servers; I understandtheir purposes, but it seems they can both have the same features. Or if the power supply would suffice (I estimated the Power and doubled it).

So here is my current understanding of the filling system, based upon the manual and cyberjock's slide show. Basically, a collection of HDD make a VDEV, and a collection of VDEVs make a zPool (a.k.a. a pool or volume or Zvol). A collection of pools also makes up a volume. Within the pools I can create datasets, which would be my "Images" drive and my "Video" drive, using the above examples. Within these datasets I can create a folder subsystem?

So let me try a scenario. I want to allocate 1TB for all users as a place to keep their own files, and 2TB which would contain the common office files multiple users would be accessing it. I could call these datasets, naming them "Personal" and "Shared". These two datasets would be on the same Pool, I'll call "Main". This pool would be made up of three 1TB HDD (or any combination and capacity for that matter).

Maybe then I'd create a new Pool called "Media" and have two datasets, "Images" and "Videos". I can create a folder system, similar to Windows, to help organize the files on these datasets. I'm also comparing to Windows since the users will be using Windows OS. This pool would have a VDEV of (3) 3TB HDD.

But it doesn't really seem to matter how I organize my datasets, VDEVs and pools (this seems more useful for having a "usable" pool and a "backup" pool, and leave the VDEVs and datasets as organizers). Now if I use RAID0, I would have (almost) the full 12TB; the 3TB in "Main" and the 9TB in "Media". If I use RAID-Z2, I end up with a lot less space, and from my understanding of RAID, and my generic explanation of it, this 12TB might actually be 3TB (given that there are now 4 HDD, added one :D ), since the same data is now being saved on a pair of HDD, in case of a failure.

However, RAID is not a replacement for a backup, just HDD failures. If I go ahead and use (6) 3TB drives, this leaves me with just enough capacity for all the data, and also fills up all available SATAs and bays. I would require a second similar setup, which I can then back up all the data too, on a daily/weekly/monthly basis?

I could also go ahead and use 4TB HDD, but saw those WD used a lot, and were more appropriate for the setup. Just as well, I could upgrade the casing to have 8 internal bays, but would need some hardware to expand the MoBo's SATA plugs.

Is it also a bad idea to have the main and backup in the same computer case? Or should these be separate? Backups should be on a RAID as well? The only problem with a shared main/backup would be if the MoBo or power supply fails, this might leave room for crashes.

Here's a rather stupid question: Can RAM can be upgraded even after the NAS has been up and running and in use? I see that HDD have some restrictions (cannot remove pools, add HDD to VDEVs), and I'm seeing if there are any other rules like this.

I've been reading through the 8.3.1 Documentation, and in 1.4.4 and it says 7200 RPM SATA disks are designed for single-user.. ...not designed for multi-user writes. Isn't this a problem if there will a lot of users pulling and saving files from the NAS?

Apologies for any late-night typos or ramblings, as well as the tl;dr posting.

 

[joeschmuck]

Holy crap you ask a lot of questions... :D

The file system will support all that you want to do, no issues there. When you create a dataset you can set how much room that dataset can use. Windows (CIFS) is covered just fine. As for using mixed capacities of hard drives, I recommend against it as you best results will be with the same size drives but maybe I don't understand what you are asking.

Let me ask you this... What are your requirements for your NAS? Do you have high transfer speed needs as well?

Lastly, just because you have all these questions I would highly recommend you setup FreeNAS in a virtual machine and give it a test drive.

 

[jgreco]

One quick comment, if the ASUS makes you happy, great, finally someone comes in here with a server-grade board. If you want to save a few bucks, though, you can look around to see if you can find the Supermicro X9SCM around for cheaper (as low as $150 in bulk pack).

 

[Joe]

Yeah... Sorry about that. I hope your eyes aren't bleeding. As I type I tend to think and have questions, so i type those, then I just ramble.

I would actually prefer to have similar sized HDD; I guess my questions were beating around the idea of having a "dedicated" HDD for different functions. It now seems to me that FreeNAS doesn't actually work in the same way. I'm thinking in the sense of, if you add a (4) 1TB HDD to WinOS, in My Computer you would see four individual HDD, each with a 1TB capacity. If I was to do the same in FreeNAS (or other similar-type OS??), it doesn't matter the quantity of drives I have, it will just show the 4TB in total. I can then split that up to my choosing, e.g., 500GB for "Personal", 1.5TB for "Main", 2TB for "Media". The OS will handle the "partitioning" (assuming I don't use RAID and will have the complete capacities available). Back on topic a little bit, I am now looking at Seagate Desktop HDD.15 ST4000DM000 4TB 64MB Cache SATA 6.0Gb/s 3.5" Internal Hard Drive as a storage media, in some even quantity (using that calculator, I would then have 7.3TB of usable space). Even capacities between HDD, larger capacity compared to the original, similar price.

My requirements - reliability, manageability, upgrade-ability, and performance (not in that order necessarily). We will have multiple users, perhaps 10 at any given time, but could theoretically be 25 users, pulling and saving files at once (although actual programs and computing will be done on a client/local side, such as data entry). I would say yes to the high transfer speeds, we have video files that are nearly 64GB in size, and imaging files that can be 1GB in size (they are non-PID medical files). We may have one computer that takes an image, save it to the NAS, and use a different computer to pull up that same file and analyze it. Ideally, we may want to allow outside network access (retrieve a file from home, etc., but that is probably beyond the scope of this thread).

I do hope to get a chance to try FreeNAS in a virtual machine soon, just haven't gotten the chance to sit down that much.

Nothing is set in stone, jgreco. I'm not familiar with brands (but I am aware of ASUS). If the Supermicro is a better choice, I'd be happy to use that. Is Supermicro, in general, a reliable brand to consider, or is it just that specific board? I take it as well, it was a better choice to pick a server board, opposed to a desktop motherboard?

 

[joeschmuck]

Looking at your requirements you need to think about how much speed you need. Some folks here could tell you how to get fast quick response times. You would likely need a large L2ARC if you are frequently accessing large files to speed things up but you can add that later, just leave one SATA port available for a SSD however more RAM is more favorable as it's the primary cache and is much faster than the SSD but if you reboot the machine then the SSD is faster than the rotating drives.

Also you might want to have more than one Ethernet port on your system, I just don't personally know how well this is supported in FreeNAS. It's worth checking into to maximize multiuser throughput, keeping in mind most people are using 1Gb/sec ports so 90MB/sec throughput is typical on a good system. If you wanted 10Gb ports then be prepared to spend a lot of money but they are faster and you need them on both the server and the client.

A FreeNAS system is very upgradable and expandable.

Also don't think that you must specifically assign specific amounts of disk space to each user, ZFS works a little different. You have the entire drive space available and lets say you have a 10TB pool. You have 30 employees. You can assign each employee a dataset and if you like only they can access it. In this dataset you can set a maximum size available or leave it at the default which means they could have all 10TB's available. Of course I'd establish a limit and if you need to increase it then that is an easy thing to change. Now you can create a few other datasets and leave them at the defualt so they can expand as needed.

So lets say you are running out of space, you can just add another group of drives and a you have expanded it. There are a few options here and before you go trying to expand you need to fully understand what you are doing or your data will be at risk. Never add a single drive to a pool, if that one drive dies then all your data (ALL of it) is gone. Also if the data needs to be highly reliable then go for a RAID-Z3 setup and ECC RAM system.

Sorry, I ramble on too.

 

[jgreco]

Nothing is set in stone, jgreco. I'm not familiar with brands (but I am aware of ASUS). If the Supermicro is a better choice, I'd be happy to use that. Is Supermicro, in general, a reliable brand to consider, or is it just that specific board? I take it as well, it was a better choice to pick a server board, opposed to a desktop motherboard?

If you walk into an Equinix data center and look around, you will see HP, Dell, IBM, and Supermicro gear all over the place. You will be hard-pressed to find ASUS gear.

ASUS has a good reputation in the build-your-own-box gamer community, and I'd say it is well-deserved.

Supermicro is the equivalent in the server world. No other manufacturer offers such product breadth. Anything that's supposed to work on a server is very likely to work, or at least if it doesn't, won't be because of Supermicro's part of the equation.

That having been said, I have no specific reason to doubt the ASUS board will perform acceptably and even admirably. I wouldn't have said jack except that the price I was seeing for the ASUS seemed high compared to the similar Supermicro I mentioned, which typically can be found for ~$160 in retail box or ~$150 bulk pak.

 

[cyberjock]

I just bought 2xX9SCM-F boards and I don't expect to have any problems :)

http://www.superbiiz.com/detail.php?name=MB-X9SCMFB#

 

[titan_rw]

but if you reboot the machine then the SSD is faster than the rotating drives.

Although when you reboot the machine, you have to wait for the l2arc to repopulate anyway. The l2arc is just an extension of the arc, and tracked by the arc, so the old contents of it is not used after a reboot.

I agree though, more ram is far preferable to an l2arc. On boards that are limited to 32gig ram, I'd always max out system ram before even considering an l2arc.

 

[joeschmuck]

Although when you reboot the machine, you have to wait for the l2arc to repopulate anyway. The l2arc is just an extension of the arc, and tracked by the arc, so the old contents of it is not used after a reboot.

Okay, I thought the L2ARC would stay populated but I'm sure you're correct because that does ring a bell. Sorry for misstating that.

I just bought 2xX9SCM-F boards and I don't expect to have any problems :)

http://www.superbiiz.com/detail.php?name=MB-X9SCMFB#

I can't wait to hear how it goes and all the system details.

 

[jgreco]

Yeah there's some tuning crap you can do to the L2ARC to cause it to fill very aggressively after a reboot, otherwise it takes a really long time. But a reboot on a system with L2ARC where the system is dependent on the L2ARC for sustained performance basically translates into tears until the L2ARC heats up.

On the other hand, 16GB RAM on the N36L with a reasonably-sized amount of data being deduped (maybe 1-2TB) ... L2ARC makes that very feasible.