Question on setting up drives for ZFS

[tomr]

Sorry for posting again. I didn't get a response and thought it would be better to most my question here than on the "Installation" section of the forum.

I've been researching ZFS and am hoping for some guidance. Currently I have two 1 terabyte drives and two 1.5 terabyte drives. For configuring a vdev, I am thinking of getting one more 1.5 terabyte drive thus giving me 3.0 terabytes of available storage. I would add this vdev to the storage pool. 2nd, is it possible for me to get another 1 terabyte drive and make a vdev and add it to the same storage pool? Just trying to utilize the drives that I have.

I am going to use this to store media files such as .iso rips and music. Also I will store user documents such as word docs and important family photos (these will be backed up using Cloudberry Lab running on a win 2003 server with the nas folders mapped on the server). I just want to have the security of having some redundency with our media files but they will not be backed up remotely.

Any additional guidance would greatly be appreciated.

Thanks

 

[matthewowen01]

if you were to get the 2 new drives, you could setup a single pool consisting of two raidz's one of 1 TB drives and the other of 1.5 TB drives. but that would give you 6 drives with only a single disk of redundancy. that scares me. i never go less than 2 disks ever, and highly recommend people don't either for any data they care about.

i'd buy 2 more 1.5 TB drives, set them up as a raidz2 with the original 2. then use the 2, 1 TB drives in a mirror and use them for not so important data, your iso's you can recreate.

 

[tomr]

If I went ahead and purchased four 1.5 drives and used all 6 in one pool would that be better? How many drive loses could I handle.

Sorry, for sounding like such a noob. I'm a programmer who likes to dabble on the hardware side.

I would actually like to know what you would recommend for my setup. I do have 4 gig of ram on the motherboard and it's a quad core.

 

[TapRackPull]

It's all about risk tolerance.

If you set up 6 1.5TB drives in a RAIDZ1 you would have 7.5TB of total storage capacity with 1 parity drive. So in this situation, you could lose 1 HDD and retain your data. If you lost a second drive you would have a total loss of data.

If you set up 6 1.5TB drives in a RAIDZ2 you would have 6TB of storage space and 2 parity drives. In this situation you could lose 2 drives without losing data. You're third consecutive HDD failure would result in a total loss of data.

From a statistical point of view: (Home disasters excluded, since there is no predicting that and regardless of setup your data is likely destroyed)
The odds of losing 2 HDDs at nearly the same time is on the order of winning the lottery. - Of course people win the lottery daily.
The odds of losing 3 HDDs at nearly the same time is nearly infantismal.

From a practical standpoint:
The information that you have mentioned storing on this NAS is non-critical, family photos not-withstanding. Since you will have an off-site backup of those, let's remove them from consideration. There is no practical reason that you can not power-down this NAS at any given moment should a HDD fail. If your family was hell-bent on watching a movie that evening, that is what RedBox/Netflix is for. You are a winner with either setup. In my view, you should take a step back, and determine how much space do you need, as that will likely determine the importance of that extra 1.5TB.

Start with 'How much data do I have?':
If you your current capacity is a 1TB drive chock full of data and you build a RAIDZ1 system with 7.5 times your current capacity, is that enough for the next year maybe two? For the average user its probably even overkill. But, only you know how much data you have and expansion you will likely need in the coming weeks/months. Do everything possible to not place yourself in a situation where your storage needs exceed your storage space too quickly. You need time to plan ahead when it comes to expanding a RAIDZ configuration.

If it were me:
This is only my opinion, so take it with a grain of salt (and maybe a shot of tequila). I might be more of a gambler than some. Assuming this fits into your storage space needs in the future, I would buy 1 more 1TB and 1 more 1.5TB. I would set them both up as there own vdevs in a RAIDZ1. That would yield you a combined 5TB of storage space. From my perspective, the benefit of this is two-fold. First, it saves you some cash upfront and maximizes the use of the drives that you currently have. Second is risk stratification. Playing the statistics game again, the odds of losing 2/3 of your drives from one vdev at the same time is low. When you break your disks into two groups, you signifcantly reduce the odds that both failures will come from the same vdev.

 

[tomr]

Thanks for the reply.

Just some more questions:

1. If I go with the 2 separate vdevs, can I combine them into the same storage pool because of their different sizes?
2. I am going have this is a machine using a gigabyte motherboard (2 years old) with 4 Gig of ram with the though of using the 1 terabyte drives in one vdev and 1.5 in the other. Is 4 gig of ram going to be enough?
3. Should I avoid the onboard sata connections and get a SATA Card for the machine?

Little more background, I have 1 gigabyte switch with cat 5e connections for my network. All machines have a gigabyte card. I am anticipating on getting some good transfer speeds. I'll just have to double check the drive speeds.

Thanks again everyone for your help!

It's all about risk tolerance.

If you set up 6 1.5TB drives in a RAIDZ1 you would have 7.5TB of total storage capacity with 1 parity drive. So in this situation, you could lose 1 HDD and retain your data. If you lost a second drive you would have a total loss of data.

If you set up 6 1.5TB drives in a RAIDZ2 you would have 6TB of storage space and 2 parity drives. In this situation you could lose 2 drives without losing data. You're third consecutive HDD failure would result in a total loss of data.

From a statistical point of view: (Home disasters excluded, since there is no predicting that and regardless of setup your data is likely destroyed)
The odds of losing 2 HDDs at nearly the same time is on the order of winning the lottery. - Of course people win the lottery daily.
The odds of losing 3 HDDs at nearly the same time is nearly infantismal.

From a practical standpoint:
The information that you have mentioned storing on this NAS is non-critical, family photos not-withstanding. Since you will have an off-site backup of those, let's remove them from consideration. There is no practical reason that you can not power-down this NAS at any given moment should a HDD fail. If your family was hell-bent on watching a movie that evening, that is what RedBox/Netflix is for. You are a winner with either setup. In my view, you should take a step back, and determine how much space do you need, as that will likely determine the importance of that extra 1.5TB.

Start with 'How much data do I have?':
If you your current capacity is a 1TB drive chock full of data and you build a RAIDZ1 system with 7.5 times your current capacity, is that enough for the next year maybe two? For the average user its probably even overkill. But, only you know how much data you have and expansion you will likely need in the coming weeks/months. Do everything possible to not place yourself in a situation where your storage needs exceed your storage space too quickly. You need time to plan ahead when it comes to expanding a RAIDZ configuration.

If it were me:
This is only my opinion, so take it with a grain of salt (and maybe a shot of tequila). I might be more of a gambler than some. Assuming this fits into your storage space needs in the future, I would buy 1 more 1TB and 1 more 1.5TB. I would set them both up as there own vdevs in a RAIDZ1. That would yield you a combined 5TB of storage space. From my perspective, the benefit of this is two-fold. First, it saves you some cash upfront and maximizes the use of the drives that you currently have. Second is risk stratification. Playing the statistics game again, the odds of losing 2/3 of your drives from one vdev at the same time is low. When you break your disks into two groups, you signifcantly reduce the odds that both failures will come from the same vdev.

 

[TapRackPull]

1. If I go with the 2 separate vdevs, can I combine them into the same storage pool because of their different sizes?
2. I am going have this is a machine using a gigabyte motherboard (2 years old) with 4 Gig of ram with the though of using the 1 terabyte drives in one vdev and 1.5 in the other. Is 4 gig of ram going to be enough?
3. Should I avoid the onboard sata connections and get a SATA Card for the machine?

The answer to your first question is a simple yes. When you set up your 2 vdevs, give them the same name. For example, you create a RAIDZ1 for the 3 1TB HDDs and give it the name 'Snufalupagus'. Then you create the RAIDZ1 for the 3 1.5TB HDDs. When you name it, if you call it 'Snufalupagus', it will be added to the same pool 'Snufalupagus'. If you name it 'Cookie Monster' you will generate two pools. One named 'Snufalupagus' having 2TB total space and the other 'Cookie Monster' with 3TB total space.

As for RAM, ZFS style configurations are known for being RAM intensive. 4GB should be sufficient, however, you may not get the absolute highest read/write performance that you could. I have read that you should have 1GB per TB of space, including parity drives. I think that is grossly exagerated, but I can not prove it. If money is no concern and you just wat to upgrade your system RAM, I'm sure the system would not balk at having 6GB or 8GB available, but at some point cost does become an issue. I would start with what you have. If you are getting poor performance, then invest in more. I would venture to guess you will probably be fine in the short term.

In terms of SATA cards, my last post had you installing 6 HDDs into your system. Assuming that you have 6 available SATA connections on your motherboard, then there is no reason that you must buy a SATA controller. Put that on your to-do list, and plan on buying one or two down the road if/when you need to expand your system. At that point though, you may be looking at changing out cases if you need additional HDD bays, adding memory, upgrading power supply... My best advice is use what you've got. Like I said previously, you need time to properly plan the expansion of a ZFS system. Besides, by the time that you are ready to upgrade, hopefully all the SATA controllers will be somewhat more cost efficient. If you don't have enough open SATA ports on your motherboard, then there are a number of options. I would recommend a google search on SATA controllers, and SAS controllers and get familiar with them. I do not claim to know much about them as I have not had a need as of yet. I have a small, but (for me), effective system. If your OS is running from an HDD it would be a great idea to istall it to a USB and run it from there as that opens up a SATA connection.

In so far as adding a SATA controller is concerned, if you were to the money and buy a controller down the road after you have an established system, FreeNAS is supposed to recognize the HDD structures regardless of whether they are attached to the motherboard or not. So let's say you get things up and runnig tomorrow with 6 onboard HDD and no controller card. Then in 6 months you need to expand and get 4 2TB drives for a steal. You buy a controller card and move any 4 connections from your motherboard to the new card and install your new HDDs to the motherboard. Let's say you do this because your new drives are SATA3 6GB/s and your motherboard is compatible whereas the new controller card is not. FreeNAS will see that the original drives are still there, and from your perspective nothing will have changed. But I should warn you that this is an area that I have not personally explored, so if I am wrong someone please correct me.

 

[matthewowen01]

If I went ahead and purchased four 1.5 drives and used all 6 in one pool would that be better? How many drive loses could I handle.

Sorry, for sounding like such a noob. I'm a programmer who likes to dabble on the hardware side.

I would actually like to know what you would recommend for my setup. I do have 4 gig of ram on the motherboard and it's a quad core.

if you bought 4 more, i'd do a raidz2, not a raidz1. you'll know no fear like the 4-8 day shipping time you wait while your dead drive gets replaced and you have no redundancy. having 2 disks means you have loose a disk, and not sweat bullets while you wait for the replacement. (and when the UPS tracking number says delayed due to train derailment...)

i'd steer clear of the 2 vdevs, you'll get better performance with it, but you still have your gigabit cap. you also only have 1 disk or redundancy instead of 2.

From a statistical point of view: (Home disasters excluded, since there is no predicting that and regardless of setup your data is likely destroyed)
The odds of losing 2 HDDs at nearly the same time is on the order of winning the lottery. - Of course people win the lottery daily.
The odds of losing 3 HDDs at nearly the same time is nearly infantismal.

from a statistical standpoint, choosing 2 specific disks out of all the disks in the world and having them fail "at nearly the same time is on the order of winning the lottery". However, choosing 2 disks, from a collection of 6 that came from the same manufacturing batch, that were subjected to the same wear and tear, dying within the time it takes to ship a replacement drive is quite significant.

Should I avoid the onboard sata connections and get a SATA Card for the machine?

the on-board sata connectors should have 'unrestricted' bandwidth so use them for sure. if you need to buy additional controller cards, choose wisely. a 4 port SATA 2 PCI card will perform poorly (1 gb/s total bandwidth), the PCI bus is tiny. a 4 port PCIe 1x card should do fine, but can be fully saturated by 4 sata 2 disks. (2 gb/s available vs 12 gb/s possible, from disk cache) remember a decent modern HD can push gigabit.

 

[tomr]

Matt, thanks for your comments. Between yourself and TapRackPull I feel much more informed about pursueing my setup. I think for now I'm OK with the one drive failiure. I have workarounds where if the system goes down because of a drive I can retrieve my backed up data manually from Amazon S3 until I get it back up and running. I'm going to go off and order some disks and get the system up and running.

Thanks to everyone who was kind enough to take the time and give me some guidance. I hope to soon be able to repay back when I get some hands on knowledge.

Tom

if you bought 4 more, i'd do a raidz2, not a raidz1. you'll know no fear like the 4-8 day shipping time you wait while your dead drive gets replaced and you have no redundancy. having 2 disks means you have loose a disk, and not sweat bullets while you wait for the replacement. (and when the UPS tracking number says delayed due to train derailment...)

i'd steer clear of the 2 vdevs, you'll get better performance with it, but you still have your gigabit cap. you also only have 1 disk or redundancy instead of 2.



from a statistical standpoint, choosing 2 specific disks out of all the disks in the world and having them fail "at nearly the same time is on the order of winning the lottery". However, choosing 2 disks, from a collection of 6 that came from the same manufacturing batch, that were subjected to the same wear and tear, dying within the time it takes to ship a replacement drive is quite significant.



the on-board sata connectors should have 'unrestricted' bandwidth so use them for sure. if you need to buy additional controller cards, choose wisely. a 4 port SATA 2 PCI card will perform poorly (1 gb/s total bandwidth), the PCI bus is tiny. a 4 port PCIe 1x card should do fine, but can be fully saturated by 4 sata 2 disks. (2 gb/s available vs 12 gb/s possible, from disk cache) remember a decent modern HD can push gigabit.