ZFS Pools Question

[krone6]

Hi all.

So if I get this right by reading on the forum and a few FAQ's, I can start adding a ZFS volume made out of 2 1TB drives, and then add 2 1TB drives at a time to this same pool, it'll be redundant? If so, in what ways? When one drive dies, what happens to the data and what will be my total size of the data?

I'm looking for expandibility but with some protection. Even a simple "raid-1" type of protection is fine. I have a hardware 8 port areaca-1220 card, but apparently it'd be easier to maintain with ZFS on freenas.

Thanks in advance for any help and expanation on how ZFS and the volumes/pools will work for my needs.

 

[cyberjock]

Check out my guide in the sig. It'll answer all of your questions.

 

[krone6]

Check out my guide in the sig. It'll answer all of your questions.

Thanks. That helped me out a lot and I'm starting to realize the question I almost asked on here would have been silly compared to going the ZFS route.

I do have a few more questions so I have something to think about for the future. I'm probably going to go with a 2-disk zfs mirrored setup. I'll buy 2 disks at a time and mirror them, then put the vdev into the zpool, though I need your opinion on this.

1: Reliably speaking, is this better than every drive in a giant raid-z2 format? I'm not after speed. If it gets 1MB/S that's quick enough for my use (obviously it'll be faster though).
2: If I go this route and have 6 1TB disks, will I have 3 1TB mirrored drives to share across my network, or 1 3TB drive with the ability for 1 drive to break in each mirror and all data be fine?

Also, what keywords should I google to create a second, identical server to act as a backup NAS to my main one? If every part were to fail at once in the main one, no data is lost and i'm alerted while at the same time the backup NAS takes over in real time. I probably wont' need this setup realistically, but my needs change as I gain more knowledge, and it sounds like a fun project to work on; I already have the case (same one) to rackmount already too.

Last, is it wise to buy sata add-on cards to add in additional hdd's? I only have 6 ports on my freenas server, but can store up to 12 hdd's if i wanted to. If I go this route, does it matter how much I pay for one?

Thanks in advance for clearing up these questions.

EDIT: Should I bother upgrading my cpu/ram based on this scenario?

Client: Integrated NIC transferring across network to freenas server at 57MB/S (started at 55).
Server: Semperon 140 with 2GB of g.skill 1066 ddr3 ram. has a 2 drive (black/green 1TB WD with two different rpm's) with no optimization done. Integrated nic too.

 

[cyberjock]

1: Reliably speaking, is this better than every drive in a giant raid-z2 format? I'm not after speed. If it gets 1MB/S that's quick enough for my use (obviously it'll be faster though).

If you have 10 drives mirrored versus 10 drives in a RAIDZ2 and you have both drives fail in 1 mirrrored set you will lose everything. With RAIDZ2 you have to lose any 3 drives to lose everything. Statistically, there is a chance that you may lose some or all data using mirrors. Some people elect to use mirrors because they don't need much CPU power.

If you absolutely MUST have the expandability of adding 2 disks at a time, this is your only option. If you can build the entire array all at once, that's preferred. With a bunch of drives mirroring you are literally paying double per GB of storage.

2: If I go this route and have 6 1TB disks, will I have 3 1TB mirrored drives to share across my network, or 1 3TB drive with the ability for 1 drive to break in each mirror and all data be fine?

Depends on how you setup the mirroring. Either one are possible AFAIK. It just depends on how you setup the mirroring. A RAID0+1 would give you different results from a bunch of RAID1s. I have never tried mirroring because I don't like the idea of paying double the price for the same space. Once you start looking at buying alot of storage space, paying double makes it that much higher. Not to mention the chances that you lose both drives of a mirrored set at the same time.

Also, what keywords should I google to create a second, identical server to act as a backup NAS to my main one? If every part were to fail at once in the main one, no data is lost and i'm alerted while at the same time the backup NAS takes over in real time. I probably wont' need this setup realistically, but my needs change as I gain more knowledge, and it sounds like a fun project to work on; I already have the case (same one) to rackmount already too.

Rsync is the typical method for having a production server and backup server. With Rsync you setup your production server to backup to the backup server on a schedule you specify.

Last, is it wise to buy sata add-on cards to add in additional hdd's? I only have 6 ports on my freenas server, but can store up to 12 hdd's if i wanted to. If I go this route, does it matter how much I pay for one?

It's possible. Some people repurpose some RAID cards by flashing the firmware with a non-RAID firmware.

EDIT: Should I bother upgrading my cpu/ram based on this scenario?

Client: Integrated NIC transferring across network to freenas server at 57MB/S (started at 55).
Server: Semperon 140 with 2GB of g.skill 1066 ddr3 ram. has a 2 drive (black/green 1TB WD with two different rpm's) with no optimization done. Integrated nic too.

You should verify that your NICs are compatible with FreeBSD. NICs seem to bite alot of newbies. That's why the guide says to check or just buy Intel NICS.

With RAM, just as the guide says, you can never have too much. You'll learn by example if you don't have enough RAM. The system will panic when heavily loaded. I always look at nothing less than 8GB of RAM. You get the prefetch caching when you have more than 4GB of RAM installed. As cheap as RAM is, I always just max out the machine. I'd rather not deal with a kernel panic if I can avoid it. If you are already looking a spending large amounts of RAM on hard disks, what's an extra $50-100 to max out the RAM?

RAM is the one thing that will remind you if you skimp on it. You'll have odd performance issues and random kernel panics.

 

[krone6]

If you have 10 drives mirrored versus 10 drives in a RAIDZ2 and you have both drives fail in 1 mirrrored set you will lose everything. With RAIDZ2 you have to lose any 3 drives to lose everything. Statistically, there is a chance that you may lose some or all data using mirrors. Some people elect to use mirrors because they don't need much CPU power.

If you absolutely MUST have the expandability of adding 2 disks at a time, this is your only option. If you can build the entire array all at once, that's preferred. With a bunch of drives mirroring you are literally paying double per GB of storage.



Depends on how you setup the mirroring. Either one are possible AFAIK. It just depends on how you setup the mirroring. A RAID0+1 would give you different results from a bunch of RAID1s. I have never tried mirroring because I don't like the idea of paying double the price for the same space. Once you start looking at buying alot of storage space, paying double makes it that much higher. Not to mention the chances that you lose both drives of a mirrored set at the same time.



Rsync is the typical method for having a production server and backup server. With Rsync you setup your production server to backup to the backup server on a schedule you specify.



It's possible. Some people repurpose some RAID cards by flashing the firmware with a non-RAID firmware.



You should verify that your NICs are compatible with FreeBSD. NICs seem to bite alot of newbies. That's why the guide says to check or just buy Intel NICS.

With RAM, just as the guide says, you can never have too much. You'll learn by example if you don't have enough RAM. The system will panic when heavily loaded. I always look at nothing less than 8GB of RAM. You get the prefetch caching when you have more than 4GB of RAM installed. As cheap as RAM is, I always just max out the machine. I'd rather not deal with a kernel panic if I can avoid it. If you are already looking a spending large amounts of RAM on hard disks, what's an extra $50-100 to max out the RAM?

RAM is the one thing that will remind you if you skimp on it. You'll have odd performance issues and random kernel panics.

Thanks for the help so far. I just have some follow up questions and I should be set. I wanted to try to combine all of my questions in one thread to reduce spam.

1: If I have 10 hard drives in a zpool, and every 2 drives are a mirrored vdev, then would I have 5GB of total space when I share this volume to my network? (If those terms are right).

2: If both hard drives in one vdev dies at once, does the data in the entire zpool get erased, or just in that vdev?

3: Isn't it more expensive to buy 3-4 hard drives at a time instead of 2? I can't do a raid-z since I am starting with 2 drives. I could buy another one and start my raid-z, but then If I want to expand wouldn't I need to buy in 3's or 4's instead of 2's to maintain reliability?

4: What are the effects of having different RPM drives in a vdev? Currently my mirror has a WD black drive at 7200rpm and a green at 5400.

5: Is it me, or did windows lie to me? It said 55MB/S, and it felt like 55MB/S, but it didn't look like 55MB/S. I noticed every second only 4MB was transferred instead of 40-50 like it said. Why is this? If it makes a difference, I'm increasing my ram to 8GB tomorrow due to the deal I found.

6: What is the difference in a 3 drive mirror'ed vdev and a 2 drive vdev with a hotspare? Wouldn't they function very similar?

7: About my sata-addon question. Since you say it's a plausible scenario that I can buy such a card to use, are there any key things I should look out for? Things such as pci vs pci-e for example? And does it matter where my sata cable's plugged into? Should I plug every other hard drive into the card vs motherboard, or fill up the 6 on my motherboard and then the 4 or so on the card as I expand?

These should be the last major questions I have. I didn't expect to have so many, though again, thanks for taking the time to read them and answer them.

 

[cyberjock]

Thanks for the help so far. I just have some follow up questions and I should be set. I wanted to try to combine all of my questions in one thread to reduce spam.

1: If I have 10 hard drives in a zpool, and every 2 drives are a mirrored vdev, then would I have 5GB of total space when I share this volume to my network? (If those terms are right).

2: If both hard drives in one vdev dies at once, does the data in the entire zpool get erased, or just in that vdev?

3: Isn't it more expensive to buy 3-4 hard drives at a time instead of 2? I can't do a raid-z since I am starting with 2 drives. I could buy another one and start my raid-z, but then If I want to expand wouldn't I need to buy in 3's or 4's instead of 2's to maintain reliability?

4: What are the effects of having different RPM drives in a vdev? Currently my mirror has a WD black drive at 7200rpm and a green at 5400.

5: Is it me, or did windows lie to me? It said 55MB/S, and it felt like 55MB/S, but it didn't look like 55MB/S. I noticed every second only 4MB was transferred instead of 40-50 like it said. Why is this? If it makes a difference, I'm increasing my ram to 8GB tomorrow due to the deal I found.

6: What is the difference in a 3 drive mirror'ed vdev and a 2 drive vdev with a hotspare? Wouldn't they function very similar?

7: About my sata-addon question. Since you say it's a plausible scenario that I can buy such a card to use, are there any key things I should look out for? Things such as pci vs pci-e for example? And does it matter where my sata cable's plugged into? Should I plug every other hard drive into the card vs motherboard, or fill up the 6 on my motherboard and then the 4 or so on the card as I expand?

These should be the last major questions I have. I didn't expect to have so many, though again, thanks for taking the time to read them and answer them.

1. Yes, you'd have only 5GB of usable space.
2. If any vdev fails(loses all redundancy + 1 more disk) in a zpool, you lose all data in the zpool.
3. If you are mirroring you are paying double the price per GB(50% of your drives are mirrored). If you use RAIDZ2, then only the percentage of drive that are parity data. 10 drives mirrored cost the same as 10 drives in a RAIDZ2, but the RAIDZ2 will cost you less per GB of available disk space.
4. Unexpected odd performance issues can result. Nobody does it mostly because nobody wants to find out. For many people, you either jump in with both feet and buy all new drives, or you reuse old if you don't care about performance
5. Windows' meter is averaged. If you did 200MB/sec for 3 seconds, then it paused for 2 seconds, it would say something like 125MB/sec or so. Writing 1 50GB file is always faster than writing 10k files totaling 50GB.
6. There is no such thing as a 3 drive mirror. At least, I've never heard of anyone doing that. Hotspares don't work quite right in FreeNAS, so they aren't recommended at this time. Why you'd go with a 3 drive mirror instead of a RAIDZ1 is beyond me. You'd be paying more for redunancy than for actual disk space.
7. Doesn't matter what drives are plugged in where. FreeNAS sees the disks as devices in a zpool. PCIe is preferred to minimize bottlenecks. There's a post somewhere in the forum with a link to a bunch of FreeNAS controllers, their prices, and how well they functioned along with any bottlenecks found. Overall, a 12 port 1xPCIe card won't perform as well as an 8 port 4xPCIe card. You have to look at total number of ports as well as the card interface.

 

[krone6]

Thanks for the info. In that case, I'll leave the system as is and buy 6-8 drives at once to raidz2 (i know about the 4, 6, 10 thing for raidz, so probably just 6) and upgrade them as time goes by. Probably start with some 500GB or 750GB ones, it'd still get me to 2-3TB compared to my 1TB.

Also, I never said I was going to hot-spare, just wondered if there was a difference between those two things. I read an article that treated the mirror+hotspare thing as the next coming. He sounded as if it would save all of man kind or something, but I didn't see much point in that much redundancy at that low of size.

EDIT: Question about earlier on raid-z2. If I went with an 8 drive setup, would it be better to have 2 4 drive vdevs in raid-z2 or 1 vdev with 8 drives on raid-z2. Judging by your power point, it'd seem safer since I'm doubling my reliability. I'll be able to have any 2 drives fail in BOTH vdevs instead of 1, even if there were 8 total drives, 3 failures mean all data is gone.

 

[JaimieV]

Careful now - two times (four drives in a RAIDZ2 vdev) uses the same number of disks, gives no more resilience, uses more CPU and is generally slower than a straight mirror setup. Kinda pointless.

Without running the probability calculation I think you'd be very slightly better off (statistically) with 8 drives in RAIDZ3, where *any* 3 fails means you're still running.

As ever, remember to plan your backup strategy alongside the initial array. Are you looking for high availability, or is offline recoverable just as useful? So would you be better off eg putting 5 disks into a RAIDZ1 and using the remaining three as a stripe set elsewhere to back up to? Because RAID is not backup, no matter how high the number after the Z!

 

[krone6]

Careful now - two times (four drives in a RAIDZ2 vdev) uses the same number of disks, gives no more resilience, uses more CPU and is generally slower than a straight mirror setup. Kinda pointless.

Without running the probability calculation I think you'd be very slightly better off (statistically) with 8 drives in RAIDZ3, where *any* 3 fails means you're still running.

As ever, remember to plan your backup strategy alongside the initial array. Are you looking for high availability, or is offline recoverable just as useful? So would you be better off eg putting 5 disks into a RAIDZ1 and using the remaining three as a stripe set elsewhere to back up to? Because RAID is not backup, no matter how high the number after the Z!

I'm just looking to build a server that'll offer reliable storage and will run 24/7 if possible. Luckly windows can see ZFS, so i can take the server offline and plug a disk up to my main computer if i have too. i just want it to be simple enough that i can risk a disk or two dying to give me time to replace but still keep my data around.

When you say raid isn't a backup, then what is? Why even do what the people on here do in that case? I thought the point with zfs and raid is to increase reliability so your data doesn't get erased on disk failures?

If it helps, I can't buy 8 disks at once (7). I just don't have the cash, but I can probably buy 3 to start off with a 4 disk raid-z.

In your opinion starting off with 4 drives and expanding up to 8 in the future, what would be the most simple system to allow me to not worry too much on losing my data and still give me some time to replace a disk on failure?

EDIT: What is your nas system like? That'd probably help me a lot more.

EDIT2: I'm going to use green drives too by the way. I'm rarely going to transfer that often on the green drives, only the blacks for my CCTV, so I don't think I need that much performance. They're also going to be put into low-power states often as well to conserve on power. The CCTV hdd will be the one that'll be up 24/7 writing 24/7.
(all WD drives currently)

 

[cyberjock]

Careful now - two times (four drives in a RAIDZ2 vdev) uses the same number of disks, gives no more resilience, uses more CPU and is generally slower than a straight mirror setup. Kinda pointless.

Not true. If you have mirrors if drive 0(0a and 0b) and drive 1(1a and 1b) versus RAIDZ2 you have better resilience with RAIDZ2. If drives 0a and 0b fail simultaneously you lose all data. With RAIDZ2 you can lose ANY 2 drives and still be online. RAIDZ2 obviously needs more CPU power than a mirror, but does provide some protection. How likely are you to have a a second mirror fail at the same time? About 33% of the time when one disk is already failed.

So statistically, you are better off with a RAIDZ2 of 4 drives, if CPU performance doesn't limit you. I'm really not sure why so many people argue the opposite, as you did JaimieV. There is a difference if you don't just look at the "hard numbers" and take into account the design.

I think someone calculated out the chances of RAIDZ2 saving your data where a mirror fails at something like 65%. I wish I could find the exact article I read about it.

 

[krone6]

Not true. If you have mirrors if drive 0(0a and 0b) and drive 1(1a and 1b) versus RAIDZ2 you have better resilience with RAIDZ2. If drives 0a and 0b fail simultaneously you lose all data. With RAIDZ2 you can lose ANY 2 drives and still be online. RAIDZ2 obviously needs more CPU power than a mirror, but does provide some protection. How likely are you to have a a second mirror fail at the same time? About 33% of the time when one disk is already failed.

So statistically, you are better off with a RAIDZ2 of 4 drives, if CPU performance doesn't limit you. I'm really not sure why so many people argue the opposite, as you did JaimieV. There is a difference if you don't just look at the "hard numbers" and take into account the design.

I think someone calculated out the chances of RAIDZ2 saving your data where a mirror fails at something like 65%. I wish I could find the exact article I read about it.

So in your opinion would it be wise to start with a 4 drive raid-z2 and then expand to an 8 later? I can't buy all 8 at once due to cash. I'm looking for something that's simple, but still offers a form of protection as well. I'd love space too of course, but i'd rather focus more on reliability instead.

Plus, does it matter if the storage drives are green western digital drives? I'm not going to transfer that often. Maybe a few times every few days, they'll just store info. The black WD drives will be my 24/7 performance ones.

And what rpm is a 5400-7200? Is it safe enough to use in a raid with 7200rpm green drives? Oh, and what's your setup like?

 

[cyberjock]

I use green drives and I have had zero problems(I have 22 of them total). I like them because they run alot cooler and draw less power. But make sure you change the wdidle settings. Google for how to do it, there's tons of posts in this forum describing how to do it.

With FreeNAS you really are better off building it all at once the way you want it. Trying to do small ugprades after a system is built is difficult, not to mention you typically lose some efficiency with setting up the zpools how you want it right now. My recommendation would be 6 drives of RAIDZ2. Per my guide there are thumbrules to follow for maximum performance. One of those thumbrules would dictate that RAIDZ2 have 4, 6 or 10 disks.

Personally, making arrays of less than 5 or 6 disks is a real waste of money because you pay double per GB of usable space because of space lost for parity.If you buy 4 drives now and 4 in 3 months you will pay more for the same usable space as if you had just bough 6 drives now.

If you can't afford 6 disks right now I'd wait until you can. There's no substitute for doing it right the first time, because you can't "fix" it after its made if you are unhappy. You can always choose to use 2 old smaller disk to create the array, then swap them out one at a time in a month or two when you have more cash. Just keep in mind that your total zpool size will be limited accordingly based on what size drives you use.

 

[Stephens]

EDIT: What is your nas system like? That'd probably help me a lot more.

Mine's in my signature.

I think someone calculated out the chances of RAIDZ2 saving your data where a mirror fails at something like 65%. I wish I could find the exact article I read about it.

I don't think this is the article you're referring to, but it's pretty good too.

With FreeNAS you really are better off building it all at once the way you want it.

This is true for many reasons.

 

[krone6]

I use green drives and I have had zero problems(I have 22 of them total). I like them because they run alot cooler and draw less power. But make sure you change the wdidle settings. Google for how to do it, there's tons of posts in this forum describing how to do it.

With FreeNAS you really are better off building it all at once the way you want it. Trying to do small ugprades after a system is built is difficult, not to mention you typically lose some efficiency with setting up the zpools how you want it right now. My recommendation would be 6 drives of RAIDZ2. Per my guide there are thumbrules to follow for maximum performance. One of those thumbrules would dictate that RAIDZ2 have 4, 6 or 10 disks.

Personally, making arrays of less than 5 or 6 disks is a real waste of money because you pay double per GB of usable space because of space lost for parity.If you buy 4 drives now and 4 in 3 months you will pay more for the same usable space as if you had just bough 6 drives now.

If you can't afford 6 disks right now I'd wait until you can. There's no substitute for doing it right the first time, because you can't "fix" it after its made if you are unhappy. You can always choose to use 2 old smaller disk to create the array, then swap them out one at a time in a month or two when you have more cash. Just keep in mind that your total zpool size will be limited accordingly based on what size drives you use.

Thanks. You confirmed what I was going to do. I was going to have 2 1TB Black WD drives for my CCTV (24/7 writes) and 6 500GB 7200rpm drives for my storage. I'd love to go lower in rpm, but I have a green 7200rpm drive, so why waste what I already have. Plus, it was only $50 for 1TB, good deal at the time. I guess I'll just have to pickup 4-5 drives in the future and then do my storage.

Last question since my others were answered. I have a 500GB 5400-7200rpm green drive. Is this similar to a 7200rpm only and is it worth using as one of my 500GB drives for the 6 drive raid-z2 i'll work towards? What I wanted to use it for can't work, but I'd love to do something with it instead.

 

[krone6]

Nevermind on the questions. I talked to some people who used zfs before and they've gave me some good advice.

 

[JaimieV]

So statistically, you are better off with a RAIDZ2 of 4 drives, if CPU performance doesn't limit you. I'm really not sure why so many people argue the opposite, as you did JaimieV. There is a difference if you don't just look at the "hard numbers" and take into account the design.

Thanks - you are of course right, thanks for the correction. I was doing exactly what you suspected, just counting the drives.

If Krone6 is still reading, my config (including backup mechanism!) is as seen at the top of the thread http://forums.freenas.org/showthread.php?9287-Rsync-or-Replication

 

[krone6]

Thanks - you are of course right, thanks for the correction. I was doing exactly what you suspected, just counting the drives.

If Krone6 is still reading, my config (including backup mechanism!) is as seen at the top of the thread http://forums.freenas.org/showthread.php?9287-Rsync-or-Replication

Yup, I'm still reading. I'll read that thread later after school.