Best course of action with additiobal drives

[pechkin000]

I currently have two pools of 3x4tb drives each configured in Raidz1. I now purchased 2 additional drives of the same size. Naively I thought I could just add extra drive to each pool to expand their capacity however I realized that that's not the case. What is my best course of action here? My goal is to expand storage capacity as much as possible without compromising the data I already have in those pools or degrading the redundancy I have with raidz1. I can't buy another drive to create a new raidz1 array because the controller card only supports 8 drives in total. I could possibly transfer the data I have in each of the array to another server temporarily to rebuild a new pool with different configuration. So basically what I am asking is now that I have 8x4tb drives what is my best configuration that maximizes storage and has at least as good of the redundancy as Raidz1 I have now and is it possible possible to achieve this without moving existing data ?
Thanks in advance for any input.

 

[diedrichg]

Raidz2 should be your sweet spot. It will give you a 2-drive failure and still retain data.

You would need to rebuild your pools into one though to maximize storage space. So, yes, you will need to replicate your data (which is always scary and nerveracking until it's back on the server!), and then rebuild a single Raidz2 pool and then retransfer your data back. How much RAM do you have?

 

[danb35]

Do I understand correctly that you have two separate pools? In that case, you could add your two new disks as a mirror to one of the pools, increasing its capacity by 4 TB. That's really your only option to expand (one of) your existing pools while retaining redundancy, and not having to shuffle your data.

Other options would be (1) do as @diedrichg suggests and build a new RAIDZ2 pool using all 8 disks (probably the best, in the long term), or (2) rebuild one pool at a time into 4 x 4 TB RAIDZ1 pools. Option 2 doesn't need as much temporary storage for your data, but option 1 is better in the redundancy department. Either way, two disks could fail without loss of data, but with option 2, they have to be the "right" two disks--one in each pool. If two disks fail in the same pool, it's toast. With option 1, any two disks can fail without loss of data.

 

[pechkin000]

Thanks for the reply! I have 16gigs in there currently which I guess is cutting it close now with two additional drives. I could get two more sticks to make it 32 but isn't it a bit of an overkill?
danb35 - yes I have two separate Raidz1 pools sx4tb drives each.
I think I am going to go for the single Raidz2 pool, just need to find a spot to stash roughly 4TB of data for a min lol. I have to say I am a bit nervous about transferring so much stuff, its a lot of premier pro video projects as well as a lot of videos cataloged by our digital asset management software so all of that is going to be off line and at risk of getting corrupted during the two transfers, but I guess this is the best option....

 

[diedrichg]

In FreeNAS there is never too much RAM. From other comments in the past people have said 16GB is perfectly fine for 8 drives in Raidz2. As @danb35 mentioned, you can add them as a mirror but that gives you storage at the cost of redundancy.

Also keep in mind that you will want to set your pool(s) quota at about 80%, as I have read, there is some overhead that will prevent you from accessing your data if you go over 90% capacity.

 

[david kennedy]

Thanks for the reply! I have 16gigs in there currently which I guess is cutting it close now with two additional drives. I could get two more sticks to make it 32 but isn't it a bit of an overkill?
danb35 - yes I have two separate Raidz1 pools sx4tb drives each.
I think I am going to go for the single Raidz2 pool, just need to find a spot to stash roughly 4TB of data for a min lol. I have to say I am a bit nervous about transferring so much stuff, its a lot of premier pro video projects as well as a lot of videos cataloged by our digital asset management software so all of that is going to be off line and at risk of getting corrupted during the two transfers, but I guess this is the best option....

Just wanted to add my two cents.

I have a box with the following:
10x2 tb drives in raidz2
10x3 tb drives in raidz2
In two separate pools with 16GB and haven't had any issues running this for more then a year now despite both pools being around 50% allocated.
It is a lightly used media box and even if i added more ram it likely wont help much as the read workload is all over the place anyhow (no ARC benefits).

I think the myth about 1gb/tb should die off, just like the myth about “(2^n)+p” as the "optimal" raid config.

People have been spreading "optimal width" logic for a long time, until one of the lead developers of ZFS decided to tackle it (http://blog.delphix.com/matt/2014/06/06/zfs-stripe-width/)

Flip side, there is a real concern about using too little ram (<8gb) and having issues with your pool.



"...there is some overhead that will prevent you from accessing your data if you go over 90% capacity."

You can go above 80% but ZFS changes its allocation mechanism and slows down (a lot).

 

[pechkin000]

So I decided to rebuild the array. I was under impression that I would get extra storage space if I did Raidz2 with 8 drives. However, if I tried to build straight 8 drive array in raidz 2 it said that its not optimal, the only optimal configuration it allowed was 4drives in two mirror arrays:

Code:

[root@freenas] ~# zpool status
  pool: storage
state: ONLINE
  scan: none requested
config:

        NAME                                            STATE     READ WRITE CKSUM
        storage                                         ONLINE       0     0     0
          raidz2-0                                      ONLINE       0     0     0
            gptid/4ff304d1-20f3-11e4-9b11-002590d98818  ONLINE       0     0     0
            gptid/50565b73-20f3-11e4-9b11-002590d98818  ONLINE       0     0     0
            gptid/50c9725c-20f3-11e4-9b11-002590d98818  ONLINE       0     0     0
            gptid/512cfbbb-20f3-11e4-9b11-002590d98818  ONLINE       0     0     0
          raidz2-1                                      ONLINE       0     0     0
            gptid/51905612-20f3-11e4-9b11-002590d98818  ONLINE       0     0     0
            gptid/51f39b17-20f3-11e4-9b11-002590d98818  ONLINE       0     0     0
            gptid/52572ea6-20f3-11e4-9b11-002590d98818  ONLINE       0     0     0
            gptid/52be013c-20f3-11e4-9b11-002590d98818  ONLINE       0     0     0

Which only gave me just under 13 tb storage on 8 4tb drives... is this really the best I can do? I basically didn't gain any space at all by adding two drives, I actually lost some space. When I had 2 Raidz1 pools 3x4tb each I had actually slightly more storage space roughly 7+7 tb

 

[cyberjock]

It's safe to do 8 disks in a single vdev. The whole "optimal" thing is gone in 9.3. The old thumbrules for what an optimal zpool is are basically null and void with compression and compression is enabled by default with FreeNAS starting with 9.2.0.

 

[david kennedy]

So I decided to rebuild the array. I was under impression that I would get extra storage space if I
Which only gave me just under 13 tb storage on 8 4tb drives... is this really the best I can do? I basically didn't gain any space at all by adding two drives, I actually lost some space. When I had 2 Raidz1 pools 3x4tb each I had actually slightly more storage space roughly 7+7 tb

I guess you can think of it as you didnt "lose" space, you gained redundancy.

As Cyberjock says, just do the 8 in raidz2. Even if there is a "performance" or "storage" hit it would probably be minor.

 

[pechkin000]

Is there any way to see how much of a performance hit it will be? I am using this storage primarily for Premier Pro projects to be accessible over the network so I would like to maximize performance if possible.

 

[cyberjock]

The performance hit is not a given percentage. It depends on factors that are out of your control. Just use the 8 disks in a single RAIDZ2 like I said. What you are losing is statistically unimportant.

 

[pechkin000]

Thanks cyberjock! That's all I needed to hear. I appreciate everybody's input.

 

[david kennedy]

Thanks cyberjock! That's all I needed to hear. I appreciate everybody's input.

Sorry if I confused you by saying there would be a performance or storage hit. If you read these forums you will find all sorts of fanatical theoretical discussions about why “(2^n)+p” is the best. These "theoretical" issues don't translate well into the real world and fortunately one of the leads for ZFS actually stepped in to help address the issue/concerns.

I ran a "non-optimized" 7 disk pool for around 1.5 years (rebuilt it when I ran out of space). Never noticed anything worth mentioning.

 

[cyberjock]

Actually, they translate VERY well to the real world. The reality is that, since we are not running 10Gb, it doesn't take much hardware to saturate a 1Gb LAN port. If you had an 8 disk vdev versus 7 disk, even if we assumed a 75% penalty (which is definitely 10 fold too high) you'd still be bottlenecked by the LAN port.

But today the whole "optimal" thing goes out the window thanks to compression. So unless you build the "optimal" array, disable compression, and plan to use 10Gb+ you shouldn't worry about it.

 

[david kennedy]

Actually, they translate VERY well to the real world. The reality is that, since we are not running 10Gb, it doesn't take much hardware to saturate a 1Gb LAN port. If you had an 8 disk vdev versus 7 disk, even if we assumed a 75% penalty (which is definitely 10 fold too high) you'd still be bottlenecked by the LAN port.

But today the whole "optimal" thing goes out the window thanks to compression. So unless you build the "optimal" array, disable compression, and plan to use 10Gb+ you shouldn't worry about it.

How does it translate to the real world when people's use cases vary so greatly?
Anyhow, from the Delphix article (where many of the SUN devs have gone to):

A misunderstanding of this overhead, has caused some people to recommend using “(2^n)+p” disks, where p is the number of parity “disks” (i.e. 2 for RAIDZ-2), and n is an integer. These people would claim that for example, a 9-wide (2^3+1) RAIDZ1 is better than 8-wide or 10-wide. This is not generally true.

And this:
The strongest valid recommendation based on exact fitting of blocks into stripes is the following: If you are using RAID-Z with 512-byte sector devices with recordsize=4K or 8K and compression=off (but you probably want compression=lz4): use at least 5 disks with RAIDZ1; use at least 6 disks with RAIDZ2; and use at least 11 disks with RAIDZ3.

How many devices are still 512K?

 

[cyberjock]

Sorry, I'm not going to sit and try to argue my side. I've gotten an email from one of the ZFS devs on this topic. His email is why the "optimal" stuff is removed from FreeNAS 9.3. If you don't want to accept my argument you don't have to. I've been part of the discussion within iX over this topic. I really don't want to hash this out, especially since it really doesn't matter. The whole optimal thing is completely void because of compression.

 

[david kennedy]

Sorry, I'm not going to sit and try to argue my side. I've gotten an email from one of the ZFS devs on this topic. His email is why the "optimal" stuff is removed from FreeNAS 9.3. If you don't want to accept my argument you don't have to. I've been part of the discussion within iX over this topic. I really don't want to hash this out, especially since it really doesn't matter. The whole optimal thing is completely void because of compression.

Where is the "argument" stemming from? You said it applies to the real world and I am curious as to how/why?

At the lowest level I think we are on the same page, that the "optimal" stuff was a bad choice for people to try to enforce because the material impact is very low. You then countered my point stating it does apply to the "real world" yet agree that creating non-optimal pools is ok due to compression?


The point of these boards is to share knowledge and learn, correct?

 

[no_connection]

http://blog.delphix.com/matt/2014/06/06/zfs-stripe-width/

Read it for the Dr. Strangelove reference if nothing else.

 

[david kennedy]

http://blog.delphix.com/matt/2014/06/06/zfs-stripe-width/

Read it for the Dr. Strangelove reference if nothing else.

I'm familiar with the posting, in fact i actually included it in one my own posts.
It says that "optimal stripe width" is more a myth and doesn't answer the question i have posted.

 

[no_connection]

Sorry I missed that.

Here is another link that might be useful
http://web.archive.org/web/20140403...s.org/ritk/zfs-4k-aligned-space-overhead.html
Although free space is probably a poor indicator as that changes constantly. And I doubt any one would fill up different systems and see what can hold more data. And what the ramifications are.