Setting up a raidz1 with different sized disks - FreeNAS 9.1

[iBobbe]

Hi everyone,
I'm trying to set up a raidz1 pool with 3 disk for my FreeNAS 9.1.0 machine for media storage purpose, but I'm having some trouble with it due to my inexperience.

My system is based on a Asus E45M1-M Pro motherboard with 8GB of RAM.

I have the following disks:

• 1x 3TB
• 2x 2TB
• 1x 1TB

and I've roughly 2TB of data (in NTFS) on one of the 2TB drives.
My (probably wrong) idea is to set up a Raidz1 pool with the three empty drives (3TB, 2TB, 1TB), which should has a 2TB usable space, copy my data into it and then replace the 1TB disk with the remaining 2TB one, expanding the pool to 4TB usable.
If someone can recommend a better idea is well accepted :)

ZFS Manager in FreeNAS 9.1 doesn't allow you to create a raidz1 pool with different sized disks, so I start searching a solution and i found that is possible to do it using the shell and here comes the trouble.

I first tried with

Code:

zpool create Media raidz ada0 ada1 ada2

getting the error
"invalid vdev specification
use '-f' to override the following errors:
raidz contains devices of different sizes"

than I tried with

Code:

zpool create -f Media raidz ada0 ada1 ada2

getting
"cannot mount '/Media': failed to create the mountpoint"

Tried some other solution without success.

Zpool status tells me that everything is online and zpool list says that 'Media' has a size of 2.72T (the real size of the 3TB disk) instead of the supposed 2TB.
At this point I'm confused, can someone help me to explain what happened and how I should proceed?
Thanks in advance for your patience!

 

[paleoN]

At this point I'm confused, can someone help me to explain what happened and how I should proceed?

You needed to create the pool under /mnt as / is read-only. Export the pool, zpool export Media, and auto import via the GUI. As for the space zfs list.

Personally, I would destroy the pool and recreate it with partitions like the GUI does. For each disk:

Code:

gpart create -s gpt ada0

gpart add -b 128 -i 1 -t freebsd-swap -s 2G ada0

gpart add -i 2 -t freebsd-zfs ada0

Then when you recreate the pool specify the ZFS partitions.

 

[iBobbe]

Thank you very much paleoN!
I had temporarily installed FreeNAS 8.3 that allows me to create raidz1 with different sized disks, after I had upgraded to 9.1.1, but if I needed to create a new raidz1 pool I could not.

I've deleted the pool via the GUI, than

Code:

gpart create -s gpt ada0
gpart add -b 128 -i 1 -t freebsd-swap -s 2G ada0
gpart add -i 2 -t freebsd-zfs ada0
gpart create -s gpt ada1
gpart add -b 128 -i 1 -t freebsd-swap -s 2G ada1
gpart add -i 2 -t freebsd-zfs ada1
gpart create -s gpt ada2
gpart add -b 128 -i 1 -t freebsd-swap -s 2G ada2
gpart add -i 2 -t freebsd-zfs ada2

Now should I proceed like this?

Code:

zpool create -f /mnt/Media raidz ada0p2 ada1p2 ada2p2

 

[paleoN]

Now should I proceed like this?

Yes. You still want to export then auto import via the GUI afterwards.

 

[iBobbe]

Ok, I'll try soon!

When I'll have copied my data on the zpool and replaced the 1TB drive with the 2TB one, I could add the 1TB disk as cache? Does it will make any difference?

 

[paleoN]

When I'll have copied my data on the zpool and replaced the 1TB drive with the 2TB one, I could add the 1TB disk as cache? Does it will make any difference?

Yes, it makes a difference. It will likely give you lower performance. Cache device should be at least an order of magnitude faster, latency/IOPS, than the main pool devices.

 

[iBobbe]

Thank you for the help and all the explications ;)

 

[strange]

Greetings,

I am having the same problem described above so i followed the steps given, but when i try this:

gpart create -s gpt ada0

I get this error:

gpart: geom 'ada0': Operation not permitted
gpart: geom 'ada0': Operation not permitted
any suggestions?

 

[Dusan]

gpart: geom 'ada0': Operation not permitted

Did you destroy any previous pool/volume?

 

[strange]

Did you destroy any previous pool/volume?

No, it was was a fresh install with a bunch of unformatted drives

 

[jyavenard]

Solution you could take would be that rather than adding the whole disk; use partitions instead.

Create a 1TB partition on your 2TB and 3TB disks. And use that for creating your raidz (use the whole 1TB disk as 3rd drive)

Then you can do it again for the remaining 1TB on the 2TB disk and another one the 3TB.

If you wanted to live on the edge; creating a third 1TB partition on the 3TB drives would give you 2 raidz1 array, of 2TB usable each, with the 2nd one not having as much redundancy (if the 3TB drive fail, you lose it all)

Or create a mirror using the 1TB partition on the 2TB and 3TB disks. The remaining 1TB space on the 3TB disk would be lost unfortunately. Or use is as-is. A plain 1TB volume with no redundancy.

Performance won't be the best obviously, but you'll maximise the usable space, you get all the benefits of a raidz1 and mirror.

The other issue is that you'll lose compatibility with other OS, and not all ZFS implementation will recognise this setup.

You'll need to do everything using the command line

 

[jyavenard]

I missed that you didn't want to lose the existing data on one of the 2TB disk.

What I describe above is still okay. You'll be able to replace one of the partition with the whole disk once you've copied your file across.

Depending on the order in which you are doing things; and if you're patient enough you could end up with a raidz1 with 2TB*3
plus 2*1TB mirror

So start with creating 1TB partitions on the 3TB and 2TB disks
Create Raidz1 pool using those 2 1TB partition + whole 1TB disk

Copy your file across from the 2TB disks; once done. Degrade the array by replacing the 1TB disk with the 2TB disk (whole disk)

Once resilvering (rebuilding) is done; repeat : degrade the array, remove the 1TB partition; and replace it with the same whole drive.

Do that again with the 3TB disk

You could create then a 1TB mirror using remaining 1TB on the 3TB disk, and the 1TB disk.

Will take a while... And there's the risk that should one disk fail when copying things across; you lose it all.

 

[strange]

Actually my situation is not exactly like the OP: i have 3 x 1TB drives, 2 x 1.5TB drives and 2 x 2TB drives, none of which have any data on them. I am going for a RAIDZ2 solution of all 7 disks, giving me 5TB of usable space. I plan to grow the pool slowly over time by replacing smaller drives with larger ones. You idea is interesting though - i could basically create a 1TB partition on each drive and then use the GUI to create the raidz2 pool, right? Only thing is, gpart is what is giving me the error ;)

 

[Dusan]

You idea is interesting though - i could basically create a 1TB partition on each drive and then use the GUI to create the raidz2 pool, right? Only thing is, gpart is what is giving me the error ;)

Not via GUI, if you want to go in this direction then you need to do everything via CLI. I'm actually not really sure how would GUI react to such setup.
As for the gpart failing. You can try either:

1. Install FreeNAS 8.3.2 and create the pool there and then upgrade back to 9.1.1. The old ZFS Volume Manager was more permissive than the current one.
2. If you run gpart as root and you still get the error you can try running sysctl kern.geom.debugflags=0x10 to disable various geom safety checks (geom tries to prevent you from damaging existing filesystems, but it may be confused for whatever reason).

 

[strange]

Thanks Dusan, that worked. Onwards to victory!

 

[MtK]

Thank you very much paleoN!
I had temporarily installed FreeNAS 8.3 that allows me to create raidz1 with different sized disks, after I had upgraded to 9.1.1, but if I needed to create a new raidz1 pool I could not.

I've deleted the pool via the GUI, than

Code:

gpart create -s gpt ada0
gpart add -b 128 -i 1 -t freebsd-swap -s 2G ada0
gpart add -i 2 -t freebsd-zfs ada0
gpart create -s gpt ada1
gpart add -b 128 -i 1 -t freebsd-swap -s 2G ada1
gpart add -i 2 -t freebsd-zfs ada1
gpart create -s gpt ada2
gpart add -b 128 -i 1 -t freebsd-swap -s 2G ada2
gpart add -i 2 -t freebsd-zfs ada2

Now should I proceed like this?

Code:

zpool create -f /mnt/Media raidz ada0p2 ada1p2 ada2p2

I've tried the same with a 6 drive raidz2.
partitioned/created the swap and then:

Code:

# zpool create -f /mnt/Storage raidz2 ada0p2 ada1p2 ada2p2 ada3p2 ada4p2 ada5p2
cannot create '/mnt/Storage': invalid character '/' in pool name
use 'zfs create' to create a dataset

when I tried this again I got:

Code:

# zpool create -f Storage raidz2 ada0p2 ada1p2 ada2p2 ada3p2 ada4p2 ada5p2
invalid vdev specification
the following errors must be manually repaired:
/dev/ada0p2 is part of active pool 'Storage'
~# zpool status
  pool: Storage
 state: ONLINE
  scan: none requested
config:
        NAME        STATE     READ WRITE CKSUM
        Storage     ONLINE       0     0     0
          raidz2-0  ONLINE       0     0     0
            ada0p2  ONLINE       0     0     0
            ada1p2  ONLINE       0     0     0
            ada2p2  ONLINE       0     0     0
            ada3p2  ONLINE       0     0     0
            ada4p2  ONLINE       0     0     0
            ada5p2  ONLINE       0     0     0
errors: No known data errors

 

[MtK]

solved this by running:

Code:

zpool create -R /mnt -f Storage raidz2 ada0p2 ada1p2 ada2p2 ada3p2 ada4p2 ada5p2

 

[gpsguy]

You should have used the GPTID's instead.

See this recent thread for more details.

solved this by running:

Code:

zpool create -R /mnt -f Storage raidz2 ada0p2 ada1p2 ada2p2 ada3p2 ada4p2 ada5p2

 

[MtK]

fixed:

Code:

# zpool status
  pool: Storage
 state: ONLINE
  scan: none requested
config:
 
        NAME                                            STATE     READ WRITE CKSUM
        Storage                                         ONLINE       0     0     0
          raidz2-0                                      ONLINE       0     0     0
            gptid/8a29a838-59d6-11e3-b1ac-002590d91105  ONLINE       0     0     0
            gptid/8ff0f4ba-59d6-11e3-b1ac-002590d91105  ONLINE       0     0     0
            gptid/900cdead-59d6-11e3-b1ac-002590d91105  ONLINE       0     0     0
            gptid/90258a06-59d6-11e3-b1ac-002590d91105  ONLINE       0     0     0
            gptid/903cd8c3-59d6-11e3-b1ac-002590d91105  ONLINE       0     0     0
            gptid/9186c47b-59d6-11e3-b1ac-002590d91105  ONLINE       0     0     0
 
errors: No known data errors

 

[strange]

Ok, so my raidz2 pool has been set up correctly as per the above, but when i go to the gui and click view volumes, nothing shows. I've tried importing, but it finds nothing. What am i missing here? Is there another step?