SOLVED FreeNAS volume layout advise

[Fred974]

Hi all,

I have been using FreeNAS for over 3 years and I though I was a goodish user and knew my way around it untill I started reading @cyberjock document on explaining VDev, zpool, ZIL and L2ARC
https://drive.google.com/file/d/0BzHapVfrocfwblFvMVdvQ2ZqTGM/view

I do encourage people to go and read it as it is I a really well written master piece :)

This also left me with a lot of question around how to design my new FreeNSD server..

I have a Dell PowerEdge C2100 with 48GB ECC + 24x 146GB SAS HDD. I know that they are old and small and I am planning to replace them for bigger one in time one at a time as per the doc advises.

The aim of that new FreeNAS box will be to provide storage to our new XCP-NG cluster via iSCSI target on FreeNAS. For security and reliability, we want to use raidz3.
I have been reading the calomel article on ZFS Raidz Performance, Capacity and Integrity and they have provided the following benchmarking results for raidz3

Code:

11x 4TB, raidz3 (raid7),	   30.2 TB,  w=552MB/s , rw=103MB/s , r=963MB/s
12x 4TB, raidz3 (raid7),	   33.6 TB,  w=452MB/s , rw=105MB/s , r=840MB/s
23x 4TB, raidz3 (raid7),	   74.9 TB,  w=440MB/s , rw=157MB/s , r=1146MB/s
24x 4TB, raidz2 (raid6),	   82.0 TB,  w=434MB/s , rw=189MB/s , r=1063MB/s
24x 4TB, raidz3 (raid7),	   78.1 TB,  w=405MB/s , rw=180MB/s , r=1117MB/s

@cyberjock doc said that

RAIDZ3 should have the total number of drives equal to 2n + 3. (ie 5, 7, 11, etc drives for the VDev)

and that

It is not recommended that VDevs contain more than 11 disks under any circumstances.

So I am guessing that creating a signle zpool of 24 HHD is not recommended..

Which configuration should I be using to have a balance between performence an reliability?
Should I be creating 2x zpool of 2 vdev with 11 disk per vdev on raidz3?

Also does FreeNAS has the notion of 'spare drive' for auto replacement?
I am a bit confused her and I would really appreciate further advise from experience user please

 

[Stux]

Well, what is the cluster using the iscsi disk for?

Because the answer you want is probably mirrors.

Assuming performance matters.

 

[danb35]

RAIDZ3 should have the total number of drives equal to 2n + 3. (ie 5, 7, 11, etc drives for the VDev)

The 2^n+p rule is long (long) since outdated.

Which configuration should I be using to have a balance between performence an reliability?

Assuming general storage, I'd probably go for one pool consisting of three, eight-disk RAIDZ2 vdevs.
Scratch that, as I'd missed the part that says you're going to use this for block storage. If that's needed, you probably care more about IOPS than about throughput, and I'd concur that mirrors are probably better.

Also does FreeNAS has the notion of 'spare drive' for auto replacement?

It does, though it's of questionable value unless the server is going to be inaccessible to anyone who could manage swapping a disk.

 

[MatthewSteinhoff]

To answer your question directly, I agree with all of the above: number of drives is mostly irrelevant now and mirrors are probably what you want.

I'll add that you can get a 500G SSD for the cost of a 146G SAS drive and the SSD will have five times the bandwidth and a bazillion times the IOPS. I know you already have the SAS drives but I'd encourage you to run the numbers with SSDs for a VM use case.

More importantly (to me)...

XCP-NG cluster

We're also looking to replace XenServer 6.x with XCP-ng instead of going to the Citrix-crippled 7.x series. I'd love for you to offer an quick opinion as to have the XCP-ng project is going to survive and if it seems ready for business.

Cheers,
Matt

 

[melloa]

I'd love for you to offer an quick opinion as to have the XCP-ng project is going to survive and if it seems ready for business.

I'd second and third that. Although a home/hobbyist user, I run all on VMs under ESXi and bhyve, but am always looking for better things.

If @Fred974 has any insights to offer, I'd be very appreciative.

 

[Fred974]

I'll add that you can get a 500G SSD for the cost of a 146G SAS drive

Will any SSD do or do I need enterprise one? Could share some links please.

We're also looking to replace XenServer 6.x with XCP-ng instead of going to the Citrix-crippled 7.x series. I'd love for you to offer an quick opinion as to have the XCP-ng project is going to survive and if it seems ready for business.

Most people I know that was running xenserver has jumped over to XCP-ng and have no issue so far.
One of them is running a hosting business and is enjoying the unrestricted features. I also managed to compile xen orchestra from source ans now have the full xen orchestra running just fine.

I think there is a massive community support behind the project and I cannot see it going bust..Only a couple on month in the project, and they are already production ready.
Also they got so much for financial support that they are now able to develop beyond what xenis offering..

At the moment, I rung my hosting business on FreeBSD jails and bhyve..
I love *BSD but Xen Orchestra is offering so much more in term of 'easier life' and less headache.

Now going back to my question, you all sugested mirror raid.. I am confuse here. How do you do a mirror raid in FreeNAS and if I loose 1 disk, how reliable is the mirror raid?
I need good reliability

 

[danb35]

How do you do a mirror raid in FreeNAS

Is this a trick question? You select "mirror" in the volume manager.

if I loose 1 disk, how reliable is the mirror raid?

If you lose one disk, the remaining disk continues to work. That's how mirrors work. But if you lose both disks in one mirror, your pool is lost.

 

[MatthewSteinhoff]

Will any SSD do or do I need enterprise one?

Any SSD will work. Every SSD will be faster than a conventional spinning drive by a meaningful amount.

Once again, this is a case where you need to run the numbers for your use case.

We use the cheapest SSDs we can find. Because they are so much faster than conventional disks, we don't care to figure out which is absolute fastest. We don't buy the highest endurance SSDs because we are okay with replacing them as they wear out. Performance and capacity are rapidly increasing while prices are falling. We're okay with buying 500GB Samsung EVOs for $90 and having them burn out in a couple years. By the time we go to replace them, we'll be able to get 2TB SSDs with better performance for about the same price.

If you live in a write-heavy world and would be burning out drives more often, splurging on enterprise SSDs might be a thing you do.

We purchased four ADATA 960G SSDs a couple years ago because they were cheaper than anything else. We run them as a striped mirrors for our VMs and they are under 50% capacity. Depending on which SMART data you believe, they have half their life left or 90% of their life left. In any case, when they die in another year or two, we'll have gotten our money's worth and won't hesitate to replace them with whatever is available for a low price.

Some forget the I in RAID stands for Inexpensive. Not us.

Cheers,
Matt

 

[Fred974]

Is this a trick question? You select "mirror" in the volume manager.

If you lose one disk, the remaining disk continues to work. That's how mirrors work. But if you lose both disks in one mirror, your pool is lost.

Hi, sorry, I didn't word that question very well... Yes I know how to create mirror raid lol.
Are you suggesting using all 24 disk on I pool in mirror?

Sent from my VTR-L09 using Tapatalk

 

[danb35]

Are you suggesting using all 24 disk on I pool in mirror?

Yes, I'd be looking at a single pool of striped mirrors--12 two-disk mirror vdevs striped together. If you need more redundancy, you could use 8 three-disk mirror vdevs instead, but you'd be using 2/3 of your total capacity for redundancy that way.

 

[Stux]

Also means you will get a capacity boost as soon as you replace two drives in the same vdev.

 

[Fred974]

I'd be looking at a single pool of striped mirrors--12 two-disk mirror vdevs striped together. If you need more redundancy, you could use 8 three-disk mirror vdevs instead, but you'd be using 2/3 of your total capacity for redundancy that way.

When using three-disk mirror vdevs, can i loose 2 disks and still be running?

 

[danb35]

When using three-disk mirror vdevs, can i loose 2 disks and still be running?

Sure can.