Raidz3 OR (raidz2 and hot spare) for VM storage?

kazix

Dabbler
Joined
Jun 17, 2019
Messages
18
Hi,
Server for FreeNAS: HP ML 350 G5, controller e200i (each disk as RAID0).
Disk: 8 x 300 GB SAS.
Destination: iSCSI for VM.

It is better add all disk to pool, example:
- RaidZ2 => (8 x 300) - (2 x 300) = 1800 (two disk can die)
- RaidZ3 => (8 x 300) - (3 x 300) = 1500 (tree disk can die)

or better create RaidZ with dedicated Hot Spare Disk
- RaidZ => 6 x 300 = 1800 and two Hot Spare Disk

Read this forum and see that better for VM are mirror vdev.
If i have only 8 disk - how create vdev mirror?

(2 x 300) + (2 x 300) + (2 x 300) = 1200 and two Hot Spare Disk?
but can be one logical disk?

Maybe someone help me?
 

garm

Wizard
Joined
Aug 19, 2017
Messages
1,555
Don’t, if this is a business or other critical system, don’t. Find someone with experience in ZFS systems administration and redo the system design.

If this is a hobby project, don’t. Spend 40 hours reading the forum and underlying documentation. Especially (but not limited to) the hardware recommendations. ZFS is not just a file system.
 

sretalla

Powered by Neutrality
Moderator
Joined
Jan 1, 2016
Messages
9,702
You should probably understand that hot spares probably aren't as helpful as you think.

A hot spare will only be automatically brought into play if the disk that needs replacing goes completely offline (so not when it starts experiencing errors, but when it's completely dead).

This can be useful in the case where you can't physically get to the location where the server is very quickly (remote support of a location with a plane ride in between). In almost all other cases, you're better off having those drives outside the system (cold spares) and monitor the system and intervene when you start to see errors by doing the manual steps needed to replace the drive.

(2 x 300) + (2 x 300) + (2 x 300) = 1200 and two Hot Spare Disk?
but can be one logical disk?
Technically you can do this, but any disk lost will lose all your data (there's no redundancy in that setup where you have 1200 GB available).

If you mean each of the 2 x 300 as mirrored pairs (mirrored VDEVs), then yes, that's something that will work, allowing for OK IOPS and allowing for one disk to be lost in each VDEV before all data is lost.

As suggested by @garm you need to learn about ZFS and FreeNAS before you decide on the right design for you and put anything important on it.
 
Joined
Oct 18, 2018
Messages
969
Hi @kazix, welcome to the forums.

The advice above is certainly correct, but hopefully not too discouraging. FreeNAS uses zfs and both are designed around data integrity and are quite different than other filesystems and nas solutions.

For one, zfs uses software raid rather than hardware raid, and the terminology is different. Here is a nice terminology primer that may help clarify a little bit.

Here is also an explanation of pools, vdevs, and more.

Briefly, zfs exposes pools to the user on which to store data. Pools are made of up 1 or more vdevs striped together. Vdevs come in several forms and are made up of 1 or more disks. Importantly, if a single vdev within a pool fails you lose your entire pool and it is likely unrecoverable. Therefore, redundancy within a vdev is important. Whether you go with mirrors, RAIDZ1, RAIDZ2 or RAIDZ3 vdevs depends on your use case and budget.

(2 x 300) + (2 x 300) + (2 x 300) = 1200 and two Hot Spare Disk?
but can be one logical disk?
As @sretalla said, this will definitely work. In the terminology above you'd have 1 pool to store data on composed of 3 vdevs. Each vdev will be two disks and can tolerate a single drive failure. You'd also have two hot spare disks.

FreeNAS is very good at keeping your data safe but isn't a simple solution that you can build and forget about. Looks like you've already done a bit of research, hopefully some of the responses here have helped you narrow in your build.
 

kazix

Dabbler
Joined
Jun 17, 2019
Messages
18
@garm @sretalla - what wrong with you, dude?
it's not NASA or Russian Nuclear server, it even not computer in hospital - if fail, nobody die
i just want try this solution - FreeNas with ZFS is smart, very smart
i always suprise people, who say: don't do this, don't do that, don't touch, don't install linux, don't install FreeBSD, don't use FreeNAS :)
why not try this? believe me, nobody die - it only my time and yours help :) (thanks @PhiloEpisteme )

back to reality:
I don't have different server and at this moment i don't have budget to buy different hardware.
Maybe i wrong start this post, maybe should be:

How do you build your storage if you have 8 disk 300 GB SAS.

My first thing - RaidZ3, but for block device better use mirror/striped.
How to build secure ZFS mirror/striped on 8 disk?
Or maybe better use hardware RAID for this or mix hardware RAID and ZFS?

Tomorrow i get this server.
 

sretalla

Powered by Neutrality
Moderator
Joined
Jan 1, 2016
Messages
9,702
Or maybe better use hardware RAID for this or mix hardware RAID and ZFS?
Don't do that. Hardware RAID kills the visibility of disk operations and ZFS can't do its work properly. Many people have lost their data going down that road.

@sretalla - what wrong with you, dude?
I'm trying to help you to understand how not to lose your data... maybe that wasn't what you wanted (so ignore me if necessary).

How to build secure ZFS mirror/striped on 8 disk?
I didn't want to spend much time googling, so here's how to do it in the legacy GUI:
https://www.ixsystems.com/community...-striped-mirrors-aka-raid10-in-the-gui.62228/
I'm sure you could find a similar guide for the modern GUI if you wanted to.
 
Last edited:
Joined
Oct 18, 2018
Messages
969
Or maybe better use hardware RAID for this or mix hardware RAID and ZFS?
I agree with @sretalla, you'll want to avoid mixing hardware RAID with FreeNAS.

How d you build your storage if you have 8 disk 300 GB SAS.
This is a great question and depends a lot on how you plan to use your machine, your budget, and how important redundancy is for you.

Tomorrow i get this server.
Before you purchase anything it may be worth posting all of your complete parts list including model numbers. Also include what you own vs what you'll buy. If you also expand on how you plan to use it and what your approximate budget is folks on the forums can help narrow in your build. In general I find folks on the forums very good at helping people make the most of their budget for their specific use case.
 

garm

Wizard
Joined
Aug 19, 2017
Messages
1,555
what wrong with you, dude?
If you put single disk raid0 arrays as devices in a vdev you will eventually lose that vdev as ZFS is tricked to believe its writing to a disk when in fact it’s talking to a controller. ZFS does several things with the assumption that something that presents itself as a disk is in fact a disk. Forum is full of people with your setup complaining that they lost there stuff, but what is even worse, you actively cripple ZFS. If you do not want data integrity then just use what ever you got as it was intended..
 

HoneyBadger

actually does care
Administrator
Moderator
iXsystems
Joined
Feb 6, 2014
Messages
5,110
If this is for a purely testing and learning environment, people may be less shocked at the hardware you're using here. As it is, an iSCSI VM workload is one of the most demanding for ZFS to handle, and your hardware is definitely not what I would consider "fit for purpose." You're likely to get a bad impression of FreeNAS/ZFS in general by using this, but a large number of the problems will ultimately resolve back to the hardware. It's like trying to carve a prime rib with a serving spoon. Can it be done? Yes, but the results will definitely not do the source product justice.

Consider that some of the other responders may be trying to protect not just your data, but the time you spend working on this and trying to extract blood from a stone. You are proposing to use hardware from 2008 in 2019, as well as a embedded entry-level RAID controller that is known to offer poor performance even when compared to other RAID controllers.

Moving up even one generation to the HP G6 series gets you into DDR3 memory, off of the old FSB-based technology, and into supporting the E5600-series Xeons which are much better on a CPU performance perspective; adding a supported non-RAID HBA like an LSI 9211-8i or an equivalent reflashed OEM controller (Dell PERC H200) would put you in a much more performant and ultimately supported platform.

However with the hardware you have, I cannot in good conscience suggest you use it for anything other than learning the GUI of FreeNAS, and I certainly wouldn't rely on putting data you care about on those iSCSI LUNs. If this is the hardware you have to use, and it's going to be for data you care in any way about, avoid ZFS. Build a hardware RAID, set up a certified OS on the platform, and present storage that way.
 

IQless

Contributor
Joined
Feb 13, 2017
Messages
142
It's like trying to carve a prime rib with a serving spoon. Can it be done? Yes, but the results will definitely not do the source product justice.
An exceptionally beautiful way to describe it, thank you for that! :)
 

kazix

Dabbler
Joined
Jun 17, 2019
Messages
18
@HoneyBadger
your kill me this spoon!!!

However with the hardware you have, I cannot in good conscience suggest you use it for anything other than learning the GUI of FreeNAS, and I certainly wouldn't rely on putting data you care about on those iSCSI LUNs. If this is the hardware you have to use, and it's going to be for data you care in any way about, avoid ZFS. Build a hardware RAID, set up a certified OS on the platform, and present storage that way.
this is the smartest answers, really.
FreeNAS is fantastic, but as you all say: Not this time (in short).
Plan is simple: first - test FreeNAS on this Server, second: look for different OS without ZFS :(

@PhiloEpisteme - i get it, that mean all services and files are move to another server, i can install different OS
Next time i will ask on the forum before buy anything.
 

kazix

Dabbler
Joined
Jun 17, 2019
Messages
18
ok, i just test my configuration.
8 Disk SAS (each as RAID 0 on Hardware RAID) give me almost 900 GB and two Spare. (Mirror)
Theoretically (on different hardware) - this will be a good solution?

RaidZ2 with one Spare give me almost 1,3 TB. (RaidZ2)

When i can't use FreeNAS i see how much i love it. :mad:
 

Attachments

  • Mirror.png
    Mirror.png
    68.5 KB · Views: 436
  • RaidZ2.png
    RaidZ2.png
    66.1 KB · Views: 417
Joined
Oct 18, 2018
Messages
969
8 Disk SAS (each as RAID 0 on Hardware RAID) give me almost 900 GB and two Spare. (Mirror)
Theoretically (on different hardware) - this will be a good solution?
No, this is not a good solution. As was mentioned previously, do not mix hardware RAID with FreeNAS or zfs at all. Ideally you would just plug your drives straight into your motherboard or get an HBA flashed to IT mode and plug them straight into that. Turn off hardware RAID functionality.
 

kazix

Dabbler
Joined
Jun 17, 2019
Messages
18
@PhiloEpisteme i know that, im just testing - what i can say, it works.
like @HoneyBadger said - im looking something else, but why not testing?

BTW i ask about logical disk, if i have connect this 8 disk to controller without RAID.
This scenario: 3 x Mirror + 2 Spare is good?
 

kazix

Dabbler
Joined
Jun 17, 2019
Messages
18
maybe SPARE is my habit and always on Hardware RAID i given one disk "extra".
If something go wrong and one disk die - raid rebuild automatically.
i think, it is good

ZFS offer RAIDZ2 and RAIDZ3 - this is something new for me.
My first question was: better use RAIDZ2 org RAIDZ3 or RAIDZ and SPARE.
In this case (for block device) better use MIRROR, ok - so try think something, theoretically.
this is only 8 disk, can't add more - without SPARE will be 4 x MIRROR?
 

garm

Wizard
Joined
Aug 19, 2017
Messages
1,555
ZFS will generally not just drop a drive without prior notice. Sure a drive can have a catastrophic mechanical or firmware failure but in my experience there will be S.M.A.R.T or ZFS checksum indications prior to such a failure. I’ve been using ZFS more or less sins it’s introduction to FreeBSD and my experience is that by choosing your hardware a bit careful then you should ample time to replace a dying disk. Coupled with proper backups and the risk to data is significantly reduced without spares. Using the spares for backup is probably a better investment given that you listen to the signals from your machine.

If you do not have effortless access to the hardware then the spare might start making sense.
 
Joined
Oct 18, 2018
Messages
969
@PhiloEpisteme i know that, im just testing - what i can say, it works.
like @HoneyBadger said - im looking something else, but why not testing?
You're right, certainly good for testing. I guess if you have no other way to connect the drives its a good free way to check out the OS etc. If you can connect the drives without the hardware RAID even better. :) I guess the fear is you'll have some early issue because of the mix of hardware raid and software raid and get a bad impression of FreeNAS. haha

Also, I think @garm's explanation of how you can usually go without spares is a great one. If you keep a backup, set up regular scrubs, short and long SMART tests, email notifications, and you have easy access to the machine your data should be safe without spares.

without SPARE will be 4 x MIRROR?
Yeah, that would be great. You could pick up 1 extra HDD to have in your desk for use as a replacement when one of the 8 in the machine go bad.
 
Top