Ideal Layout For Future Expansion on a Small Budget

[suppaduppax]

I am currently running Freenas on a Dell T3600 with a 3x750gb RaidZ1 configuration. I back up the more important data to a Dell Optiplex FreeBSD box using 2x500gb Stripe as a temporary backup solution as I improve my home server setup.

I recently purchased a HP DL180 G6 server to migrate my Freenas box to. The T3600 only has room for 3 drives (you can squeeze in 4 uncomfortably) so now with the new server, I’ll have 12 bays to work with. I have 5x1tb drives and an additional 3x2tb I can use across my new server and backup machine (I will most likely sell the T3600 to offset the money spent on the DL180 after i have everything set up). Overtime I will have access to more used 1tb hdds through a friend of mine. I have an 8tb hdd coming in that I will use as a secondary backup of my critical files, copying to it every month-ish and then storing it offsite.

My budget is tiny as you can probably tell from all the used hardware including the hdds.

I was wondering what the best long term zfs layout for both a freenas server and backup server would be.

What I use my current server for:
Media, Plex/Emby Server
Sonarr/Radar/Transmission
Backup for 2 desktop computers
NextCloud
VMs to run Linux for game servers
Learning and playing around with server stuff

Here are my thoughts:

Mirrors
Start off with 4x1tb (2 2x1tb) striped mirrors in the DL180. Have a 3way 3x2tb mirror for the backup server. This way I can expand the main server easily over time as I get more 1tb drives. When the backup server runs out space I can buy 2 or 3 larger drives to upgrade that pool and use the 2tb drives that were replaced inside the main server to upgrade its capacity.

RaidZ1
When I set up my first FreeNAS server I used the Dell optiplex and I went with a 3x750gb raidz1 configuration because that’s all I could fit into the chassis. When I got the T3600, I migrated my Freenas server over and I turned my old server into a backup server. Now that I have my 12 bay HP server, an option would be to just migrate the current server onto it and use additional 1tbs to create more 3x1tb vdevs to expand the pool. I know this is really bad for redundancy but it gives me more usable space without having to rebuild and destroy my current pool.

RaidZ2
With a 5x1tb RaidZ2 configuration I can have decent storage with good redundancy but upgrading won’t be as easy. It also leaves me with 2 bays leftover in the 12 bay server after adding another 5 drive vdev but I can use that for cache down the road. I can still do a 3way 2tb mirror for my backup server using this layout.

If I used 6x1tb hdds to make a RaidZ2 configuration (it would actually have to use one of the 2tb hdds to get 6 drives— 5x1tb + 1x2tb) I would have a decent amount of usable storage without sacrificing redundancy. But expanding the pool becomes costly down the road since I would have to have more drives upfront. It also leaves me with only a 2way mirror for my backup server.

Other considerations:
I can wait a few months to somehow gather enough 1tb drives to make a 12x1tb RaidZ2 or RaidZ3 configuration to have the most usable space right up front. Upgrading will be much much later on, if at all, and by then it might be worth just buying a DAS or come up with a new solution when the time comes. The biggest downside would be performance and also that I’m coming close to filling up my current server and I’m not sure I’d be able to wait to have enough drives to do this.

I’m still trying to figure out how much redundancy I’m actually going to need since I will have 2 different backups and can restore it all if worst comes to worst.

I know this is all extra complicated with a minimal budget but hopefully someone can offer some insight or point out some things I haven’t even considered.

 

[Chris Moore]

I can wait a few months to somehow gather enough 1tb drives to make a

Why are you looking at 1TB drives?

 

[suppaduppax]

Cuz they’re free lol

 

[Chris Moore]

LOL... Can't beat free.

 

[suppaduppax]

No you can’t! Feel like such a hobo but my budget is minuscule. Already pushed it with the HP DL180 G6 I just bought

 

[Heracles]

Hi Suppaduppax,

indeed, it is hard to beat free stuff. But here, it may impact you by forcing you to load your pool too much, by forcing you to reduce your redundancy or by turning your pool in a mix-N-match of different vDevs.

So here, you have to make your priority :
--Cost
--Speed / efficiency
--Redundancy
--Capacity

You can not have it all... So here, I understand that cost is your first priority. Ok.

What is your second ? Third ? and what is it that you are clearly ready to sacrifice in the name of the others ?

A few options would be :
Raid-10 with 1 or 2 hot spare
2x 6 drives RaidZ2
1x 8 drives RaidZ2 for 1 pool ; 4 bays ready for a second pool

An important thing to learn about FreeNAS is how critical planning is and how difficult it is to change things once done.

Good luck with your setup,

 

[suppaduppax]

An important thing to learn about FreeNAS is how critical planning is and how difficult it is to change things once done.

Trust me I know. I’ve been scouring the forums and googling around for months to gain some insight as to what to do.

Raid-10 with 1 or 2 hot spare

Would this be a pool of mirrors? What is raid 10 in zfs terms?

forcing you to load your pool too much

What do you mean by load my pool too much? Having too much space I won’t use right away?

You can not have it all... So here, I understand that cost is your first priority. Ok.

What is your second ? Third ? and what is it that you are clearly ready to sacrifice in the name of the others ?

You’re right, cost is #1 priority.

I guess the priority list would go like this:
1 - cost
2 - capacity
3 - redundancy
4 - speed / efficiency

It’s the capacity over redundancy that I’m a little unsure about and what makes it a difficult choice is expandability. If I go with mirrored pairs I wouldn’t have good redundancy and although capacity looks bad, being able to expand it quite easily really makes up for it. If I go RaidZ2 I have much better redundancy and although I may have some added capacity up front, it suffers long term due to it being difficult to expand. Speed and efficiency mean very little to me and can be sacrificed right off the bat.

I guess I’m kind of in a bit of a back and forth with myself between mirrored pairs and raidz2 and I can’t seem to make up my mind.

 

[Chris Moore]

Would this be a pool of mirrors? What is raid 10 in zfs terms?

Yes, a pool of mirrors like this:

Code:

  pool: iSCSI
state: ONLINE
  scan: scrub repaired 0 in 0 days 00:08:32 with 0 errors on Tue Jun  5 00:08:33 2018
config:

        NAME                                            STATE     READ WRITE CKSUM
        iSCSI                                           ONLINE       0     0     0
          mirror-0                                      ONLINE       0     0     0
            gptid/a1ae863b-2a0a-11e8-bbf6-002590aecc79  ONLINE       0     0     0
            gptid/a41c596b-2a0a-11e8-bbf6-002590aecc79  ONLINE       0     0     0
          mirror-1                                      ONLINE       0     0     0
            gptid/a9cd1b9f-2af2-11e8-8661-002590aecc79  ONLINE       0     0     0
            gptid/ab1dbee6-2af2-11e8-8661-002590aecc79  ONLINE       0     0     0
          mirror-2                                      ONLINE       0     0     0
            gptid/0243b38d-2a0b-11e8-bbf6-002590aecc79  ONLINE       0     0     0
            gptid/05e0a0d6-2a0b-11e8-bbf6-002590aecc79  ONLINE       0     0     0
          mirror-3                                      ONLINE       0     0     0
            gptid/0ca57011-2a0b-11e8-bbf6-002590aecc79  ONLINE       0     0     0
            gptid/0dce1e70-2a0b-11e8-bbf6-002590aecc79  ONLINE       0     0     0
          mirror-4                                      ONLINE       0     0     0
            gptid/11269a55-2a0b-11e8-bbf6-002590aecc79  ONLINE       0     0     0
            gptid/1237afc0-2a0b-11e8-bbf6-002590aecc79  ONLINE       0     0     0
          mirror-5                                      ONLINE       0     0     0
            gptid/70a040a4-5163-11e8-a76f-0cc47a9cd5a4  ONLINE       0     0     0
            gptid/194a8867-2a0b-11e8-bbf6-002590aecc79  ONLINE       0     0     0
          mirror-6                                      ONLINE       0     0     0
            gptid/1ee68b6a-2a0b-11e8-bbf6-002590aecc79  ONLINE       0     0     0
            gptid/229c431a-2a0b-11e8-bbf6-002590aecc79  ONLINE       0     0     0
          mirror-7                                      ONLINE       0     0     0
            gptid/25ec5bdf-2a0b-11e8-bbf6-002590aecc79  ONLINE       0     0     0
            gptid/275c216f-2a0b-11e8-bbf6-002590aecc79  ONLINE       0     0     0

errors: No known data errors

Some people (not me) use RAID-10 to describe it although ZFS does not use that terminology.
In ZFS all vdevs are striped together in what might be called a RAID-0. All parity / redundancy is done at the vdev level, so if your vdevs were RAIDz2, many people think of that as being the same as RAID-60, even though that is not the terminology that ZFS uses.

I guess the priority list would go like this:
1 - cost
2 - capacity
3 - redundancy

Those three usually compete with one another. Some people even draw that out as a triangle and say that you must choose. A kind of middle ground, I think, is RAIDz2 with 6 disks. It gives you use of most of the capacity of four disks while giving you the protection from failure that two disks can fail without data loss, not that you want to be in that position, it just means that you can have a disk fail and still be protected from a disk failure. It is not the best performing option and it does cost six disks to get almost four disks worth of capacity. It is always a trade at some level.
Here is an article with a discussion that might be interesting, but I don't know if it will help you decide or just leave you with more questions:

The 'Hidden' Cost of Using ZFS for Your Home NAS
https://louwrentius.com/the-hidden-cost-of-using-zfs-for-your-home-nas.html

 

[Heracles]

Hi again,

Indeed, Raid-10 is a pool of mirrors. You create a set of mirrors and join them in a single pool and that is the equivalent of Raid-10.

Loading your pool too much means using too much space in your pool. The higher the load, the slower and poorer performance is the pool. It is easy to figure out :
If your pool is used at 25% or less, almost anytime FreeNAS needs to write something to the disks, it can do it right away.
If the pool is loaded between 25 up to 50%, most of the time FreeNAS can write as soon as needed but here and there, it will need to wait for the drives to find free space because at that very moment, the drive were at a sector that was already used.
Should your used space reaches between 50 to 75%, then most time FreeNAS will have to wait for the disks to find a free spot before writing.
Above 75%, FreeNAS is always waiting all the time about the drives for them to find one of the few last spot where indeed, something can be written.
If your pool reaches 90% of used space, you even enter a danger zone where you may crash your server and end up unable to access / use it.

So from your priorities, you can plan your setup for a pretty high usage of your pool capacity because speed / efficiency is the feature you are ready to sacrifice in for the sake of others.

Also, you will rather loose your data (sacrifice redundancy) in the name of an increased capacity.

So next step is to evaluate of much you need as of now and how much you will need before being able to put more money in your setup.

So as of now, I would say :
Make yourself a first vDev of 2x 2TB drives as mirror
Also make another vDev of 5x 1TB drives as RaidZ1
Use your 3rd 2TB drive as a hotspare

So according to your priorities :
1st : No cost - you already own all the drives
2nd : Capacity you will have 2+4 = 6 TB of capacity
3rd : Redundancy : Both of your vDev have only a single redundant drive but are covered by a hot spare in case of a problem
4th : Speed / efficiency ; Such a mix-N-match of vDev is not great and you will probably load the pool at a high %

With the 4 extra bays, you can create more mirrors of 1 ou 2 TB drives and your hot spare will be able to double cover them as well.

Should you wish to put some money in your server, you can buy yourself a set of 12x 6+ TB drives.
When ready, you create a temporary mirror of 2x 6TB (8 used bays + 2 new ; 10 used ; 2 free)
You move your data to that temp mirror.
You destroy your actual pool and remove your old drives. (2 used bays ; 10 free)
You create a first 6x 6TB RaidZ2 vDev and a new pool ( 8 used bays ; 4 free)
You put back your data online in that new pool
You remove the temp mirror and create a second RaidZ2 vDev you add to your pool.

And you are then ready with a 12 drives pool made of 2x 6 drives RaidZ2 vDevs.

At the end, you can not have it all : cost, capacity and redundancy fight each other. Be very aware of your choice and to sacrifice redundancy means being ready to loose it all...

 

[Chris Moore]

I guess I’m kind of in a bit of a back and forth with myself between mirrored pairs and raidz2 and I can’t seem to make up my mind.

I had a similar concern when I started out and the thing that made the decision for me was the dual disk redundancy of RAIDz2. I have been in situations where I had two disks go bad at the same time and I didn't, above all, want to loose my data to a drive failure. I even used RAIDz3 for a while.