economical home storage music production parts list proposal

[jgreco]

Mirrors are not better than RAID-z2 unless you have a need for speed that you didn't discuss.

Mirrors are SO much better than RAIDZ2. Except if you want cheap.

Mirrors have a 100% predictable amount of overhead, compared to the variable parity consumption of RAIDZ.

Mirrors allow for the decrease or increase in amount of redundancy on the fly.

Mirrors allow you to go beyond RAIDZ3 levels of data protection.

Mirrors allow for additional read IOPS from each component.

Etc.

Mirrors are SO much better than RAIDZ2. However, most people won't want to eat the cost, and most also don't need to.

 

[ramar]

Mirrors are SO much better than RAIDZ2

thanks. you must have written the article i read a few days ago. may i ask, if a mirrored drive needs to be replaced, can i remove it a place a new one (or a bigger one) in? if a bigger one is added, am i correct in thinking it would not increase the usable storage until all the drives have been replaced or will the usable space increase when all the mirrors are replaced?

 

[jgreco]

Well, generally:

1) You can add the new disk to the vdev, allow it to rebuild (which allows data to be retrieved from both a failing drive and also the remaining good one or ones), and then remove the failing disk from the vdev. This allows you to maintain as much redundancy as possible during the rebuild, which is especially nice for two-way mirrors. Always plan a free drive bay if you can! The disk must be at least as large as the current vdev, "hopefully obviously."

2) You can replace a drive with a larger drive. The vdev will remain at the existing size, but you can plan for future expansion by setting "zpool set autoexpand=on mypoolname". So if you start out with two 1TB drives, and you replace one with a 4TB drive, that vdev will still only be 1TB. However, when you replace the OTHER one with a 4TB drive, the vdev size jumps to 4TB and the pool size increases by 3TB too. ONLY the devices within that single mirror vdev matter for purposes of resizing.

The pool cannot be downsized, vdevs cannot be downsized, and vdevs cannot be removed from the pool. However, unlike a RAIDZ vdev of six disks, where you would need to buy six new disks to bump up the size through replacement, ZFS will be able to cope with upgrading a mirror with only two drives replaced. It might not be balanced, space- and IOPS-wise, but it works.

The downside to mirrors is that mirrors are not space-efficient. However, they're incredibly flexible, often offer the best performance available in ZFS, etc.

 

[ramar]

Always plan a free drive bay if you can!

this is what i wanted to know, i need a free drive bay, or one that has an unused drive in it, ready to mirror which ever drive needs help. i would create a 3-way mirror and then remove and replace one of the drives as a reserve for the next anticipated failure. is that how you would do it? alternatively could i have a 3 mirror vdev and when one fails from a two-way mirror, reassign one drive from the 3-way to the failing 2-way leaving 2 2-way vdevs once the failing drive is turned off? or would it be better to leave the reserve drive empty until it is needed?

 

[Ericloewe]

alternatively could i have a 3 mirror vdev and when one fails from a two-way mirror, reassign one drive from the 3-way to the failing 2-way leaving 2 2-way vdevs once the failing drive is turned off?

That seems awfully complicated and almost the same as having a hot spare.

 

[Chris Moore]

this is what i wanted to know, i need a free drive bay, or one that has an unused drive in it, ready to mirror which ever drive needs help. i would create a 3-way mirror and then remove and replace one of the drives as a reserve for the next anticipated failure. is that how you would do it? alternatively could i have a 3 mirror vdev and when one fails from a two-way mirror, reassign one drive from the 3-way to the failing 2-way leaving 2 2-way vdevs once the failing drive is turned off? or would it be better to leave the reserve drive empty until it is needed?

If you were going to assign a drive as a hot spare that might be a reasonable idea but having a drive actually participating in a mirror means it is getting wear and tear while it's just sitting there doing nothing useful. Instead, you should just set it on shelf and wait until it's needed and then put it in the server. I don't even like the idea of having a hot spare because the drive is sitting there getting worn out while it's waiting for another drive to fail. There are some high-availability servers at work where I do that but there's no point in doing that at home.
I keep two spare drives of each size that I need, just in case, but I keep them in a cabinet where I can grab them when I need them.

Sent from my SAMSUNG-SGH-I537 using Tapatalk

 

[jgreco]

That seems awfully complicated and almost the same as having a hot spare.

Having a hot spare with mirrors has the advantage that the hot spare can be reassigned automatically.

For a small number of mirrors, such as two mirror vdevs, you have a 50/50 chance that the failed device will be a member of the three-way mirror.

If you keep the spare as part of one of the active vdevs, it is actually participating and cannot be in a spun-down state. This has the upside of meaning the drive is well-tested, but the downside of meaning wear and tear. The ideal hot spare is plugged in and ready to go, but spun down, so that the system can automatically begin replacing the failed drive when it happens that you're out getting smashed on New Year's Eve.

 

[jgreco]

If you were going to assign a drive as a hot spare that might be a reasonable idea but having a drive actually participating in a mirror means it is getting wear and tear while it's just sitting there doing nothing useful.

Having a working copy of data is not "nothing useful." Whether it happens to be the right data at the time of a failure is another matter.

Instead, you should just set it on shelf and wait until it's needed and then put it in the server. I don't even like the idea of having a hot spare because the drive is sitting there getting worn out while it's waiting for another drive to fail.

Well that seems silly. Have it spin down if you're that uptight about wear and tear, but it isn't really useful to have a drive on the shelf if there's a possibility of having it on standby.

 

[jgreco]

or would it be better to leave the reserve drive empty until it is needed?

Figure out a policy that suits. The policy for important data here is that "a drive failure shall not compromise redundancy", which precludes the use of RAIDZ1 or two-way mirrors. However, two-way mirrors or RAIDZ1 might be just fine for backups or other transient data. There's a pragmatic aspect to this as well. If you don't have the drive bay to spare, well, then you have to deal with it as it happens. That kinda sucks, but it is what it is.

 

[ramar]

Mirrors have a 100% predictable amount of overhead, compared to the variable parity consumption of RAIDZ.

why is the predictability a big deal?

 

[ramar]

That seems awfully complicated and almost the same as having a hot spare.

you have a point. i didn't know that a hot spare was a thing.

 

[ramar]

I don't even like the idea of having a hot spare because the drive is sitting there getting worn out while it's waiting for another drive to fail.

i see. i understand this point, which might work for me because i am only using it for myself, i am not hosting, however, i might leave it on and if i do and a drive fails, would my data be damaged? alternatively, if i shut it down accept when i am working on music, i would shut down my hot spare too, so having it on when my NAS was on might not be big deal and having it set on auto might really be a good idea. i am not doing a lot of production at the moment, but that is partly because i am trying to figure all this out and i am remodeling my house. also, i might use SSDs, so maybe the wear would be less than a spinning disk. since i don't need a lot of storage right at the moment, i could get small SSDs and increase the storage in a few years

 

[ramar]

Whether it happens to be the right data at the time of a failure is another matter.

that was my first thought. there is a reason we have RAID and mirrors.

 

[Chris Moore]

i might use SSDs, so maybe the wear would be less than a spinning disk. since i don't need a lot of storage right at the moment, i could get small SSDs

If you are building a storage pool with SSDs, the entire discussion changes. Every answer changes because of the speed and endurance SSDs have that spinning disks do not.

That is a decision that should be made first. Decide FIRST if it will be spinning disk or solid state, then decide on the layout of the pool and once the layout is decided, you can't change from SSD to HDD because the way things work are just too different.

 

[ramar]

If you don't have the drive bay to spare, well, then you have to deal with it as it happens. That kinda sucks, but it is what it is.

i am considering which case to get (not right away), so this consideration is a factor. would it make sense to have a "project Drive" and a "Library Drive", each with 2 mirrors? that would be 6 drives. should i mirror the boot drive and the FreeNAS drive with 2 mirrors each?
6 more so that makes 12. a hot spare makes 13. what would be a good case for 13 drives and 3 80Plus gold PSUs. i guess the economical build is now high end. i will start small and build it up to this when i can. maybe i will start with the 6 drive fractal design Node 304. can i install 2 PSUs in that case? 2 FreeNAS mirror drives, 1 Boot Mirror Drive, 2 Library Drives, 2 Project Drive. 2+1+2+2=7 still a bit steep.
2 FreeNAS mirror drives ($120), 1 Library Drive ($160), 1 Project Mirror Drive ($160).
2 (120GB SSD @ $60)+ 1 @ $160 WD EasyStore 8TB + 1 @ $160 WD EasyStore 8TB = $440. lower:
2 (120GB SSD @ $60) + 1 @ $85 WD Red 2TB + 1 @ $85 WD Red 2TB = $290
case is $100, Supermicro X9SCM-F (i don't remember if it handles more than 10 drives) Motherboard is $120, 16GB ECC RAM $80, Pentium G2020 CPU $15, RAID controller $60, is 450W enough? 80Plus gold PSU $80, fans $100= $830, i think. i am not going to start buying for at least a few months. if i can't use the motherboard, then maybe i can't use the CPU. i'm too tired to look this up. thanks guys

 

[ramar]

you can't change from SSD to HDD because the way things work are just too different.

thanks Chris. what i am thinking is generally i research ad nauseum for 6 months so it will depend on the price when i get closer to the purchase date.
do you know how they use the WD easystore disks? do they shuck them and install the bare drives? they were $160 at bestbuy until a few minutes ago, but now they are showing $300 https://www.bestbuy.com/site/wd-eas...2zM0HTegnBDXx2p4yiVTA84F92xH2u4caAgS1EALw_wcB i thinnk it's Google or Best Buy thinkin i might see the price going up and snap them up.
i might look for second hand ones, mirror, hot spare auto reassign them.

 

[jgreco]

why is the predictability a big deal?

Because RAIDZ calculates parity for a variable sized block.

So if you look at the first two lines, this is what people "expect" RAIDZ does, because that looks similar to RAID4 (RAID5 without staggered parity). But that's a best case scenario. In practice, ZFS will always allocate an even number of sectors by padding (see the "X" entries on lines 7 and 8) and if you have a particularly small block of data - one sector - that you need to protect, ZFS still needs to use a parity sector for that (see the P/D at the end of line 3). So in a bad scenario, someone who has created a pool with a bad recordsize/blocksize/volblocksize strategy could potentially find that they're chewing up 100% overhead for parity rather than the "1 of N drives" overhead they were expecting. Ironically this gets WORSE with RAIDZ2 and RAIDZ3, because for that same single sector protection, you end up needing 3 parity sectors to protect a single data sector for RAIDZ3. Which means that, yes, you can actually cause some bizarre situations.

Now, admittedly, that's all through the magic of bad design, but people who don't understand this property don't understand it is bad design. RAIDZ is AWESOME at protecting large runs of sequential data, because almost all of it ends up being those optimal situations shown in the picture.

It isn't unusual for someone to show up here wanting to make their home lab all-in-one box with meager resources do as much as they can, though, so they'll pick RAIDZ1 or RAIDZ2 to "make the most of the space", and they will get smacked down by this issue, and by the fact that the performance of a RAIDZ vdev is limited to approximately the IOPS of a single member device. With mirrors, the variable parity issue doesn't exist (you're consuming a massive amount of space for redundancy but it's fixed), write speeds are limited to the IOPS of the member devices, but reads can be spread out between the member devices, so read speeds are anywhere between what a single member device is capable of, and the total sum of all member devices. This is very awesome especially if you have heavy random read workloads.

 

[jgreco]

thanks Chris. what i am thinking is generally i research ad nauseum for 6 months so it will depend on the price when i get closer to the purchase date.
do you know how they use the WD easystore disks? do they shuck them and install the bare drives? they were $160 at bestbuy until a few minutes ago, but now they are showing $300 https://www.bestbuy.com/site/wd-eas...2zM0HTegnBDXx2p4yiVTA84F92xH2u4caAgS1EALw_wcB i thinnk it's Google or Best Buy thinkin i might see the price going up and snap them up.
i might look for second hand ones, mirror, hot spare auto reassign them.

Those go on sale every few months for $160. Google it for the history. I wouldn't worry about it too much, just wait for the next sale.

 

[ramar]

Because RAIDZ calculates parity for a variable sized block.

So if you look at the first two lines, this is what people "expect" RAIDZ does, because that looks similar to RAID4 (RAID5 without staggered parity).

looking at the chart, it shows LBA\Disk. What is LBA? by the way, thanks for all this. i am pretty sold on mirroring drives, is that RAIDZ? and ECC memory and i will get the general idea of RIAD 0 through 12. is that what being described? we got 5 disks; A through E, correct? we have colors and the letters P, D & X and sub script numbers 0, 1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13. i am a little lost but i want to understand.
thanks for the tip on the 8TB easystore disks sales. i guess they brought them down to $120 for black friday. i'm hoping they bring down prices on SSDs too because of M.2 coming down, but i need to wait either way. i'll have 4 primary drives; FreeNAS on a 120GB SSD, a Boot and primary program, a library and a project, and a hot spare. 5. 2-way mirror is 10. well i'll have one in the computer, so 9. 3-way, 14 and i'll have 3 hot spares. and i need a backup too, so 15. better have 2, 16 and i'll have some external drives. sounds economical. it's not anyones fault, it's the nature of reality.
thanks!

 

[Ericloewe]

What is LBA?

Logical Block Address, the address of a block on disk: https://en.wikipedia.org/wiki/Logical_block_addressing

letters P, D & X

In the context of that document, Parity, Data and Padding