raidz using one or more (striped) vdevs

[raid40000]

I have a new system built (thanks for verifying the hardware specs btw) with 12 drives, mixing SAS and SATA controllers, the board is a Supermicro that allows up to 16 drives in total.

- RAM on the system is maxed out. 32 to 64GB RAM. No SSD caches of any kind for ZLOG/ZIL,etc.

Now the system is installed and I am wondering about the different alternatives that I have.

My needs:

- Loads of medium to small files. 20 to 100 people accessing it randomly and performance on writes is valuable, however, regardless of the setup I deploy in the in it will outperform any current system present.
- People working on those files is using media tools, and office suites. Like Adobe Creative Cloud suite, some video streaming and random pdf documents in and out.
- Backup of VM disks. Pretty big files in general.
- Few non-critical virtual machines exposed via NFS to VMWARE or other virtualization environments. Those are not running databases and so on but should not impact dramatically the normal use of the FreeNAS system.
- Around 10 to 18TB data is plenty and in the long run another server can be acquired.

Options considered:

- 2 VDEV, 6 disks each on Z3. Reasoning: I am prioritizing reliability so I do not mind to waste 9 extra TB and I can do stripped volumes to increase IOPS in case I need them in the future.

- 1 VDEV, 12 disks on Z3. Reasoning: simple to setup, still 3 disks can fail and I have more usable space. Less complexity if something goes wrong in the future ?

- 2 VDEV, 6 disk each in Z2: Reasoning: more space, more speed. But less reliable.

Concerns:

A. What is the real loss in terms of IOPS by just using 1 VDEV in z3 vs 2VDEV in z3? Is it worth it?
B. What is the impact on re-silvering by the fact of having 2 VDEV vs 1 VDEv?
C. Will Z3 have a huge negative impact on writing speed when many people is using the NAS compared to other alternatives ? I have read few benchmarks here and there, and does not seems to be significant ?

At the moment I am bound by gigabit ethernet, but the board is able to 10Gigabit and LAGG would be setup eventually.

All comments and constructive/destructive advice welcome.

 

[depasseg]

A RAID Z vdev will have the IOPS of the slowest disk and the bandwidth of the sum of the data drives. And 7200rpm drives will have ~150iops. Imagine 20-100 randomly accessing files on a single drive. Even double (with 2 vdevs) will likely be a performance bottleneck. You can try it and see how it goes, but you will likely need to move to mirrors and/or a lot more RAM and a large L2ARC.

A - you will get ~half the IOPS with a single vdev vs 2 vdevs.
B - not sure
C - I don't think it's significant.

 

[Nick2253]

The first thing that jumps out at me is the VMs. Using FreeNAS as a VM datastore is incredibly demanding, regardless if they are running databases or not. There's a reason why almost everyone here who does it uses cache drives. (https://forums.freenas.org/index.ph...xi-nfs-so-slow-and-why-is-iscsi-faster.12506/)

My second thought is that your two primary use cases are almost opposites of each other: periodic backup of huge files, and frequent and numerous read/writes of small files. Of your three options, I think the last one is the best balance for your use case.

RAIDZ3 is great for extra redundancy, but your I/O won't be great. The extra calculations for that third parity drive really take a hit on random writes. However, it won't impact sequential writes much at all. Multiple vdevs go a long way to improving I/O, and RAIDZ2 is a great balance for redundancy.

However, if I may suggest a forth option:

Two pools: 6-8x drives in stripped mirrors, and 4-6 drives in RAIDZ2. Store your in-use data on the stripped mirrors, and backup to the RAIDZ2 pool. You can also replicate the first pool to the second for backup purposes.

If you want to get fancy, and you don't have a lot of storage needs, you might be able to do 4x SSDs in the stripped mirrors pool. That would give you insane I/O, plus the security from the RAIDZ2 pool for backup.

 

[raid40000]

@Nick2253 You are right on the fact that it seems a bit "multi purpose" FreeNAS box, and yes: virtualization when the other end is forcing "sync" is demanding, and to be honest after reading the comments I will not even consider it. VMs are out of the equation other than for quick tests, also I have another box with local storage that is meant for that.

I would go for z2 on two devs but the performance gain on that does not compensate the risk versus a Z3 (one or two vdevs) and I mention this because the reality is that the server is not sitting in a super cold and well maintained area, nor the 3TB WD RED are the bests disks out there. So I would rather go with z3 regardless of anything.

So 2dev in z3 is a bit "silly" as some people may be thinking (just doing that to increase IOPS a bit without involving SSD caches), but again all in all the ~9TB I am wasting in redundancy are not going to be needed anytime soon, if the space becomes an issue the best approach can be to get another server or expand it with another VDEV.

Then on your fourth suggestion: that is a good point ! and since the board has 16 ports in total 8 SAS + 8 SATA if in the future I need to have more IOPS or use it to store few non-critical VMs that need to perform I can add some SSD disks and create a new pool using that 4 drive vdev.

Not quite sure creating two pools z2 and z3 of spinning disks is going to make a huge impact at the moment but is another approach really.

 

[depasseg]

If you are that concerned about the likelihood of losing 4 out of 6 disks, then I assume you have a second server for backups (replication).

 

[raid40000]

@depasseg, sadly that is also a reason why I do z3. No backup server, except some remote location aka Internet based backups, and upload speed is not great.

If this delivers and becomes an issue I will push for a serious deployment, then things should be better. But there are a few things I do no control and that will have to change in the near future.

So you can have an idea, I have a "cache" system and the current storage is network bounded (and takes like 2h to move a 8GB file, seconds for a less than 50MB file sometimes. If lucky.) Then the other system that was setup was delivering like 5-10MB/s via windows based share, with a single computer.

That bad, this is why I am setting up a new "quick" deployment to start with FreeNAS vs going for a random vendor NAS worth thousands that will end up not performing which was the plan others had.

 

[depasseg]

So if you need a blend of performance and capacity, consider 5x 3-way mirrors and a warm spare. I'm really concerned with the load that 20-100 users will put on 1 or vdevs.

 

[Nick2253]

I would say that RAIDZ3 makes sense in certain cases, especially when you have a super wide vdev, or you are building archive storage. However, given the size of your vdevs and your use case, I would say that RAIDZ3 is overkill. With the number of users you have, you need multiple vdevs. Two RAIDZ2 vdevs might cut it, but I would have to test to be sure. If performance is really critical, then stripped mirrors (or stripped 3-way mirrors as @depasseg suggested) would be the best route.

It's really hard to build maximum reliability and maximum performance into the same box, especially when you're talking smb/enterprise workloads. In your case, I would also recommend a second box: a high-performance main array, and a highly reliable backup array. The backup array doesn't have to be fancy (there are a number of off-the-shelf solutions under $500 that would work for this use). Paired with 8-10TB drives, you'd have a whole bunch of space.

 

[rs225]

sadly that is also a reason why I do z3. No backup server, except some remote location aka Internet based backups, and upload speed is not great.

When the brown stuff hits the air mover, you are going to wish you used raidz1, and used those extra drives for some non-redundant backup.

So if you need a blend of performance and capacity, consider 5x 3-way mirrors and a warm spare. I'm really concerned with the load that 20-100 users will put on 1 or vdevs.

Better yet, 5x 2-way mirrors, and a full backup.

 

[raid40000]

@Nick2253 @rs225 I will be backing up into a remote location (S3 or the current network shared resource - over night probably), but at some stage there will be too much data for that, and then I will suggest also some additional backup locally. Bear in mind, I also have to keep some data compliance regulations in the organization.

In the end, since both the backup and server are in the same location they could eventually burn or get stolen. Which also reminds me that I must consider adding crypto, despite its complexity. For certain files, while the data is very important, the impact of losing delivered work would be not as dramatic.

@rs225 you are 200% right. But, the reality is that I do not have another storage where to replicate or backup the NAS. Not even spare drives, and when I issue the warning I will recommend to get another box just for backup as soon as possible, even a FreeNAS mini like box with z1.

@Nick2253 I think it will be the plan, once everyone uses this and it has some big amount of files I will suggest to backup it up properly, specially if I decide to use S3 and the bill starts to be big. As long as there is awareness on the risks, not having a backup on early use can be traded.

 

[rs225]

I think it will be the plan, once everyone uses this and it has some big amount of files I will suggest to backup it up properly, specially if I decide to use S3 and the bill starts to be big. As long as there is awareness on the risks, not having a backup on early use can be traded.

There are a lot of real sad, sad, songs that start out like this.

 

[Nick2253]

There are a lot of real sad, sad, songs that start out like this.

Unfortunately, it's too true.

Another concern is that ZFS is much harder to recover data from. The costs of recovery have come down, but it's still an order of magnitude more expensive than recovery from other files systems. An unfortunate side effect of the other benefits of ZFS.

Anyway, don't think about backup later, don't plan for backup in the future, just do it today.

 

[tvsjr]

@Nick2253 I think it will be the plan, once everyone uses this and it has some big amount of files I will suggest to backup it up properly, specially if I decide to use S3 and the bill starts to be big. As long as there is awareness on the risks, not having a backup on early use can be traded.

Why S3? Look at Glacier - specifically intended for backup storage.

 

[raid40000]

@tvsjr I prefer Glacier indeed, I use it at the moment. I did not see a plugin for that though... and it is somewhat slower to upload. Do you have more detailed instructions on the setup ? I could access the share and upload using something but I would rather handle that via FreeNAS directly on the very same server.

@rs225 @Nick2253 you are totally right, one way or another there will be backup in place. Worst case, using whatever upload bandwidth we have to "the cloud". No point in not having backups regardless of the ZFS setup. I am not going to risk.

 

[Robert Trevellyan]

I agree with the suggestion of striped mirrors, perhaps 2-way with one or more hot spares.

I prefer Glacier indeed, I use it at the moment. I did not see a plugin for that though

You don't need a different plugin for Glacier if you set automatic migration on your bucket(s). You might also consider B2 via a jail: https://www.backblaze.com/b2/docs/integrations.html

 

[Nick2253]

Why S3? Look at Glacier - specifically intended for backup storage.

I would disagree with that statement.

Glacier is intended for archiving, not backup, which is an important distinction. Restoring even a couple hundred GBs from Glacier is incredibly sloooooooooow. Especially in a business environment, any delay at retrieving your backup can be incredibly costly.

On the other hand, if you are achieving data (for example, old business records for tax/regulatory compliance), then Glacier is ideal, because you'll have plenty of notice if you need to recover that data, so there's really no penalty if it takes you hours or days to get your old data.

I could accept the argument that Glacier would be acceptable as a tertiary or quaternary backup level, based on balancing the likelihood of need to retrieve your data at that level vs cost to store the data, but your primary and secondary backups should fast backups: generally the primary being an onsite very high speed backup (like disk), and your secondary being a fairly high speed offsite backup (like tape or fast cloud).

 

[Stux]

Stick some spare disks in the same chassis and backup to that.

At least you are then free to reshape your primary pool.

 

[raid40000]

@Stux, that would be ok for having a fast pool with SSD disks but not for backup. I have other servers with some free bays I could use. However, the issue is that if I rely on physical security there... backups are useless either way.

 

[maglin]

OP you are not limited in the number of disks on SAS. I have 15 disks on a single SAS2 port. So an enclosure with a SAS expander will get you more disks then you can handle.

Also maybe TrueNAS is something you might want to look into. It seems exactly what you need along with support.


Sent from my iPhone using Tapatalk

 

[Stux]

@Stux, that would be ok for having a fast pool with SSD disks but not for backup. I have other servers with some free bays I could use. However, the issue is that if I rely on physical security there... backups are useless either way.

A backup in the same chassis is better than no backup at all

Two backups is better than one

And local and offsite backup is the way I go.

My point was, I guess, that it's better to have a backup than excessive redundancy levels, because I think that the chances of loss due to f$&ckup are higher than exceeding your raidz2 redundancy.

Raid is not backup. It's a method of increasing availability. How long it takes to restore from backup is a consideration, and that's why the offsite backup should be the backup's backup, not the primary backup.