New Setup ZFS and suggestions

[brianmills]

Hi, I've been exploring setting up a NAS using OpenFiler or FreeNAS. I've explored OpenFiler separately, and I'm now looking into FreeNAS 8. I'm technical but no little about FreeNAS. I intend to use it for a variety of things, I run a small business, and I already have a dozen or so VM's setup for various things in our office (dev/test environments, build servers, software tools etc).

So I expect to use:
- NFS shares (for server data shares)
- SMB shares (for windows pc's data and backups)
- AD for user management (already setup, just hooking up the NAS/Filer too it)
- possibly iSCSI for VM's to reside on
- AFP (I have 2 mac's as well) and I dont believe OpenFiler supports this.
- The main reason for trying FreeNAS as well as it all runs at home, so I'd like to be able to use some of the multimedia features once they come back into v8 in the future

I have an old 1366pin i7 Asus based machine with 7 SATA ports I intend to use for the task (one day I might upgrade/downgrade it to an Ivy/Sandy bridge based i3 system)

My intention with was to:
- Run FreeNAS of a USB drive
- 8GB Ram already in the existing machine
- 4 * 2TB Green drives in a ZFS Array (sorry I might not have the terminology quite right here).
- 1 * 1TB or 2TB ZFS mirrored drives for iSCSI area or VM machine backup
- 2 Nic's aggregated for iSCSI access
- 1 Nic for client PC access to SMB/NFS/AFP shares

Some questions I have come up with though:
- Does that hardware sound reasonable?
- I dont know a lot about ZFS, so I was looking for advice on suggested arrau setup there.
- Would I be better with one ZFS array of 6 2TB disks, and then use pools to segregate out the data and provide snapshots etc from there?
- Do you think performance will be reasonable using that setup?
- I've had a little trouble with AD setup already with users not apearing yet (although commands like wbinfo -p and -t seem to suggest it is working) Any suggested links there?
- Some concerns around weather ZFS is ready yet, it seems as though now it is with v8 of FreeNAS (with the updates from FreeBSD underneath)
- I've read that SMB/CIFS performance isn't great on FreeBSD, is that a significant issue? or is it a small difference to linux.

 

[praecorloth]

Green drives are usually terribad. Give serious consideration to not using them. They tend to be slow and have unacceptably short life spans.

I've used iSCSI for datastore location for as many as 5 VMs on a single gigabit card. Worked out just dandy as long as the gigabit card was a dedicated iSCSI line. Like from the ESXi box to the FreeNAS box. Link aggregation may help you if you're using the iSCSI target for more than one ESXi box. I think the way FreeNAS's LAGG works is it alternates connections. Even numbered connection to the first NIC, odd to the second NIC. If that is indeed the case, and you only have one ESXi box, a team of two NICs would be good in the future when your setup grows.

I would guess your performance will be spectacular, assuming you load that bad boy down with memory. The i7, even first generation, are not lacking in power by any stretch of the imagination. (Says the AMD guy. :) )

I can't speak to the availability of AD. I don't think I've seen a functional set up yet. That could be quite the thorn in your side.

I consider ZFS to be ready. I've been using it to help safeguard the files on my NAS at home. I've recommended it to a number of small businesses who have implemented it. I would recommend it again. Certainly over basic Windows file sharing.

I have run in to CIFS performance issues in FreeNAS. It seems a lot of people have. Depending on what your users are accustom to, it may not be that bad. I think the worst case I've seen around here has been 20MB/s transfer rates on gigabit (and that may have been due to overzealous tweaking, who knows?). Among my friends and I, we've got everything nailed down to about 60MB/s on gigabit at minimum. Not trying to troll here, but in this, I think Linux has FreeBSD beat. Slow transfer rates will definitely cause you headache for a one time, massive transfer. But unlikely the users will notice unless they routinely work with files larger than a few hundred MB. Because of that, as well as ZFS, I personally will still choose, and recommend FreeNAS over a Linux based NAS solution.

 

[brianmills]

Drives

Great, I was on the edge betwen Green and Black drives from WD. Black it is.

Sounds like I will just have to try CIFS, and try tweaking it with the documentation, and see what I get.

What does anyone think about the 2 ZFS Volumes. 2 disks mirrored for VM's, and 4 disk array for other data?
Then using data sets within each of those for various purposes. I was intending to do this so the VM's could get maximum performance off 2 mirrored disks without being affected by other data copy's.

 

[praecorloth]

What does anyone think about the 2 ZFS Volumes. 2 disks mirrored for VM's, and 4 disk array for other data?
Then using data sets within each of those for various purposes. I was intending to do this so the VM's could get maximum performance off 2 mirrored disks without being affected by other data copy's.

That's basically what I have right now, except my drives are significantly smaller. Except my 4 disk array is one giant pot that I've locked down via permissions at the directory level. People have access to their own little directories that way, and a shared area for...well, sharing stuff I guess. :) But that route isn't for everyone since I believe you have to ssh in and set the permissions that way. I don't think there's a web interface for modifying permissions on specific directories.

Now that I'm thinking about it, given that the 2 disk array is going to be a mirror of a big iSCSI file extent, I'm not sure having it be ZFS is really useful. It'll take processor time to scrub when the time comes around, and it's not like you're going to make snapshots of a single, giant file. AmIright?

 

[brianmills]

The 4 disks would be a big section, but I might create several iSCSI data set sections. I'm not sure how I'll use the iSCSI mirrored disks, its not something I've used before directly.

Snapshots might be useful for backup I guess, I can take a snapshot and back that up (although that could mean inconsistent states of the virtual machines I guess).
It also might be useful to make a separate iSCSI section for each machine, so I could snapshot them at the storage layer individually.

Are you thinking just UFS would be better for the iSCSI area?

 

[Joshua Parker Ruehlig]

green drives have shorter life spans? I though cause they run slower theyd have longer life spans, less heat (metal flluctoation), vibration. Could be wrong about harddrives though, lol guess all my drives are gonna crap out on me eventually =] good thing I'll have raidz2 to save my butt

 

[praecorloth]

Let's talk a bit about ZFS snapshots. I definitely wouldn't recommend using ZFS snapshots for backups of entire iSCSI file extents. It just doesn't strike me as the problem that ZFS snapshots is trying to solve. Here's to hoping my terminology is correct. I don't have FreeNAS right in front of me at the moment.

So let's just say for the moment you have your 4 disk array in one big CIFS pool. People have their individual directories within that one share, and you have ZFS snapshots enabled. If someone deletes a file accidentally, you can go back to that snapshot, and clone the snapshot to a directory on that 4 disk array. In that directory, ZFS will dump all of the files that were modified within the time frame for that snapshot. Likely you find the deleted file, restore it, and remove the directory with the cloned ZFS snapshot.

I wanted to give that run down because I have used ZFS snapshots poorly in the past. I had used the rollback feature when I had screwed up a file. This rolled back everything I had done on my 4 disk array in the past 15 minutes. Which means I had to copy a couple of movies over to the server again. No biggie.

However, if you were to use ZFS snapshots on an array that only contained iSCSI file extents, you might be in for a larger world of hurt. A file extent will show be presented to ZFS as a very large, single file. My guess is you're going to hit one of two extremes. Either ZFS will not detect any changes in the file extent, and thus never snapshot. Or it will detect that the file extent has changed, snapshot the entire file extent because ZFS will have no knowledge of the file system within the extent, and your snapshots will quickly overrun your storage capacity.

I hope that was coherent. I'm at work and so I can't proof read a whole bunch.


In the end, I guess my recommendation would be use the 4 disk RAID-Z2 (I'm told for odd number of disks, use RAID-Z, even number of disks use Z2), and the space on that would be consumed via CIFS shares. Your 2 disk array, use UFS since (I'm guessing) it's likely to have most of the space consumed right away by a very large iSCSI extent for your virtual machine storage.



Re: Short life span on Green drives. I believe the problem is with cheaper components or something. I've just been hearing all over the Internet about the Green series having an unusually high, unusually early failure rate. And given that they spin at 5900 RPMs rather than 7200, you're looking at slower access times. Definitely something you wouldn't want in an environment where the server was sharing multiple files to multiple clients. Definitely wouldn't be wanted for iSCSI storage.

 

[Joshua Parker Ruehlig]

Thanks for the insight praecorloth.

It's something about the way zfs stripped the drives where you want an even amount for data drives, I've even heard you want a power of two for data drives, which severely limits your possible disk setups. I'm sticking with the second rule cause it make sense to me but I can't prove it, and don't 100% know if it's true.

I can see how green drives would probably skimp on parts, I just hope Samsung didn't decide to do that. Mine, spin at 5400 which is even slower, lol. Though I still think my limiting factor for read/write speed is network throughput not the drives. Can't say the same for a high IOPS environment where there's multiple ISCSI drives being access at the same time.

 

[paleoN]

It's something about the way zfs stripped the drives where you want an even amount for data drives, I've even heard you want a power of two for data drives, which severely limits your possible disk setups. I'm sticking with the second rule cause it make sense to me but I can't prove it, and don't 100% know if it's true.

The ZFS Best Practices Guide is no doubt the source of that. In particular:

• (N+P) with P = 1 (raidz), 2 (raidz2), or 3 (raidz3) and N equals 2, 4, or 6
  • 3 disks - (2+1) raidz1
    
  • 4 disks - (2+2) raidz2
    
  • 5 disks - (4+1) raidz1
    
  • 6 disks - (4+2) raidz2
    
  • 6 disks - (2+1) + (2+1) raidz1 - 2 vdevs
    
  • etc.

You can ignore raidz3 part, until FreeNAS 8.3 that is.

 

[StephenFry]

green drives have shorter life spans?

Don't worry about it, i purchase thousands of drives every year and while there are slight differences between production facilities, models, brands, etc, I've yet to see a significant difference in lifespan (or even speed) for green vs non-green drives.

 

[cyberjock]

I used to use Green drives on a RAID array. They work fine as long as you use the wdidle.exe to set the time to "parK' the heads. By default they park after 6 seconds(I think) and they can hit this limit quickly because of how data is accessed. I set mine to the maximum of 5 minutes and had no problems. Other people that didn't change the setting got SMART failure imminent warnings after just a few months of using the drive because there is a counter for the # of parks until "estimated failure".

I will say I wouldn't buy green drives to "save the planet". You aren't talking about saving alot of power. You likely could save more electricity than a whole array of green drives just by buying one CFL. The whole "green is better" for hard drives is kind of a joke.

 

[brianmills]

Some good points you make praecorloth. Thanks for the comments on iSCSI, it makes complete sense that ZFS snapshots wouldn't necessarily do a good job there as it wont understand the file system.

Wont RaidZ2 mean I loose more space? Does anyone know why the recomended number of drives affects the choice between RaidZ and RaidZ2?

 

[survive]

Hi brianmills,

With raidz you lose the capacity of one drive in the pool to the parity data (parity data is distributed across all the drives, but it amounts to 1 drive's worth). With raidz2 you lose 2 drives worth of capacity to hold the parity data as there are 2 different parity methods used to protect against the dreaded "double disk failure".

It looks like you are planning to have 6 drives total in your pool, so it might make sense to go with 6 2TB drives and make 1 raidz2 pool and use datasets to segregate your data. You can create "zvol"'s for your iscsi data and share them out to the boxes that need iscsi.

The i7 should have plenty of grunt to run most anything you can throw at it, but I would encourage you to add more RAM....nothing makes ZFS happier than having a ton of RAM to use, and cheap as DDR3 is these days there' really no reason not to.

-Will

 

[brianmills]

Nice. More RAM sounds like a plan.

In terms of 6 drives, I had intended to segregate my data into 2 sets of drives
2 drives mirrored for VM sharred storage
4 drives in ZFS RaidZ for NAS Data

My motivation to do this is to ensure the 2 drive VM sharred storage space performs as well as possible.
Do you think I'd be better to just have one RaidZ2 pool of disks and setup the segregated VM vs NAS data space using a datasets?

 

[StephenFry]

Do you think I'd be better to just have one RaidZ2 pool of disks and setup the segregated VM vs NAS data space using a datasets?

If you are planning to build this system, my recommendation would be to just try both scenarios out. You will need to do some testing and training anyway - if you don't, you will regret it, I promise. Make both the 6 drive Z2 and the 2/4 drive set combo and see how both perform.

I highly prefer the Z2 setup for 6 large drives, and my performance is absolutely great, I can fill my Gbit connection and could easily team two NICs and fill that bandwidth.

As always, it's a personal choice if you want to risk it with a raid-Z - if data availability is not super important and you have a good backup protocol, Z2 might not be necessary, but as I've said in other posts, raid-z with the current crop of 2TB and larger drives is asking for trouble.

 

[paleoN]

In terms of 6 drives, I had intended to segregate my data into 2 sets of drives
2 drives mirrored for VM sharred storage
4 drives in ZFS RaidZ for NAS Data

RaidZ2 and up are used for 2 main reasons, actually more but...

1. The larger the number of disks in a vdev the longer it takes to rebuild.
2. The larger the capacity of the drives the longer it takes to rebuild.

Longer rebuild times lead to greater chance for a 2nd disk to fail and remember a resilver is hard drive intensive process.

6 drive vdev is the starting recommendation for raidz2, which I will echo.

If you do decide to create 2 pools, one being the mirror, I don't recommend raidz2 for the 4 disk pool. I would setup the 4 disks as striped mirrors. If you are losing half the pool to redundancy you might as well as get the performance.

That being said definitely test it out first. A 6 drive raidz2 pool may surprise you with how well it performs.

I will also suggest having a cold spare waiting, sitting on a shelf.