Slideshow explaining VDev, zpool, ZIL and L2ARC for noobs!

pclausen

Patron
Joined
Apr 19, 2015
Messages
267
I'm getting ready to start a new build, and I'm looking for it to be capable of saturating a 10 Gb/s network link. My production server is windows based with an Areca 1882XI16 controller configured with a RAID60 array consisting of a pair of 12x6TB RAID6 arrays (WD Reds). This config is fully capable of saturating a 10 Gb/s link.

I was running a 50 disk ZFS array (see sig) but was never able to get much past 3-4 Gb/s on transfers. So my plan for the new build is to have a total of 12 vdevs, each with 5 2TB drives in raidz1. This system will only be powered up once a month or so to sync up against the windows server. There may also be times when I want to blow away my Windows system and reconfigure, and then restore the data from the ZFS array. I might also want to reconfigure the FreeNAS server and restore the data from the Windows box.

I already have the 60 2TB drives, some were used before in my previous build, and others are recently acquired. All have passed badblock testing and none have SMART errors that matter. Here's how I plan to configure the vdevs. I'm trying to group them together in the same vdev based on their performance characteristics:

12vdevs.PNG


I'm thinking that with 2TB drives that have been well tested, running 5 disks in raidz1 will be relatively safe. Also, this will only serve as a backup to my primary server, and as stated above, will only be brought online once a month or so to sync up against the prod server.

The mobo is an older SuperMicro X8DT6-F (embedded 8 port LSI 2008 controller that I will flash to IT P20) with a pair of Xeon L5630 CPUs and 12 x 4 GB DDR3-1333MHz ECC RAM (48GB total). I won't be running any plugins or anything else. Just a vanilla build to act as backup for my primary server.
 

pclausen

Patron
Joined
Apr 19, 2015
Messages
267
I did some additional research and it would seem that raidz1 is generally not recommended when using drives larger than 1TB, regardless of how many disks are in the individual vdevs.

I'll try anyway with 12 vdevs just to see what kind of performance I can get and then tear it down and go with 10 disks raidz2 vdevs like I had before. Maybe since I'll be starting with an empty zpool, my performance will be better than what I had before which was build over time by adding one vdev at a time.
 
Last edited:

icsy7867

Contributor
Joined
Dec 31, 2015
Messages
167
Sorry... newbie here... I have a question I was hoping I could get some guidance on and understanding.

I read through the entire slide deck and I have a question on Slide 29 (4 500GB drives in a vdev and a 2TB drive in a vdev).

Does the 2TB drive had part of the same data from the vdev of 500GB drives?

In my current setup I have 4x 1.5TB drives in Raidz1
I then added a second 2TB data drive that is completely separated from the 4 x 1.5TB Drives with independent data on them. I recently even formatted the drive and copied the data back over to it because I got a strange metadata error after upgrading to ECC memory.

Is this slide saying that if that separate 2TB drive, with a separate dataset on it fails, my 4x1.5TB drives in raidz1 would also fail? I wouldn't think this case, but after reading the slides it makes me feel even more noob'ish.

Sorry for the silly questions! Thank you for your insight!
 

Mirfster

Doesn't know what he's talking about
Joined
Oct 2, 2015
Messages
3,215
Is this slide saying that if that separate 2TB drive, with a separate dataset on it fails, my 4x1.5TB drives in raidz1 would also fail?
No, it is saying that if you have a pool/dataset that consists of two vdevs (example: one vdev having 4x1.5 in RaidZ1 and the other vdev a single 2TB) and if either of the vdevs goes belly-up, then you lose the whole pool/dataset.

Seems to me like you have two different pools/datasets, so loss of one will not scrap the other. Please let me know if I misinterpreted how your pools/datasets are currently configured.
 

MrToddsFriends

Documentation Browser
Joined
Jan 12, 2015
Messages
1,338

icsy7867

Contributor
Joined
Dec 31, 2015
Messages
167
Ahhh that ASCII art is great! Thank you!

That is what I was thinking. But after seeing the 4x 500GB drives and then tossing in the 1x 2TB drive into the diagram made me think that I was not understanding some things correctly. Just needed a sanity check. Thank you guys for your quick responses!
 

Nick@Libero

Cadet
Joined
Feb 5, 2016
Messages
8
Thanks for your incredible and detailed work @Cyberjok
(The powerpoint file is really really clear)
 

blackbinary

Dabbler
Joined
Jan 1, 2014
Messages
16
@cyberjock Yeah! Nice Guid!

i had some of the problems in the past you talked about in your guid. ( Realtek NIC problems, Scrubing multible days with Bad Memory! etc.. but my Data Survival luckily o_O... ) but still... after more than 5 years with FreeNAS i still learnd some nice things in your Guid! - Keep up your great work. :):)

And get yourself a Bitcoin address for a tip ! ;)
 

Stilez

Guru
Joined
Apr 8, 2016
Messages
529
Just setting up my first FreeNAS, thanks for your guide!

One point that would help to clarify:

Slide/page 45 states that "it is imperative that you always ensure that sufficient redundancy exists. This is one reason why RAIDZ1 is dangerous", and this is repeated a lot. But unless I missed it, it isn't made clear (and underlined! :D) that RaidZ is not mirroring, and it's way too easy for a newcomer to miss that RaidZ1 is not ZFS's version of Raid1. There's almost no discussion of mirroring vs RaidZ and pros/cons. So can you cover a bit more, the differences between using RaidZ and simple mirroring in FreeNAS, as quite a few guides emphasise that, while mirroring isn't sexy and doesn't give the largest usable space, it is fast on reading and does provides good redundancy/rebuilt/splitting capabilitiesd, especially with a mirror of 3+ drives.

And any comments on this would be very welcome here!
 

Mirfster

Doesn't know what he's talking about
Joined
Oct 2, 2015
Messages
3,215
But unless I missed it, it isn't made clear (and underlined! :D) that RaidZ is not mirroring, and it's way too easy for a newcomer to miss that RaidZ1 is not ZFS's version of Raid1.
One mention of it is in slide #57:
RAID5(and its ZFS equivalent RAIDZ1) is “dead” as of 2009. http://www.zdnet.com/blog/storage/why-raid-5-stops-working-in-2009/162


There's almost no discussion of mirroring vs RaidZ and pros/cons.
This can be seen in the FreeNas Guide:
RAIDZ1 maximizes disk space and generally performs well when data is written and read in large chunks (128K
or more).
• RAIDZ2 offers better data availability and significantly better mean time to data loss (MTTDL) than RAIDZ1.
• A mirror consumes more disk space but generally performs better with small random reads. For better performance,
a mirror is strongly favored over any RAIDZ, particularly for large, uncacheable, random read loads.
• Using more than 12 disks per vdev is not recommended. The recommended number of disks per vdev is between
3 and 9. If you have more disks, use multiple vdevs.
• Some older ZFS documentation recommends that a certain number of disks is needed for each type of RAIDZ in
order to achieve optimal performance. On systems using LZ4 compression, which is the default for FreeNAS®
9.2.1 and higher, this is no longer true. See ZFS RAIDZ stripe width, or: How I Learned to Stop Worrying and
Love RAIDZ for details.
The following resources can also help you determine the RAID configuration best suited to your storage needs:
Getting the Most out of ZFS Pools
A Closer Look at ZFS, Vdevs and Performance
Warning: NO RAID SOLUTION PROVIDES A REPLACEMENT

I find myself constantly re-reading posts, documents, stickies and threads. So much to learn that it all can't be taken in at once... ;)
 

TheDubiousDubber

Contributor
Joined
Sep 11, 2014
Messages
193
Just setting up my first FreeNAS, thanks for your guide!

One point that would help to clarify:

Slide/page 45 states that "it is imperative that you always ensure that sufficient redundancy exists. This is one reason why RAIDZ1 is dangerous", and this is repeated a lot. But unless I missed it, it isn't made clear (and underlined! :D) that RaidZ is not mirroring, and it's way too easy for a newcomer to miss that RaidZ1 is not ZFS's version of Raid1. There's almost no discussion of mirroring vs RaidZ and pros/cons. So can you cover a bit more, the differences between using RaidZ and simple mirroring in FreeNAS, as quite a few guides emphasise that, while mirroring isn't sexy and doesn't give the largest usable space, it is fast on reading and does provides good redundancy/rebuilt/splitting capabilitiesd, especially with a mirror of 3+ drives.

And any comments on this would be very welcome here!

The easier way to think of ZFS is in levels. RAIDZ1 offers a single level of protection. You have 1 disk for redundancy, if a single disk fails, then redundancy is gone and any further errors would mean data loss. RAIDZ2 means the same thing, only with 2 disks, and two disks have to fail before any subsequent errors lead to data loss. A single disk is not sufficient. The resilvering process can be quite taxing after replacing a failed disk. If you are using a RAIDZ1, and resilvering process causes another disk to fail, you are SOL.
 

Stilez

Guru
Joined
Apr 8, 2016
Messages
529
The easier way to think of ZFS is in levels. RAIDZ1 offers a single level of protection. You have 1 disk for redundancy, if a single disk fails, then redundancy is gone and any further errors would mean data loss. RAIDZ2 means the same thing, only with 2 disks, and two disks have to fail before any subsequent errors lead to data loss. A single disk is not sufficient. The resilvering process can be quite taxing after replacing a failed disk. If you are using a RAIDZ1, and resilvering process causes another disk to fail, you are SOL.
There's an irony in a question noting "There's almost no discussion of mirroring vs RaidZ and pros/cons" and asking for better coverage, which is met by an answer that contains... no discussion of mirroring vs RaidZ ;-)

A reply to the question asked, please? :)
 

fta

Contributor
Joined
Apr 6, 2015
Messages
148
Are the PDFs up to date? I only looked at the google PDF, and most of the "ZIL" stuff is actually talking about SLOG.
 

Stilez

Guru
Joined
Apr 8, 2016
Messages
529
That's possibly a second rate answer (very sorry!) - by the same token the slideshow covers RaidZ without readers being told "ZFS Primer", and mirroring is no less a fundamental setup option within ZFS than RaidZ. If it were minor and legacy that would be different, but it isn't - quite a lot of articles on ZFS actively suggest mirroring is advantageous compared to RaidZ for certain cases (especially for users not concerned by storage efficiency which will include many newcomer enthusiasts, who will look at the slideshow for information).

As the guide clear;ly covers RaidZ and doesn't defer to the Primer, and mirroring without RaidZ may well be as relevant to the slideshow's readers, could you reconsider for other newcomers, and update the slideshow to cover mirroring, at least enough that a viewer can understand how mirroring fits in and pros/cons, in the sections where RaidZ is discussed.
 

danb35

Hall of Famer
Joined
Aug 16, 2011
Messages
15,455
most of the "ZIL" stuff is actually talking about SLOG.
That may be because there's always a ZIL, and what's usually described as a ZIL is in fact an SLOG.
 

Mirfster

Doesn't know what he's talking about
Joined
Oct 2, 2015
Messages
3,215
That's possibly a second rate answer (very sorry!)
I directed you to that for several reasons:
  1. Not wanting to fill up this thread with answers that are readily available
  2. Trying to get you to meet us half way
  3. So you could see the other links in there pertaining to your question regarding
There's almost no discussion of mirroring vs RaidZ and pros/cons.

For reference in the ZFS Primer there are several links that will assist in this:
 

dc3dog

Cadet
Joined
Jun 11, 2016
Messages
8
Thanks, very informative.
 

xiongyw

Cadet
Joined
Sep 29, 2016
Messages
2
Thanks for the presentation for noobs like me.

Btw, for
To be honest.. I actually considered not including the exception. Who really will use that one exception? It's hard to believe someone would create a RAID0, then convert it to a RAID1. For noobs, do we REALLY want to be giving out a bunch of exceptions? I mean.. I dunno. I'm very torn on it. It almost seems better to say you can't add more drives than to try to give this exception.
I have a use case for that, although by the time I did it I have no clue this is an "exception" for "normal" cases:
I had a 500GB HDD containing some files using NTFS from Windows, and I wanted to "protect" those files on a ZFS mirror (using debian+zol), so I got another 500GB HDD... but I don't have extra HDDs for backup my files firstly....So:
1. create a RAID0 with the single new 500GB HDD
2. copy the files from NTFS to ZFS
3. attach the old HDD to the zpool

The following is my "log" for the status/history at that time:
Code:
443 bruin@gen8:/dev/disk/by-id$ sudo zpool status
444   pool: tank
445  state: ONLINE
446 status: One or more devices is currently being resilvered.  The pool will
447		 continue to function, possibly in a degraded state.
448 action: Wait for the resilver to complete.
449   scan: resilver in progress since Fri Jul 22 12:56:04 2016
450	 566M scanned out of 262G at 47.2M/s, 1h34m to go
451	 562M resilvered, 0.21% done
452 config:
453
454		 NAME										   STATE	 READ WRITE CKSUM
455		 tank										   ONLINE	   0	 0	 0
456		   mirror-0									 ONLINE	   0	 0	 0
457			 ata-WDC_WD5000AADS-00L4B1_WD-WCAUH0002092  ONLINE	   0	 0	 0
458			 ata-WDC_WD5000AVDS-63U7B1_WD-WCAV9EM28882  ONLINE	   0	 0	 0  (resilvering)
459
460 errors: No known data errors
461 bruin@gen8:/dev/disk/by-id$ sudo zpool history
462 History for 'tank':
463 2016-07-22.08:38:21 zpool create -f -m /home/bruin/work/tank -o ashift=12 tank ata-WDC_WD5000AADS-00L4B1_WD-WCAUH0002092
464 2016-07-22.08:38:54 zfs set atime=off tank
465 2016-07-22.08:46:37 zpool import -c /etc/zfs/zpool.cache -aN
466 2016-07-22.12:50:05 zpool import -c /etc/zfs/zpool.cache -aN
467 2016-07-22.12:56:13 zpool attach -f tank ata-WDC_WD5000AADS-00L4B1_WD-WCAUH0002092 ata-WDC_WD5000AVDS-63U7B1_WD-WCAV9EM28882




Thanks again for the slides!
 

gpsguy

Active Member
Joined
Jan 22, 2012
Messages
4,472
To add the mirror after the fact, you'd have to use the CLI.

But, rather than creating the pool from the CLI, you should have done it via the webGUI. FreeNAS uses GPTID's to keep track of the devices.

And, it creates a 2GB swap file on each disk. While you don't want the server to use swap in place of real RAM, it's main purpose is to aid in disk replacements, if the replacement disk is slightly smaller than the original disk.

So, for disk #2, you'd should have added the swap partition. And, when adding the second drive to the pool, used it's gptid.

Thanks for the presentation for noobs like me.
 
Top