I seem to have a love/hate relationship with FreeNAS and ZFS

[brando56894]

I guess you're asking for advice?

You would be better served with a SAS controller instead of just the built-in SATA of the system board. You don't tell much about your hardware, but it is a simple matter to pick up a used 24 bay server from eBay and get that working with FreeNAS.
A setup with 50ish TB of storage would be easy as long as you can afford to buy the hardware. I did have an IBM M1015 HBA that I had all my drives connected to but that randomly died when I was swapping our hardware a week or two ago, so now I'm back to using the SATA ports on my board. I'm waiting on getting my Asus X99-W so I can install my Xeon E5-1650 in it and get rid of my SuperMicro X10SDV-F-0

Sent from my SAMSUNG-SGH-I537 using Tapatalk

Not really, just a general rant/word vomit/thoughts on FreeNAS compared to unRAID haha I wish I had the space for a rackmount server but I live right in a 2 bedroom apartment right outside of NYC (and plan to move into NYC proper soon) so space is at a premium. I have a hard enough time hiding my 12 + 1 + 2 bay full tower case that houses my current server.

@brando56894 I hear ya on the funds issue and i'm sure a number of others do as wells but are you in the GUI enough to worry about it? I've lost a fair amount of data that I believed I could reclaim easily and was mistaken on some of it, that was the beginning of my raid journey. I had mirrors in my desktop for years after that and liked it for the same reasons you did but the lost space became an issue when i started to fill them up. Then i found freenas and raidZ2 and I've been happy ever since it's takes a long time to restore from a backup or re-rip all your dvd's (still haven't completed that....) I feel the time I've lost having to restore the data I've lost is worth the 25 or so percent of inaccessible storage +the offsite storage (if you do the math I actually have less then %50 of the storage usable at the moment if you include off-site storage lol). I'm upgrading to 8TB drives as my 2TB drives pass the 40k hours mark and start failing (this weekend lost one) I did what @Chris Moore suggested and grabbed a noisy, really noisy used multibay case online and it has made everything easier from backing up using @Arwen method to replacing failed drives/adding addition storage.

I can't leave things alone and love to watch statistics, so I'm always tweaking something or trying something new so I'm always in the GUI hahaha About 90% of my data can be reacquired within about a week or so, so I've come to the conclusion that it doesn't really matter if I lose it because it's all "disposable" anyway. All my important stuff is already backed up in multiple places and equals maybe about 10-20 GB hahaha I started out using RAIDZ when I first found FreeNAS 3 years ago and was baffled and kind of pissed off regarding the large amount of pre-planning that went into creating your pool since I was used to RAID5 which I had had in my server for a few years and could upgrade the array relatively easily.

RaidZ2 will actually write faster than mirrors for sequential transfers. I'd ignore the ugly gui personally... afterall... are yu really going to be staring at it much once its setup? Meanwhile you don't *have* to respect the 80% ruleguideline. Things slow down significantly at 90% though.

Is that so? Interesting! I had only used RAIDZ for a few months before I ditched it and went to mirrors and can't remember what the performance was like. I've always read that out of Striped Mirrors, RAIDZ and RAIDZ2, Mirrors were the flat out fastest and RAIDZ2 was the slowest since it has to calculate double parity.

I have a server at work that uses 6TB drives and it took over 36 hours to resilver one of the disks when I had to replace it. The pool is only 33% full, I can't imagine how long it will take to resilver once the pool is closer to capacity.

Ouch! This was one of the reasons why I went with striped mirrors, since you can resilver a new drive in just a few hours. With my current array in unRAID (4x 4 TB for data and 1x 4 TB for Parity) it takes about 18 hours to do a parity check, and I will be adding 2x 4 TB in soon

For the home user it might not seem like such a big deal, but you lose all your kids pictures, you will wish you had the extra parity drive.

This is why all my irreplaceable stuff goes in the cloud. I trust Google with my data more than I do myself, considering they're never made me lose a file but I've lost TBs due to my own dumb fault haha. All my pictures are on Google Drive/Google Photos and Facebook.

 

[cobrakiller58]

This is why all my irreplaceable stuff goes in the cloud. I trust Google with my data more than I do myself, considering they're never made me lose a file but I've lost TBs due to my own dumb fault haha.

LOL they definitely spend more time protecting my files than I do.

 

[Chris Moore]

I have a hard enough time hiding my 12 + 1 + 2 bay full tower case that houses my current server.

I don't understand why it needs to be hidden. I also don't understand living in an apartment. Move to the country where you can have a house.

I can't leave things alone and love to watch statistics, so I'm always tweaking something or trying something new

That is generally not a good thing to do if you want a stable, reliable server. Sounds more like you are looking for a learning environment.

I've always read that out of Striped Mirrors, RAIDZ and RAIDZ2, Mirrors were the flat out fastest and RAIDZ2 was the slowest since it has to calculate double parity.

There is a time value to calculating parity so you loose some raw speed going from RAIDZ, with about one disk worth of parity data to RAIDZ2, with about two disks worth of parity data to RAIDZ3, with about three disks worth of parity data. Mirrors are not intrinsically faster, the thing that gives a pool of mirrors greater performance is the number of vdevs.

Ouch! This was one of the reasons why I went with striped mirrors, since you can resilver a new drive in just a few hours.

The rebuild speed is affected by the write speed to the disk being rebuilt. In a mirror, it is just a straight copy from one disk to the other, as fast as the data can be read and written, it is done. In any other type of ZFS (RAIDZ, RAIDZ2 and RAIDZ3) , it is affected by the need to reconstruct the data from parity to figure out what needs to be written in the rebuild. The system from my work has well over 120TB of data and that affects the rebuild time as the server was having to analyze about 25TB of data to recompute the contents of the drive being replaced. A mirror would rebuild more quickly, but during the time that a drive is being replaced in a mirror, there is no redundancy, unless it is a 3way mirror... If I had it to do again, I would configure the storage server at work differently so the vdevs had fewer disks and there were more of them. The lower disk count in the vdev, the faster the rebuild because the server only needs to analyze the other disks in the vdev to recompute parity. All the other vdevs are basically idle during a rebuild.

This is why all my irreplaceable stuff goes in the cloud. I trust Google with my data more than I do myself

I try to keep my data private. I figure that it is mine and I don't want it to be exposed to the world at large. Any data that google has, it is there data, not yours.

 

[Jailer]

Oh, I thought *you* used them for target practice. ;-)

I do but just the platters as they explode quite spectacularly when struck by a fast moving object. This is of course after I've removed the magnets for more dubious needs.

 

[farmerpling2]

I would counsel having a backup of said pictures rather than going to raidz3.

Interesting thought...

During the 4-6 days while the resilver is running and new data is updated, a backup has to run, which will slow down the resilver even more.

A backup does not replace redundancy and redundancy does not replace a backup.

Generally speaking, all the additional parity drive does is, reduce the chance of the RAID set failing, which is what RAID is all about.

If you lose the RAIDZ2 because too many drives failed, the time to restore and resetup the RAIDZ2 is not going to be counted in minutes or hours, IMHO.

If it is not for business and just for movies and TV shows, a RAID Z1 is fine as long as he understands the pain in reripping all those DVD, etc.

It's all about risk management, to which we both agree.

 

[Stux]

Mirrors are not intrinsically faster, the thing that gives a pool of mirrors greater performance is the number of vdevs.

Exactly right.

N is the number of disks in a vdev, P is the disks of Parity/Redundancy

A Mirror reads at N (so double for a 2 way mirror) an HDs read speed. And writes at the N-P (so same speed as single disk for 2-way)

Raidz[123] read at n-p and write at n-p, but since n can be 6 or 8 or greater....

(Assuming COU is not a bottlebeck)

The above holds for sequential. For Random IO the number of vdevs is more important, and mirrors win that case.

 

[farmerpling2]

Exactly right.

N is the number of disks in a vdev, P is the disks of Parity/Redundancy

A Mirror reads at N (so double for a 2 way mirror) an HDs read speed. And writes at the N-P (so same speed as single disk for 2-way)

Raidz[123] read at n-p and write at n-p, but since n can be 6 or 8 or greater....

Hmmmm.... I must not be understanding the above or maybe how ZFS works...

A standard RAID1 mirror reads up to 2N (separate reads to each drive so twice the performance). There are some RAID1 drivers that will only read from 1 drive, which is sad because of performance increase that can be gained.

A standard RAID10 is about twice as fast as a RAID1.

RAID5 (3 drives) just needs to read 2 of the drives. Good drivers would have it NOT use the parity so there is no overhead to process the parity. If parity is needed, it just is a matter of XORing the 2 drives data and the correct output is created. Pretty low overhead, but a small increase.

ZFS:
I would expect (my guess) RAIDZ1 with 2 data drives to read from 2 drives that are not parity, do a checksum to verify the data is correct. The checksum adds a short time of CPU overhead (some small amount of microseconds I would guess). If it has to go to parity then there is the process of taking parity data and making it into the end result along with the checksumming.

ZFS is slightly slower because of the checksumming processing.

Going to RAIDZ2 and Z3 should be roughly the same, I would think. No need to go to parity under regular situations. If parity is required, then the overhead in CPU increases slightly.

ZFS checksum processing is not very expensive. Being forced to use all of the parity drives, because data drives failed, increases CPU usage a little more, but nothing major.

Am I wrong in how ZFS works?

RAID10 in ZFS should be roughly the same I/O load as regular RAID10, with a slight usage of CPU for checksumming.

Writes in all cases require a write to all drives ZFS and regular.

If compression is turned on for ZFS there is some amount more overhead on the write and a small amount for decompression. An interesting situation can occur where RAIDZ1 can reduce I/O's because of data that is greatly compressible. In those situations, it might be possible to compress 5 or 10 blocks of data into 1 block of ZFS storage. The write then becomes smaller ad the read is smaller!

This really shows its face in data where a lot of the same characters are used (blocks of '0' [NULL] is common in crash dumps, as an example). Text files are usually quite compressible allowing them to shrink by 100% or more. A 100,000 block file of text might be compressed down to 50,000 blocks are less with a lot of repetition.

Maybe I will go look at the code to see how it works. I am curious now...

 

[Chris Moore]

I would expect (my guess) RAIDZ1 with 2 data drives to read from 2 drives that are not parity, do a checksum to verify the data is correct. The checksum adds a short time of CPU overhead (some small amount of microseconds I would guess). If it has to go to parity then there is the process of taking parity data and making it into the end result along with the checksumming.

ZFS always checks the checksum data to verify that the data delivered is the data that was stored. If the data read does not match the checksum, it then recalculates the data using the parity data. It is all about delivering the correct information.
If you think of RAIDZ1 with only 2 disk, it doesn't work any different from a mirror. The power of any of the RAIDZ levels is spanning the reads and writes across multiple disks, more than 1, and then you have only a tenth of the data being written to each disk in a vdev instead of all the data for the vdev being written to one drive and duplicated on another. I don't know if I can explain it very well. It doesn't work the same as regular RAID, that is certain.

 

[Stux]

A standard RAID1 mirror reads up to 2N (separate reads to each drive so twice the performance). There are some RAID1 drivers that will only read from 1 drive, which is sad because of performance increase that can be gained.

N is already 2 in a mirror, so it reads at n (2) and writes at n-p (1)

 

[brando56894]

I don't understand why it needs to be hidden. I also don't understand living in an apartment. Move to the country where you can have a house.

I seriously hope this is sarcasm and not honest help, otherwise you just sound like an a$$hole.

That is generally not a good thing to do if you want a stable, reliable server. Sounds more like you are looking for a learning environment.

Some of us don't have the money and space to have two full blown servers: one for a lab environment and one for production. I've setup labs before but I don't find them too useful because they don't get the "real load" like my production server does.

Mirrors are not intrinsically faster, the thing that gives a pool of mirrors greater performance is the number of vdevs.

Umm yea they are because no parity calculations need to be done with mirrors, since they are an exact copy of the data that is on the other drive. When a drive is lost, the data doesn't need to be rebuilt from bits and pieces of the other drives, it's just copied from one disk to another.

I try to keep my data private. I figure that it is mine and I don't want it to be exposed to the world at large. Any data that google has, it is there data, not yours.

In this day and age there is really no such thing as privacy, you're being watched on and off the internet. Unless you took those pictures on a film DSLR and developed them yourself, your pictures were never private in the first place.

 

[danb35]

Unless you took those pictures on a film DSLR

What is a film DSLR? And why would pictures taken on any sort of standalone digital camera not also be private?

 

[gpsguy]

What is a film DSLR? And why would pictures taken on any sort of standalone digital camera not also be private?

I wondered the same thing. Yes, there might be personal metadata - but one can wipe the EXIF data. While I am a "gpsguy", I disable the feature to add GPS coordinates to my smartphone pictures.

If you are still using film, who develops it? Unless you do it yourself, then I wouldn't consider it to be private.

 

[anodos]

You're supposed to sell them when you're done with them?

I once stripped a bunch of old drives apart and used them to make armor for my boys.

 

[Chris Moore]

I seriously hope this is sarcasm and not honest help, otherwise you just sound like an a$$hole.

Sure, I am joking, is like yesterday I told someone to move out of California to a place that isn't insane.

In this day and age there is really no such thing as privacy, you're being watched on and off the internet. Unless you took those pictures on a film DSLR and developed them yourself, your pictures were never private in the first place.

I do have plenty of data that is private. I go to lengths to try and ensure as much privacy as I can, but some of my data has never touched the internet, plus I run a firewall inside a firewall, different IP ranges, at home to try and make double sure that nobody can hack into my network.

 

[brando56894]

What is a film DSLR? And why would pictures taken on any sort of standalone digital camera not also be private?

I wondered the same thing. Yes, there might be personal metadata - but one can wipe the EXIF data. While I am a "gpsguy", I disable the feature to add GPS coordinates to my smartphone pictures.

If you are still using film, who develops it? Unless you do it yourself, then I wouldn't consider it to be private.

Haha yea, I wasn't thinking too much when I wrote that, I was thinking of an SLR I can never remember what the acronym means since I'm not a big photo guy Pictures taken on a digital camera could "phone home" when you connect them to your PC, sure it's a long shot and you wouldn't know unless you looked at the source code, but if you're worried about storing photos in an encrypted cloud, I guess you should be cautious of that as well. All I meant was the only way that you can be sure that only you see your pictures is to take them with a film camera and develop them yourself or use a digital camera and connect it to a pc that is never connected to the internet.

Sure, I am joking, is like yesterday I told someone to move out of California to a place that isn't insane.

You can never tell sarcasm via text, that's what got me in trouble with my ex for 5 years hahaha

I do have plenty of data that is private. I go to lengths to try and ensure as much privacy as I can, but some of my data has never touched the internet, plus I run a firewall inside a firewall, different IP ranges, at home to try and make double sure that nobody can hack into my network.

Wow, a bit paranoid are we? hahaha I used to do stuff like that than I realized "no one is going to get into my network as long as I follow standard security practices, such as port obfuscation" since 95% of attacks are either by malware or port scans. Also double firewalls and strict IDS rules made want to punch babies since it was extremely restrictive. Back in the day I would use ZoneAlarm Pro and Black Ice Defender (ah it brings back memories hahaha) along with a hardware firewall, and like one two to Anti-Viruses and multiple cleaning programs. I have since moved to a purpose built security appliance such as pfSense.

 

[gpsguy]

I'm a *big* photo guy. SLR = single lens reflex. Using the mirror and pentaprism, "what you see, is what you get". Adding the "D" was for digital.

a big photo guy

Yeah, I did that for many years. I still have a Beseler 23c Dichroic. I wish I could find a home for it.

is to take them with a film camera and develop them yourself

 

[Chris Moore]

Haha yea, I wasn't thinking too much when I wrote that, I was thinking of an SLR I can never remember what the acronym means since I'm not a big photo guy Pictures taken on a digital camera could "phone home" when you connect them to your PC, sure it's a long shot and you wouldn't know unless you looked at the source code, but if you're worried about storing photos in an encrypted cloud, I guess you should be cautious of that as well. All I meant was the only way that you can be sure that only you see your pictures is to take them with a film camera and develop them yourself or use a digital camera and connect it to a pc that is never connected to the internet.



You can never tell sarcasm via text, that's what got me in trouble with my ex for 5 years hahaha



Wow, a bit paranoid are we? hahaha I used to do stuff like that than I realized "no one is going to get into my network as long as I follow standard security practices, such as port obfuscation" since 95% of attacks are either by malware or port scans. Also double firewalls and strict IDS rules made want to punch babies since it was extremely restrictive. Back in the day I would use ZoneAlarm Pro and Black Ice Defender (ah it brings back memories hahaha) along with a hardware firewall, and like one two to Anti-Viruses and multiple cleaning programs. I have since moved to a purpose built security appliance such as pfSense.

I use pfSense for one of the firewalls

Sent from my SAMSUNG-SGH-I537 using Tapatalk

 

[brando56894]

Me too, do you have Snort running? That along with firewall logs should be enough to tell you that no one is getting into your network.