-
Important Announcement for the TrueNAS Community.
The TrueNAS Community has now been moved. This forum will now become READ-ONLY for historical purposes. Please feel free to join us on the new TrueNAS Community Forums
- Forums
- Archives
- FreeNAS (Legacy Software Releases)
- FreeNAS Help & support
- General Questions and Help
You are using an out of date browser. It may not display this or other websites correctly.
You should upgrade or use an alternative browser.
You should upgrade or use an alternative browser.
vfs.zfs.l2arc boot issue
- Thread starter deltaend
- Start date
- Status
- Not open for further replies.
High marks for knowing that I had a RAID controller, however, all of the disks are setup as JBOD and the raid controller doesn't detect smart errors on any of them at the moment. After watching your guide to ZFS, I can see other issues with my ZFS setup, however, I guess my main question is to why:
1) RAID controllers would cause issues when setup as JBOD.
2) VDevs shouldn't be larger than 10 disks.
Both of these are mistakes that I have made in my deployment. In addition to the above two questions, the third question that I have is... what are the odds that I can recover the ZFS and/or what would that process be?
Oddly, ZFS appears at first glance to be far more fragile than any standard hardware RAID configuration that I've seen.
1) RAID controllers would cause issues when setup as JBOD.
2) VDevs shouldn't be larger than 10 disks.
Both of these are mistakes that I have made in my deployment. In addition to the above two questions, the third question that I have is... what are the odds that I can recover the ZFS and/or what would that process be?
Oddly, ZFS appears at first glance to be far more fragile than any standard hardware RAID configuration that I've seen.
1. Because even in JBOD the RAID controller controls the disks to such an extent that it's just bad. You are technically not in JBOD mode. If you were you wouldn't have individual RAIDs like your screenshot shows. In effect, there is no difference between doing individual RAIDs for each disk and doing a full-blown RAID. *Any* hardware RAID in any fashion is a fail. To boot, most RAID controllers mask the SMART data, which is the other main reason. There's a ton more, but the reality is that any raid controller, unless it can run in passthrough mode (which yours clearly isn't from the screenshot) is a way to lose your data. The only viable solution is to get a real controller that is appropriate for ZFS.
2. Because of how ZFS handles it's data the writes become too wide, which hurts the pools performance.
3. Statistically, your chances are extremely slim of getting your data back. I used to offer data recovery services, but I don't even offer to do hardware RAID recovery because the chances are so low that I feel guilty taking people's money for something with such a low chance of success. There is no "process" to recover the pool from a failure of this type. In fact, there are very little "processes" for recoving a damaged pool. You literally have to figure out what the problem is with your pool, then attempt to resolve the problem. Unless you are a ZFS expert you are not going to understand what you are looking at when doing ZFS debugging, and doing things willy-nilly based on Google results are a really bad idea. Of course, you have nothing to lose at this point. But if you are considering paying some company 5-figures (no joke, that's how much it will cost) then I would *not* try to do anything yourself as you can make recovery impossible. This is why my noobie presentation says there are no recovery tools for ZFS. There are literally no options for those of us that didn't win the lottery recently. ;)
ZFS is far more reliable than hardware RAID, when you set it up properly and manage it properly. Large companies that are sick and tired of lost/corrupted data from hardware RAID are switching to ZFS. The real problem is that if you don't take care of ZFS it can blow up in your face (as it has unfortunately for you). The bad news is that you aren't the first one I've dealt with this month to make your exact mistake (I think you're the 4th or 5th) and I'll probably see one or two more before the end of the month. This is not fixable in software as this is considered user-error and not a programming problem. :/
A good analogy would be a car versus a horse-pulled carriage. You could argue with the fact that a car needs oil changes, new brakes regularly, new tires, and all this other maintenance and that makes the car inappropriate for transportation where a horse doesn't as much maintenance. The reality is that if you know what you are doing and manage it properly a car is far superior to a horse-pulled carriage.
This is the part where I say "You have backups, right?" and you respond with "Absolutely!". ;)
2. Because of how ZFS handles it's data the writes become too wide, which hurts the pools performance.
3. Statistically, your chances are extremely slim of getting your data back. I used to offer data recovery services, but I don't even offer to do hardware RAID recovery because the chances are so low that I feel guilty taking people's money for something with such a low chance of success. There is no "process" to recover the pool from a failure of this type. In fact, there are very little "processes" for recoving a damaged pool. You literally have to figure out what the problem is with your pool, then attempt to resolve the problem. Unless you are a ZFS expert you are not going to understand what you are looking at when doing ZFS debugging, and doing things willy-nilly based on Google results are a really bad idea. Of course, you have nothing to lose at this point. But if you are considering paying some company 5-figures (no joke, that's how much it will cost) then I would *not* try to do anything yourself as you can make recovery impossible. This is why my noobie presentation says there are no recovery tools for ZFS. There are literally no options for those of us that didn't win the lottery recently. ;)
ZFS is far more reliable than hardware RAID, when you set it up properly and manage it properly. Large companies that are sick and tired of lost/corrupted data from hardware RAID are switching to ZFS. The real problem is that if you don't take care of ZFS it can blow up in your face (as it has unfortunately for you). The bad news is that you aren't the first one I've dealt with this month to make your exact mistake (I think you're the 4th or 5th) and I'll probably see one or two more before the end of the month. This is not fixable in software as this is considered user-error and not a programming problem. :/
A good analogy would be a car versus a horse-pulled carriage. You could argue with the fact that a car needs oil changes, new brakes regularly, new tires, and all this other maintenance and that makes the car inappropriate for transportation where a horse doesn't as much maintenance. The reality is that if you know what you are doing and manage it properly a car is far superior to a horse-pulled carriage.
This is the part where I say "You have backups, right?" and you respond with "Absolutely!". ;)
Quote: This is the part where I say "You have backups, right?" and you respond with "Absolutely!". ;)
Actually, this is where I tell you that I was smart enough to be running this as a test server for the last year and was running test data through it (1TB per hour of second-tier Hyper-V backups). I had it running as a dozen different platforms (BTSync, OwnCloud, iSCSI, NFS, FTP, etc...), just to take note of all of the possibilities and options. About the only loss is the fact that I had the entire Microsoft Technet and MSDN image database on there from this year which is going to be a pain to BTSync back to it.
I only trust systems that I have thoroughly tested and I don't trust systems that can go completely offline when a single server dies. Is it possible for ZFS to get mirrored to another server in an active-active mode (outside of paying for TrueNAS) or is it only active-backup?
Also, as a side note, I didn't trust the whole USB thing so I put some regular HDD's in their on a hardware RAID 1. What are your thoughts on that?
Actually, this is where I tell you that I was smart enough to be running this as a test server for the last year and was running test data through it (1TB per hour of second-tier Hyper-V backups). I had it running as a dozen different platforms (BTSync, OwnCloud, iSCSI, NFS, FTP, etc...), just to take note of all of the possibilities and options. About the only loss is the fact that I had the entire Microsoft Technet and MSDN image database on there from this year which is going to be a pain to BTSync back to it.
I only trust systems that I have thoroughly tested and I don't trust systems that can go completely offline when a single server dies. Is it possible for ZFS to get mirrored to another server in an active-active mode (outside of paying for TrueNAS) or is it only active-backup?
Also, as a side note, I didn't trust the whole USB thing so I put some regular HDD's in their on a hardware RAID 1. What are your thoughts on that?
Hardware RAID of any kind is going to kill ZFS. Might not be today, and might not be tomorrow, and you might even luck out over the years. But it's just not a risk worth taking IMO when the fix is a $100 M1015. For the amount of money spent for all the other hardware that extra $100 drastically improves the chances of never having a problem.
When it comes to ZFS its a "do it right or do it twice" situation.
When I used to do consultation for people and they'd send me a list of hardware they already purchased it often included a RAID card. At that point I'd tell them at point blank that if they weren't going to buy an HBA that I wouldn't do the consultation. I stand behind the builds I provided to people, and I wasn't going to be involved in anything that might be a serious risk of data loss. It wouldn't make me feel good about myself and it wouldn't look good if people were coming in here telling everyone that I helped them build a system with hardware RAID that killed their pool.
My personal code of conduct is "do it right or do it yourself". Someone paying me to help them build a system makes me their expert. And if they're going to ignore their expert right off the bat is a sign that they're more interested in saving some money than preserving their data. That's not why ZFS exists and it's not going to mesh with ZFS' design considerations.
When it comes to ZFS its a "do it right or do it twice" situation.
When I used to do consultation for people and they'd send me a list of hardware they already purchased it often included a RAID card. At that point I'd tell them at point blank that if they weren't going to buy an HBA that I wouldn't do the consultation. I stand behind the builds I provided to people, and I wasn't going to be involved in anything that might be a serious risk of data loss. It wouldn't make me feel good about myself and it wouldn't look good if people were coming in here telling everyone that I helped them build a system with hardware RAID that killed their pool.
My personal code of conduct is "do it right or do it yourself". Someone paying me to help them build a system makes me their expert. And if they're going to ignore their expert right off the bat is a sign that they're more interested in saving some money than preserving their data. That's not why ZFS exists and it's not going to mesh with ZFS' design considerations.
Actually, the RAID I was referring to was for the system install, not for the ZFS drives. Are you saying that having any hardware RAID in a system, even if only used for the install array could possibly cause issues with the ZFS setup (even when the drives used for ZFS aren't attached to the RAID controller?)? What I didn't trust was the install to USB idea as I've had many issues with "install to USB drive" configurations in the past.
We already have an HBA ready for this server so that's not an issue. Is there the possibility of an Active-Active cluster configuration for ZFS and/or FreeNAS?
We already have an HBA ready for this server so that's not an issue. Is there the possibility of an Active-Active cluster configuration for ZFS and/or FreeNAS?
Oh, I see what you are saying. Thought you were talking about mirrored RAIDs in a pool as striped devices.
Hardware RAID will work on FreeNAS for the boot device, but it's not a good long-term option. Particularly because 9.3 will have ZFS as the boot device, so you are back to the whole "Hardware RAID + ZFS = bad" scenario. On the bright side you will be able to do ZFS mirrors of the boot device, so you can take advantage of ZFS' RAID to have redundancy.
I'm running installs from USB drive myself, and have for over 2 years. I've also had 6 or 7 friends and about 25-30 consultations use USB without problems. As long as you don't buy the bargain bin USB sticks it works fine. The root is read-only, so there's no risk of wearing out the USB stick prematurely. I'm using an 8GB Corsair Voyager if that means anything. I've also purchased some SATA DOMs for the upcoming 9.3 release and I plan to use them when upgrading to 9.3.
If you want to run 9.3 you can download it from download.freenas.org. It does have it's own update feature that works, so updating FreeNAS as the devs fix bugs is now a mouse click and 2 reboots. I will warn you though that you need more than 4GB of disk space as each update is a separate ZFS snapshot, so you need space to handle that. I'm running it on my FreeNAS Mini right now and it works pretty well. It has some issues (like no https support until you specifically setup the cert) but it has been usable for me. Definitely don't run 9.3 at present if you value your data because there could be unforseen bugs.
Hardware RAID will work on FreeNAS for the boot device, but it's not a good long-term option. Particularly because 9.3 will have ZFS as the boot device, so you are back to the whole "Hardware RAID + ZFS = bad" scenario. On the bright side you will be able to do ZFS mirrors of the boot device, so you can take advantage of ZFS' RAID to have redundancy.
I'm running installs from USB drive myself, and have for over 2 years. I've also had 6 or 7 friends and about 25-30 consultations use USB without problems. As long as you don't buy the bargain bin USB sticks it works fine. The root is read-only, so there's no risk of wearing out the USB stick prematurely. I'm using an 8GB Corsair Voyager if that means anything. I've also purchased some SATA DOMs for the upcoming 9.3 release and I plan to use them when upgrading to 9.3.
If you want to run 9.3 you can download it from download.freenas.org. It does have it's own update feature that works, so updating FreeNAS as the devs fix bugs is now a mouse click and 2 reboots. I will warn you though that you need more than 4GB of disk space as each update is a separate ZFS snapshot, so you need space to handle that. I'm running it on my FreeNAS Mini right now and it works pretty well. It has some issues (like no https support until you specifically setup the cert) but it has been usable for me. Definitely don't run 9.3 at present if you value your data because there could be unforseen bugs.
Any thoughts on the LSI 9211-8i? It has a built in RAID function but I'm assuming that it shouldn't be an issue if the RAID function is disabled, correct?
If you read our stickies you'll have the answer.. ;)
So, IT mode yes, regular mode, no.
Exactly.
Dennis.kulmosen
Explorer
- Joined
- Aug 13, 2013
- Messages
- 96
I can vote for the small innolite sata doms. Very realiable. I am using those on all my clients builds.
Only downside is that when handling them, you can with very little force be able to accidental connect the power the wrong way thereby frying the sata dom. Speaking from experience. ;-)
Sent from my iPhone using Tapatalk
Only downside is that when handling them, you can with very little force be able to accidental connect the power the wrong way thereby frying the sata dom. Speaking from experience. ;-)
Sent from my iPhone using Tapatalk
I might consider a USB drive but SATA doms seem to be going backwards in my mind. They aren't really any less expensive than a small 32GB SSD but they tend to mount poorly (or not at all) while running slow compared to a 32GB SSD. And then there is the power requirements that usually need an adapter or need an additional molex run to a random part of the case design to connect to them which looks messy. Lastly, it seems to me that we are all the way back to possible reliability issues as SATA doms aren't designed for lots of writes either. At the end of the day, it seems that SATA doms loose all of those features that are convenient about USB drives (very inexpensive, don't need additional power, extremely small, doesn't take up SATA/SAS port) while not being much if any better than a full on SSD drive that would be faster and have more space.
And by the way, FreeNAS aren't the only community to use SATA doms in their builds. I've asked several companies now why they use them in their appliances and after showing them the pluses of simply using a full SATA drive, they didn't really have an answer for me either so I'm still waiting for someone to tell me exactly why anyone would want a SATA dom.
Edit: Did a quick search for Innolite sata dom and realized that it's one of the few dom's I've seen without a power cable which makes it interesting, however, it's still quite a bit more expensive than a SSD of the same size and apparently needs to directly plug into the motherboard which is odd and possibly difficult depending on your build.
And by the way, FreeNAS aren't the only community to use SATA doms in their builds. I've asked several companies now why they use them in their appliances and after showing them the pluses of simply using a full SATA drive, they didn't really have an answer for me either so I'm still waiting for someone to tell me exactly why anyone would want a SATA dom.
Edit: Did a quick search for Innolite sata dom and realized that it's one of the few dom's I've seen without a power cable which makes it interesting, however, it's still quite a bit more expensive than a SSD of the same size and apparently needs to directly plug into the motherboard which is odd and possibly difficult depending on your build.
Last edited:
SATA DOMS are definitely a step forward compared to USB, in most areas.
The fact that they can get power directly from the SATA port (or often with a tiny cable that connects to a nearby connector on the motherboard) negates the need for the molex power. They are also small and easy to replace. The 32GB SATA DOM I have does almost 500MB/sec, so I don't see a case for arguing they are slow. Not to mention that in the case of FreeNAS the performance only matters for bootups and scrubs (remember FreeNAS runs from RAM disks). And one could argue that neither of those matter particularly much.
You are correct that if you are looking solely at the disk space, DOMs aren't a particularly good benefit. You're paying most for the convenience of cabling, replacement of the DOM is almost identical to a USB stick, etc. But as you can't really do a whole lot with the boot media itself, the size really doesn't matter. Even an 8GB DOM (at $25 or so ) isn't going to break the bank, and isn't even a significant investment.
Just like everything in life, there are tradeoffs. DOMs are what the FreeNAS Mini and TrueNAS systems use, and they've been pretty reliable and accomplish their object.
The only true argument I can see someone make against the DOMs is their size. While not much bigger than a USB stick, there is the possibility of covering adjacent SATA ports on your motherboard. But, even then, most people use something like an M1015 so often the motherboard ports are totally unused. Also many DOMs out there won't cover adjacent ports (mine does and I don't care because I use my M1015).
SATA DOMs also have far superior reliability as their flash controller is far more advanced and therefore increases the reliability compared to USB drives.
FreeNAS has pretty much always recommended USB sticks, but with 9.3 that will be changing to USB and/or DOM. Yes, with 9.3 and mirrors you could mirror a USB stick and DOM for your boot device. ;)
The fact that they can get power directly from the SATA port (or often with a tiny cable that connects to a nearby connector on the motherboard) negates the need for the molex power. They are also small and easy to replace. The 32GB SATA DOM I have does almost 500MB/sec, so I don't see a case for arguing they are slow. Not to mention that in the case of FreeNAS the performance only matters for bootups and scrubs (remember FreeNAS runs from RAM disks). And one could argue that neither of those matter particularly much.
You are correct that if you are looking solely at the disk space, DOMs aren't a particularly good benefit. You're paying most for the convenience of cabling, replacement of the DOM is almost identical to a USB stick, etc. But as you can't really do a whole lot with the boot media itself, the size really doesn't matter. Even an 8GB DOM (at $25 or so ) isn't going to break the bank, and isn't even a significant investment.
Just like everything in life, there are tradeoffs. DOMs are what the FreeNAS Mini and TrueNAS systems use, and they've been pretty reliable and accomplish their object.
The only true argument I can see someone make against the DOMs is their size. While not much bigger than a USB stick, there is the possibility of covering adjacent SATA ports on your motherboard. But, even then, most people use something like an M1015 so often the motherboard ports are totally unused. Also many DOMs out there won't cover adjacent ports (mine does and I don't care because I use my M1015).
SATA DOMs also have far superior reliability as their flash controller is far more advanced and therefore increases the reliability compared to USB drives.
FreeNAS has pretty much always recommended USB sticks, but with 9.3 that will be changing to USB and/or DOM. Yes, with 9.3 and mirrors you could mirror a USB stick and DOM for your boot device. ;)
- Status
- Not open for further replies.
Similar threads
- Replies
- 31
- Views
- 5K
- Locked
- Replies
- 3
- Views
- 1K
- Replies
- 10
- Views
- 2K
- Replies
- 15
- Views
- 6K
- Replies
- 14
- Views
- 5K
