'Permanent Errors', but can't find the files?

[gait]

Hi.

FreeNAS 8.0.2 GUI showing permanent errors have been detected in the following files:
z01:<0x198002> but I can't find these files?

Anyone?

Thanks.

 

[cyberjock]

That shows that metadata in ZFS is corrupted. You should backup your zpool and recreate it.... as a RAIDZ2. RAIDZ1/RAID5 "died" in 2009. See the link in my sig for more information.

 

[gait]

That shows that metadata in ZFS is corrupted. You should backup your zpool and recreate it.... as a RAIDZ2. RAIDZ1/RAID5 "died" in 2009. See the link in my sig for more information.

Thanks - I'll look into that. Does your guide go into how to backup an entire zpool?? I am very green with his!

 

[cyberjock]

No. My guide is kind of an "intro to FreeNAS/ZFS". My sig link for my guide isn't for you specifically. I put it in my sig to give it attention since so many threads are answered in my guide if people would stop and read it :p

 

[paleoN]

FreeNAS 8.0.2 GUI showing permanent errors have been detected in the following files:
z01:<0x198002> but I can't find these files? .

Did you try clearing the errors and running another scub?

Code:

zpool clear z01

zpool scrub z01

 

[cyberjock]

Does a scrub "correct" those errors? I mean, it can't correct them because it has no parity... if it did it would correct them during a scrub. I just thought that if you do a clear and then a scrub they'll just rear their ugly head again.... would be interesting if the op did a clear, verified the "errors" went away, did a scrub and then checked to see if the errors came back...

 

[Warren Jenkins]

No. My guide is kind of an "intro to FreeNAS/ZFS". My sig link for my guide isn't for you specifically. I put it in my sig to give it attention since so many threads are answered in my guide if people would stop and read it :p

Cyberjock
I'll read it!! I've also dowloaded your PP presentation on it - all good stuff.

I must say I initially liked FreeNAS, but after having put it into a live business environment, when there is a problem with it (which there seems to be many!) it is a pain in the butt for users. We also have another problem in that one of our 'main' NAS's keeps crashing several of our hosted VM's everytime it does a large file transfer to an NFS share on the same NAS. This has rendered the it useless as we can't run backups on it now, but that's another story for another thread.

paleoN
I have cleared the error, but to no avail, even after doing a scrub. There is a 6.4Tb pool with 3Tb used and I don't have anywhere to back this up to now to even consider recreating the pool :(. This is where in a live environment when issues like this occur it gets me nervous!

Woz.

 

[cyberjock]

Warren,

Don't take this the wrong way, but it looks like you are trying to learn FreeNAS while using it in production. For example, your screenshot shows ada0 is part of your pool. FreeNAS always puts ZFS in a partition, so someone did that manually from the command line. As I was just telling someone else earlier today, if the GUI has the ability to do what you need it to do, you should be using it unless its broken. In this case, the replacement of whatever disk to ada0 should have been done from the GUI.

Additionally, and I'm not sure who's fault that is. But in a production environment a RAIDZ1 is completely unacceptable for reliability. RAID5/RAIDZ1 'died' four years ago. That's ages ago in computing. Why someone would willingly choose to use a configuration that is long since deemed unreliable is beyond me. About 80% of all users that come to the forum with lost data lost it solely because they chose to use a RAIDZ1.

When it comes to building file servers(not sure how much experience you have...) if you don't build it right the first time it will be nothing but a pain for the entire life of the server. Or you figure out what you did wrong and fix it in a way that it will never cause you problems in the future. It requires careful planning and execution.

As for the VM thing, its thoroughly documented that ESXi + NFS + ZFS = bad performance. Those 3 really don't go well together. If you search the forums there's lots of things you can try, but it will require you to tune ZFS, buy more hardware, or both. I'm not sure if your "main" NAS is a FreeNAS server, but I will warn you that we've had users that have lost their pools because of kernel panics. Partially saved data shouldn't trash ZFS, but people tweak ZFS in ways that makes that no longer true.

My understanding is that once you have corruption that is listed in zpool status the only way to correct it is to delete the offending item. In this case its metadata. I don't even know where I'd begin to figure out what that is actually for...

Don't take offense to this stuff, just making observations. I know that forum settings aren't always a good setting to start drawing conclusions about what is or isn't your fault. I'll never know the "whole picture" with your story, and for all I know you got hired last week to do this stuff. :P

 

[Warren Jenkins]

Hi Cyberjock.

Thanks for your time on this and absolutely no offence taken at all.

This setup was put together by a former colleague of mine and I believe it was as new to him as it was to me. He ran with setting it up and it worked fine for quite some time. It appears now that cracks are starting to apear which may well be because of how it is configured. Since his departure, I have tried to gen up on it more which in hindsight I should have been doing during its inception - my bad.

I always do work from the GUI as I'm not a command line junkie and with that said I'm a bit confused about why you think ada0 has been replaced as it hasn't. What makes you think it has?

I think I am going to have to get some more hardware and get everything off this box and rebuild it from scratch and steer clear from NFS then. If not NFS, What do you recommend - ISCSI?? I have hardware containg six 1Tb drives in another box that I could setup with RAIDZ2 giving better redundancy, but not sure about performance??

 

[cyberjock]

If you look at your screenshot it shows:

ada1p2
ada2p2
ada3p2
ada4p2
ada0
ada5p2

adaX is your device number, and p2 shows Partition 2 on the disk. Notice how ada0 has no partition designation. That's not how FreeNAS works if you replace a disk from the GUI. It would always create partitions. Even if you disable the swap space in the GUI(not recommended) the zpool would still have gone in ada0p1. So someone did a disk replacement some time in the past and did it from the CLI. Nothing bad about it, but its not something I recommend or endorse unless the GUI can't do its function. If the GUI couldn't' handle disk replacements there would be lots of complaints.

As for the iscsi, NFS, or CIFS thing, I'm not a fan of iSCSI. It can be used successfully, but it seems to take a lot of work for some people to get reasonable performance. NFS with ESXi for the datastore is just slow. It requires some work to overcome too.

 

[Warren Jenkins]

Thanks again mate. Looks like I've got some work to do! Do you have a preference for what to have the EXSi datasotes on? Would be interested to know as if I'm going to rebuild all of our NAS's, I want to be sure I get the best possible configuration as we have seven live systems and a further four to go in soon so would definitely like to get them right from the get go.

Appreciate all your comments.

Thanks.

 

[cyberjock]

I don't have any recommendations aside from read and read and read some more.

ESXi with iSCSI can be explained in http://support.freenas.org/ticket/1569 along with various things to consider and how you can tweak your own system. It is NOT a "do these things and all will be fine" but more of a "here's what you should look at and some parameters to try".

ESXi with NFS can be assisted with a ZIL or SLOG

Obviously the more powerful the server the better. More RAM is almost always better too.

The problem with iSCSI and NFS are NOT FreeNAS problems(some are related to using a CoW file system), so even if you went to Windows or Linux you can potentially have the exact same problems. The problems can be resolved by tuning your system for your exact workload. My settings wouldn't help your setup just like your settings won't help my setup. It's very much a personal thing for your server.

In some cases switching from ZFS to UFS can help.

But as I said above its not a "put it in and it'll work". It's going to be a battle you fight until you get the knowledge and experience to handle your server properly(assuming your hardware isn't so limited it can't handle the load regardless).

There's probably been 100 threads this year just on your problems with ESXi and FreeNAS. You aren't alone and many people give up before actually solving the problem because they don't want to take the time to do it right.

 

[paleoN]

paleoN
I have cleared the error, but to no avail, even after doing a scrub.

One last thing:

Code:

zpool clear -Fn z01

Don't forget the n and only report what it says, if anything.

There is a 6.4Tb pool with 3Tb used and I don't have anywhere to back this up to now to even consider recreating the pool :(. This is where in a live environment when issues like this occur it gets me nervous!

Far past time to go purchase something to backup to then. 3 TB drives are certainly cheap enough.