ZFS storage dismounts

[Jared Isham]

I am attempting to get my server set up and I have 15TB attached as a RaidZ2. I finally got the permissions to work properly but I am having trouble getting the zvol to stay attached/accessible. I am copying over 6TB of data from a local computer using Rsync and I got a failure report half way through. (I attached it to this post).

I am wondering if I have done something wrong. I have gone through the set up guide and set it up accordingly to the best of my knowledge.

Unfortunately I am not a network IT professional so "It's all in the guide" suggestions will put me in the exact same position I am now...or I will have it fixed in a few years when I can teach myself what the heck some of it is saying.

So, with that being said, does this report have any helpful information in it that could guide me in the right direction. I know I've got a ton of errors going, so I'm hoping to get them resolved. Fortunately, if I lose the data on the drive I have it elsewhere, as I am trying to migrate from a local drive to the server. I just want to make sure that it is operational before I start putting it to work.

Some concerns I do have is I can only go as high as 8GB of ram, should I look into a new motherboard with more RAM...I only have 4GB on it at the moment (I know it is below the minimum so I am thinking that could be the results of working below the minimum requirements)

The log is attached. Oh, and the 15TB is connected through eSATA

Thanks for any help

 

[Ericloewe]

How, exactly, are the disks attached?

 

[danb35]

The rule of thumb for RAM when using ZFS is 1 GB of RAM per TB of drive space. It can be fudged a bit, but you're way low. You're well under the minimum spec for using ZFS at all, which is 8 GB. I can't tell you if that's causing your problems, but it can't be helping.

 

[Jared Isham]

Ericloewe, Today at 2:38 PM

How, exactly, are the disks attached?​

I have a 8Bay External Enclosure connected via eSATA

danb35, Today at 3:20 PM

The rule of thumb for RAM when using ZFS is 1 GB of RAM per TB of drive space. It can be fudged a bit, but you're way low. You're well under the minimum spec for using ZFS at all, which is 8 GB. I can't tell you if that's causing your problems, but it can't be helping.​

I think what I'm thinking I'm gonna do is upgrade the system to 8GB of RAM so I can get all my data backed up and then do my work off of my local drives until I upgrade the server to be able to handle more RAM...is that a good idea or a bad one. I know the server needs some upgrading, but I'm hoping to put that off a few months. I just don't want to risk losing the data if migrate it to the new storage and then shut it down until the server can be upgraded.

 

[Ericloewe]

SATA port multipliers are terrible, and so are the devices that use them in their own right. It could be your enclosure that's causing your problems.

What's the rest of your hardware? How are the disks configured?

 

[Jared Isham]

Ah, bummer. I was hoping that I could make use of that before I spent the money transitioning to SAS ($$$).

Anyway, my set up is:
NewerTech MAXPower eSATA 6G PCIe 2.0 Controller Card
Sans Digital TowerRAID TR8M - 8 Bay SATA to eSATA (Port Multiplier) RAID-Ready Enclosure
8x 3.0TB Toshiba (by Hitachi) 7200RPM SATA 3.0

I have the disks set up as JBOD and the RAID configured as a RAIDZ2 with FreeNAS totallying 15TB of space.

I rebooted the system this morning and I'm not receiving any errors or alerts but that could easily be temporary.

 

[cyberjock]

Yeah, your setup is setup for maximum carnage. Port Multipliers are a disaster. They should never have existed, but where created with the SATA spec solely to compete with SAS. But they suck and are horribly unreliable.

The two things you need to fix.. get rid of that port multiplier and get up to 8GB of RAM. It also wouldn't hurt to switch to a PCIe controller that isn't by a "no-name" brand, especially considering you are having disk-related problems.

When it comes to money you can expect to have to spend some, even just to figure out if FreeNAS is for you. If this is a problem the best thing you can do is run it in a VM. Of course, performance will be terrible, but it will let you evaluate if FreeNAS is for you. If it is, then you can open your pocketbook and start spending.

 

[Jared Isham]

So if I was to replace my port multiplier with something like this http://www.newegg.com/Product/Produ...re=sas_enclosure_8_bay-_-16-111-173-_-Product would I be in a better position?
I would just need to get a SAS controller card to go with it. Still researching those.

My thoughts were, and tell me if I'm wrong, to upgrade the RAM to 8GB and finish copying over the data to the storage so I have it backed up there. Shut down the server and keep it out of operation until I upgrade the storage array to a SAS. I am assuming that if I just move the drives to the new enclosure in the same order then the RAID would be retained since it is a software controlled raid. I could be totally wrong, though. I'm basically trying to come up with the best shared storage option for video (My work is primarily in video editing and motion graphics). If I need to find a different solution then I'll do it but I am liking what I'm getting with FreeNAS - I'm not against putting the money into it, I just might be a little slow at making that happen.

 

[Ericloewe]

So if I was to replace my port multiplier with something like this http://www.newegg.com/Product/Produ...re=sas_enclosure_8_bay-_-16-111-173-_-Product would I be in a better position?
I would just need to get a SAS controller card to go with it. Still researching those.

My thoughts were, and tell me if I'm wrong, to upgrade the RAM to 8GB and finish copying over the data to the storage so I have it backed up there. Shut down the server and keep it out of operation until I upgrade the storage array to a SAS. I am assuming that if I just move the drives to the new enclosure in the same order then the RAID would be retained since it is a software controlled raid. I could be totally wrong, though. I'm basically trying to come up with the best shared storage option for video (My work is primarily in video editing and motion graphics). If I need to find a different solution then I'll do it but I am liking what I'm getting with FreeNAS - I'm not against putting the money into it, I just might be a little slow at making that happen.

It's better, but it still has a few points that need clearing up:

• The power supply it uses is atrocious, doesn't even have Active PFC (you can tell by the fact that it has a voltage selection switch), but it seems to be close to a standard ATX PSU with molex plugs, so a bit of hacking should fix that.
• Cooling may be inadequate (it's hard to tell from the pictures, be sure to research it a bit)
• Since it doesn't have an expander, you're most likely limited to ~1m cable length from controller to drive, due to the SATA signalling used. The advantage is that you don't need an SAS controller, if you have 8 free SATA ports (you can just get reverse breakout cables), and if the cable length limitations aren't a problem in your case.
• It sounds awfully expensive for a case and PSU for a bunch of drives, especially since it has no controller.

You can build your own enclosure with an SAS expander, which allows for SAS signalling between server and storage enclosure, by using an ATX case with tons of 5" drive bays, adding those hotswap docks, an SAS expander, and an ATX PSU hacked to turn on when the main system is powered on (this last part is a bit too DIY for most, but you can easily have it always on with one those dummy cables typically used to test PSUs). You can also get something a bit more professional (an euphemism for expensive, I'm afraid) that fits your needs without the dubious quality of the one you linked.

 

[Jared Isham]

Either way it will be a bit pricey I'm noticing. The SAS expanders I'm finding start around $300 and go up from there. Then cases that will have the space I'm looking for are much cheaper but finding something with the right amount of bays for drives is a little difficult.

I am still wondering if it is safe to get the data on the drive and shut it down until I can upgrade the server.

 

[cyberjock]

SAS expanders aren't cheap. $300 is about the cheapest you are going to find unless you are comfortable with ebay. I will tell you that at that price point for the Sans Digital it certainly does not use a SAS expander. I see no information that makes me think its SATA or SAS, so we're kind of stuck with guessing unless you call them. And then you have to deal with the potential problem where they lie to you and tell you it's a SAS expander when it's not.

I summarily ignore those devices out of habit because of the problems they've been for so many people(and not just in FreeNAS).

Sorry, but if you want to do this right you're going to have to spend cash. :(

To add to that, if your system is so old it's limited to 8GB of RAM, you stand a good chance of being disappointed with the total performance. And since RAM is "the" way to add performance you won't be able to do anything without gutting your system.

So I think you should do some soul searching and either decide to:

1. Continue down this path that many or may not go well for you.
2. Go to some alternative to FreeNAS.
3. Decide to spend the cash to do things right.

 

[Jared Isham]

So I think you should do some soul searching and either decide to:

1. Continue down this path that many or may not go well for you.
2. Go to some alternative to FreeNAS.
3. Decide to spend the cash to do things right.

Option 1 is definitely not an option for me. I can't risk losing my data.

I'll continue doing some FreeNAS alternate searching.

I definitely want to do it right since a lot of client data will be on the storage. As far as other solutions, I think I could upgrade my Motherboard for not a crazy amount of money and be able to expand to 64GB of RAM.

I was thinking, and this might be a solution for temporary until I put the cash down, get a controller that would allow me to mount all drives in my current system. That way I would not be dealing with an external enclosure...I'm still researching on cases and controller cards and such.

 

[cyberjock]

Well, you shouldn't have 8GB of RAM for a 15TB pool. That's why I didn't list "do some 1/2 job". There really isn't any kind of 1/2 job, except to continue using what you are using, which doesn't work.

As for 64GB of RAM, I'm curious to know what your plan is. I don't know of any boards that can use that much that don't become rather expensive. Even my $175 motherboard is capped at 32GB of RAM.

 

[Jared Isham]

There are plenty of them that get bad reviews, but this one isn't too badly priced with some decent reviews. http://www.newegg.com/Product/Product.aspx?Item=N82E16813138372 The cost hits if you try maxing out the RAM, but still rather affordable with all things considered.

 

[cyberjock]

You do know that AMDs aren't well known to actually work properly with FreeNAS, right?

 

[david kennedy]

Either way it will be a bit pricey I'm noticing. The SAS expanders I'm finding start around $300 and go up from there. Then cases that will have the space I'm looking for are much cheaper but finding something with the right amount of bays for drives is a little difficult.

I am still wondering if it is safe to get the data on the drive and shut it down until I can upgrade the server.

Agreeing with cyberjock here. This seems like a "1/2 job" solution.

After you buy the Sans Digital 8-Bay SAS/SATA unit, and the SAS controller card would you be better off just getting a supermicro box with more hot-swap bays and a cleaner installation?

 

[Jared Isham]

You do know that AMDs aren't well known to actually work properly with FreeNAS, right?

I did not know that. I have been using AMD because of the cheaper price point and it is fairly fast for my usage. But here is an i7 compatible board that is 64GB of RAM capable. A little more expensive, but not that much. The chips are the major price difference between the two systems. http://www.newegg.com/Product/Product.aspx?Item=N82E16813157289

Agreeing with cyberjock here. This seems like a "1/2 job" solution.

After you buy the Sans Digital 8-Bay SAS/SATA unit, and the SAS controller card would you be better off just getting a supermicro box with more hot-swap bays and a cleaner installation?

​

I'm gonna avoid buying the Sans Digital unit to get a more reliable setup, but I was gonna see if there was a way to avoid using an additional enclosure and house the server and the drives in one unit. That would require upgrading my motherboard, cpu, ram, etc.

​

I'm looking at other options but currently keeping the server on standby/shutdown.

 

[david kennedy]

I did not know that. I have been using AMD because of the cheaper price point and it is fairly fast for my usage. But here is an i7 compatible board that is 64GB of RAM capable. A little more expensive, but not that much. The chips are the major price difference between the two systems. http://www.newegg.com/Product/Product.aspx?Item=N82E16813157289

​

I'm gonna avoid buying the Sans Digital unit to get a more reliable setup, but I was gonna see if there was a way to avoid using an additional enclosure and house the server and the drives in one unit. That would require upgrading my motherboard, cpu, ram, etc.

​

I'm looking at other options but currently keeping the server on standby/shutdown.

---

I suspect a lot of us started out the same way. I had a device similiar to what you are currently using (port multiplier/SATA expander).
It worked OK for what i needed it for (add extra drives to my mac).

Later I moved to a NAS and realized that sort of thing is less then ideal for a "24x7" running system(low quality powersupply and fans).

Know what was one of the "kickers" with my multiplier? Once I discovered I wanted a ZFS solution I also discovered Solaris cant use multipliers. Things change since then and you can get ZFS on non-solaris based systems, but the point is to buy what you need, not try to shoe-horn things together. Sometimes they work out, but often are more prone to being problematic.

I now have real "rackmount servers" running in the cold-room of my house. They cost more then the original PC based stuff i had, but work a lot better.

 

[cyberjock]

Ok, nice board. I have a similar board for my desktop. But that board is filled with hardware you do not want on a FreeNAS box. Audio chips, extra Asrock special features, that stuff may cause problems with FreeBSD/FreeNAS and may prevent proper operation.

Now do you see why we recommend server-grade hardware?

And now you hopefully see why I said 'I don't know of any boards that can use that much(RAM) that don't become rather expensive."

You start doing the "right" thing and suddenly prices are out of control. That $75 motherboard is suddenly a $250+ motherboard and that $100 CPU is suddenly a $300 CPU. Then, to boot, you haven't even looked at the cost of 8GB ECC RAM sticks(which are about $100 each!). So now you are talking well over $1200 for system with 64GB of RAM.

Now you know why I gave you those three options and said 'soul searching' because this *very* quickly gets more expensive then most people realize it would be.