FreeNAS not compressing and connection issues

Status
Not open for further replies.

mattg889

Cadet
Joined
Jan 2, 2018
Messages
7
Hi,
I am not sure where this thread should be so let me know if I should move it somewhere else. It kind of fits into a few different categories.
The system spec's are as follows:
Hp Proliant dl385 g7 server
64gigs of ram
24 amd cpu cores at 2.1ghz
32gb lexar flash drive for FreeNAS os
LSI hba controller for the drives
8-1tb drives set up in raid-z2
Smart data says the drives are all good
4 gigabit nic's setup in a load balance aggregation

I have been fighting with FreeNAS for over a week now.
Ill see if I can remember everything.
I originally had FreeNAS 11.0-U4 installed and it was working great. I saw that there were some updates, so I made my first mistake and I installed the updates which updated it to 11.1.release.
After that, I started having timeout issues when coping files to my shares, copying From the shares never has any problems even now.

I reinstalled completely to 11.0-U4 and restored the volume. But the connection issues continue. So I reinstalled to 11.1.release and the connection issues continue. I thought it might be a problem with windows, so I tried copying files from ubuntu to the share and it has the same problem, it pops up saying the connection timed out.
When copying from windows to the share, the copy speed slows down until it stops then I get a error with error code 0x8007003b
When copying from ubuntu to the share, the copy speed also slows down until it stops and ubuntu pops up with a window saying the connection timed out.

So after all that I copied all my files off the nas, and reinstalled it again, this time wiping the drives in the process in case it was something with them.
But the connection issues continue and now I have new problems. The volume is not compressing anything... lz4 is enabled and de-duplication is on (I figure there should be plenty of resources for dedup, cpu usage is between 20-40% when copying to)
I copied everything back to one share and it is showing that its using 920GB, the uncompressed file size for that share is 966GB this same share was between 600-700GB(I don't remember the exact number) before I made my first mistake of updating freenas. The compression ratio for the whole volume is 1.01x and I know it was over 1.10x before. I am not sure if compression isn't working, or if dedup isn't working, or both...

I think my bigger problem is the connection issues. Compression doesn't matter if I can't get the files to copy over... Although they might be connected.
Does anyone have any insight into whats going on here?
Thank you in advance
 
Last edited by a moderator:

m0nkey_

MVP
Joined
Oct 27, 2015
Messages
2,739
The volume is not compressing anything... lz4 is enabled
If the data you're putting in the dataset is already compressed, then lz4 will not be able to compress it any more than it already is.
and de-duplication is on (I figure there should be plenty of resources for dedup, cpu usage is between 20-40% when copying to)
Turn this off. Deduplication is going to do nothing in your usage scenario.

You're probably better off adjusting the recordsize to make the most of your storage. For example, for any media shares I use a recordsize of 1MB. This in the long run will save more space than deduplication ever could.

Code:
[root@tardis] ~# zfs get used,logicalused,recordsize tank/media/movies
NAME			   PROPERTY	 VALUE	SOURCE
tank/media/movies  used		 645G	 -
tank/media/movies  logicalused  650G	 -
tank/media/movies  recordsize   1M	   local
[root@tardis] ~# zfs get used,logicalused,recordsize tank/media/tv
NAME		   PROPERTY	 VALUE	SOURCE
tank/media/tv  used		 973G	 -
tank/media/tv  logicalused  981G	 -
tank/media/tv  recordsize   1M	   local
[root@tardis] ~#
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
A 4x 1G LACP may not be a good idea and you might want to try disabling it.

How full is the actual pool? Speeds will degrade as the pool nears being full, and with a single RAIDZ2 vdev and a fair bit of memory, you may be creating a scenario where the I/O subsystem is simply unable to keep up with how fast things are being pushed at it. More memory translates to larger transaction groups being committed, and a high fragmentation rate on a fullish pool translates to slow writes, and a large transaction group being written slowly often translates to file transfer protocol-level freakouts.

In ZFS you have to keep a fair bit of free space available in order to maintain good write speeds, and a 6TB usable-space pool could be filled pretty easily. You could be seeing impacts with even less than 3TB used, especially if you've been copying files around and creating lots of fragmentation.
 

mattg889

Cadet
Joined
Jan 2, 2018
Messages
7
The data is a mix of compressed and not compressed. As I said, before all these problems, the same data was on FreeNAS using about 600-700GB of data. now its using 920GB, I would expect it to be down between 600-700 again.

Everything is set the same as it was when FreeNAS was working perfectly before I updated it. Including deduplication.
I had deduplication on before and it was seemingly working fine. I wanted it on because I know I have duplicate data on FreeNAS, Lets just say I'm not that digitally organized.
I will try with dedup off.

I saw the record size option, but I haven't touched it yet. Default just says Inherit, What's the default record size? and what should be used for backups, documents, you said 1MB for media
 
Last edited by a moderator:

mattg889

Cadet
Joined
Jan 2, 2018
Messages
7
I don't have LACP set up, I have the aggregation set to "loadbalance" LACP did not work well with my switch.
The pool is about 1.6TB out of 5TB used.
that brings up another question, i know there is drive overhead, but I am only able to use 5GB out of the 8GB of disks. I thought I should have about 6GB of space with a raid-z2.
The drives are showing about 20MBps being transferred to them. The drives I know are capable of handling at least 80GBps. So I don't think there are any i/o bottlenecks with the drives...
 
Joined
Apr 9, 2015
Messages
1,258
Load balance is still a possible problem. Turn off EVERYTHING in the "Network" "Link Aggregation" section. Anything in there can cause issues with connectivity other than failover and when troubleshooting you should turn off anything that does anything fancy and see if the problems go away.

If you need more speed to the FreeNAS a 10G connection and a switch that will allow your devices to connect is a MUCH better idea. I ordered 4 SFP+ NIC's and Transceivers and two patch cables for about 150 so I can use 10G for my desktop to my FreeNAS. It's cheap enough now that by the time you spend extra on a switch that supports LinkAgg you may as well buy 10G.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
Load balance is still a possible problem. Turn off EVERYTHING in the "Network" "Link Aggregation" section. Anything in there can cause issues with connectivity other than failover and when troubleshooting you should turn off anything that does anything fancy and see if the problems go away.

Nicely put.

If you need more speed to the FreeNAS a 10G connection and a switch that will allow your devices to connect is a MUCH better idea. I ordered 4 SFP+ NIC's and Transceivers and two patch cables for about 150 so I can use 10G for my desktop to my FreeNAS. It's cheap enough now that by the time you spend extra on a switch that supports LinkAgg you may as well buy 10G.

The Dell 5524 has recently been on eBay for as little as $75, and someone is selling a truckful of 5524P's (PoE) at two-for-$150. Pick up a good quality 10G card and twinax for eighty bucks or a crappy old Mellanox ConnectX2 for twenty bucks. That 5524 plus the X520 is a Grinch-approved setup for less than $200.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
that brings up another question, i know there is drive overhead, but I am only able to use 5GB out of the 8GB of disks.

Presumably: TB not GB

Drive manufacturers sell drives by 10^2 figures (see the great TiB-vs-TB debate, also stuff like http://usatoday30.usatoday.com/tech/products/2007-11-03-2622074867_x.htm )

I thought I should have about 6GB of space with a raid-z2.

So for RAIDZ2, you lose at least two disks to parity, more if you're storing small files, and wind up with around 5.4TiB of pool space. Try out @Bidule0hm 's calculator here

https://jsfiddle.net/Biduleohm/paq5u7z5/1/embedded/result/

But on top of that, you really shouldn't fill a pool past 80%, so that works out to around 4TB of truly usable space.

The drives are showing about 20MBps being transferred to them. The drives I know are capable of handling at least 80GBps. So I don't think there are any i/o bottlenecks with the drives...

I'd love to see a drive capable of 80GBytes/sec. Will again presume you meant MBytes/sec instead of GBytes/sec.

Hard drives have two basic modes: sequential and random access. For sequential, which means you are reading or writing LBA's sequentially, hard drives are pretty fast, and even slower ones can do 80MBytes/sec. However, for random access, hard drives are limited by seek speeds, and even a pretty fast HDD is limited to perhaps 200 seeks per second, and for 512 byte LBA's, that works out to an exciting 100 KiloBytes/sec in a pretty bad case. Yes, that's KILO bytes.. 0.1 MBytes/sec.

So the question comes down to what is the pool actually being required to do. ZFS tends to create large contiguous writes, where that is possible, which is why you can write a million small files to a fresh pool at SSD-like speeds, even though you'd THINK there would need to be lots of seeks to update metadata. That assumption is simply incorrect, and it lays down a massive sequential write of all the data. Conversely, if your pool is highly fragmented, even large file data has to be broken up into smaller blocks in order to fit into free space fragments, and when this happens, your pool performance tanks. The reality is usually somewhere in between. Those of us who want zippy, responsive pools do certain things, including keeping lots of free space on the pool, and trying to avoid things that cause fragmentation, to get those properties.

I still think that there's also a likely issue with sizing of RAM vs pool size that could be playing into this. Bug 1531 has never been truly FIXED because it is a difficult issue to actually fix, and it isn't that hard to get even the latest ZFS to stall out if you do certain things to stun it with traffic. Reducing the transaction group size and window is still a smart thing to do, though it will lower the potential peak performance, you are more likely to get a more even and consistent experience by limiting the window of opportunity, so to speak.
 

mattg889

Cadet
Joined
Jan 2, 2018
Messages
7
Sorry for the delay, been busy.
I reinstalled without de-dup just so everyone knows.

jgreco, good catch, I did mean TB. and I did mean MBytes/sec. And I found the great Tib vs TB debate too.
And FreeNAS seems to be using my gigabit connections to their fullest extent and that is good enough for me right now.

Anyhow, I think I found some answers and it was not easy. I switched back to FreeNAS 9.10 and I was still having connections drop out. I am running FreeNAS 9.3 now and it is working good enough for now. But with switching back to FreeNAS 9.3 I think I found the problem. Some portion of my memory is bad in my server. My daily email usually has the following:
Code:
reenas.local kernel log messages:
> MCA: Bank 4, Status 0x8c1ac00000080813
> MCA: Bank 4, Status 0x8c30400000080813
> MCA: Bank 4, Status 0x8c3a400000080813
> MCA: Address 0xe0c7b0

-- End of security output --

It seems that the latest version of FreeNAS can't handle the memory issue. And it just errors/times out. While FreeNAS 9.3 sees that there is a memory problem and tolerates it. I never saw anything memory related in the logs with FreeNAS 11.
I would much prefer to be using the latest FreeNAS because 9.3 doesn't like my lsi controller (Warning: "Firmware version 20 does not match driver version 16 for /dev/mps0"), but again, it's working good enough for now.
I have moved memory sticks around and I will have to wait to see if the bad memory location moves to hopefully find the bad stick.

I am going to start a new thread for the iscsi issues I am having.
As far as the connection issues I was having, I think it is a combination of the newer freenas versions not working well with memory problems, combined with, well, the fact that I have a memory issue. That's what was causing it.
Freenas says all my memory is in use right now and I haven't had any memory errors pop up since I moved sticks around. So maybe sometime in the future I will try freenas 11 again.
 
Last edited:
Status
Not open for further replies.
Top