Terrible Disk Read/Write I/O Speeds

[kr4m17]

I am new to FreeNAS but have been using Linux for years. I just upgraded my RAID 50 Linux server to FreeNAS (No hardware changes at all). I am now experiencing terrible hard disk I/O problems. At first I thought it was a network issue, but as I started to troubleshoot I realized it was drive related.

To start, I am running a Core 2 Q6600, 4 GB Memory (8 GB in the mail and will be here any day), Six 1G Ethernet Cards. My RAID controller is a 3Ware 9650SE-12ML configured as RAID 50 with 12 3TB disks.

Under Linux I was getting approximately 2.5 Gbit/sec I/O from the array, now I am getting 80 Mbit/second with peak performance of 110 Mbit/sec.

I eliminated networking as an issue by running iperf. See results:

[jsimon@storage01 /mnt/raid0]$ iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 64.0 KByte (default)
------------------------------------------------------------
[ 4] local 10.0.20.105 port 5001 connected with 10.0.20.100 port 46752
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-180.0 sec 19.6 GBytes 937 Mbits/sec​

Next I checked write speed. See results:

[jsimon@storage01 /mnt/raid0]$ time sh -c "dd if=/dev/zero of=bigfile bs=8k count=1000000 && sync"
1000000+0 records in
1000000+0 records out
8192000000 bytes transferred in 754.550775 secs (10856791 bytes/sec)

real 12m36.513s
user 0m0.383s
sys 0m22.595s​

This is definitely the problem. To keep everyone from having to do the math this is 82.830741882 Mbit/sec local read write.

If I write a smaller file that can be cached the issue seems to disappear. See below:

[jsimon@storage01 /mnt/raid0]$ time sh -c "dd if=/dev/zero of=bigfile bs=8k count=100 && sync"
100+0 records in
100+0 records out
819200 bytes transferred in 0.002206 secs (371376333 bytes/sec)

real 0m0.121s
user 0m0.000s
sys 0m0.011s​

Again to save you the trouble, this is 2833.37656403 Mbit/sec. Obviously I am okay with this I/O speed considering my RAID adapter is a 3Gbit/sec adapter.

There were zero configuration changes to the actual RAID configuration. I did change the file system to ZFS though. I originally had dedup enabled but I thought maybe that was contributing to the problem, so I disabled it.

I do think my swap partition may be a little small, but I do not remember seeing an option to manually partition the disk during install. See results:

[jsimon@storage01 /]$ swapinfo
Device 1K-blocks Used Avail Capacity
/dev/da0p1.eli 2097152 22588 2074564 1%​

None of the disks are failing and they have passed any checks I have run on them.

I am trying to think of possible reasons for this to be happening and I thought maybe 3Ware drivers/utilities could be the culprit. I have not been able to find 3Ware software for this RAID adapter for FreeBSD 8.3.

Any help or input would be appreciated. I know a lot of people are having similar issues, but i have run through the tests that I have read in other threads and can't seem to find a fix for this.

Thank you very much for your help!

Jeff

 

[jgreco]

4GB of memory is half the recommended amount for using ZFS on FreeNAS.

Using a hardware RAID controller is a bad idea, and makes debugging these sorts of problems a lot harder.

 

[kr4m17]

While I agree with you 100% about the RAM (even though this hardware setup performed perfectly under Linux). I think it's a bit ignorant to believe using an enterprise level hardware RAID controller would be a bad decision. It is certainly simple enough to troubleshoot these problems in Linux, I am just not as familiar with BSD as I am Linux.

Please feel free to offer advice to help troubleshoot the problem if you have any.

 

[jgreco]

I think it's a bit ignorant to believe using an enterprise level hardware RAID controller would be a bad decision.

(best Doctor Evil voice) Riiiiiiight.

Seriously, didn't I just finish covering this topic? Or this topic?

But also seriously, when the person with 2 posts to the forum suggests that a senior member is "a bit ignorant to believe" something, that shows a lot of testicular fortitude. I believe what I do because much time and experience lead to some small amount of expertise. I share what I do because I'd like to save you the aggravation. But quite frankly I have more entertaining things to do, so if you'd prefer to bask in your own RAID-enhanced enlightenment, then by all means.

Where's my coffee?

 

[kr4m17]

While your last response with posts to actual information was very helpful, it still doesn't address performance issue of my setup. While you make a great case that there may not be a benefit to hardware RAID in conjunction with properly configured ZFS it still doesn't explain a deficit. If RAM is the biggest factor than the fact that I am already upgrading it will help the problem.

Just so we are clear, your first comment was short and not supported with any information. It was apparently not ignorance as you clearly well informed about the topic but if you make a short unexplained comment to someone and you probably shouldn't be surprised when you come off as poorly informed.

So thank you for the information you posted in your last post, it was very concise and informative, but it still doesn't explain or give me insight into my problem.

Would you be willing to discuss what you think the source of my problem may be? I am sure you would certainly be able to lend a hand in troubleshooting.

By the way, I apologize if I insulted you, it was not my intent. I was simply commenting on your unexplained commentary.

Thanks,

Jeff

 

[jgreco]

your first comment was short

You're not always going to get lengthy responses in this age of smartphones, and the initial message led me to believe you might be able to figure this all out if you were given a gentle shove in the right direction.

Re: Deficit? Quite possibly. Completely plausible even. Read linked articles. Use HBA (if possible) or at least not hardware RAID in addition to ZFS. Or use hardware RAID with UFS, a match nearly made in heaven.

 

[kr4m17]

Well RAM upgrade is my first move... Hopefully it's that simple. After that, there isn't a way to utilize the benefits of a hardware RAID controller and ZFS together? I feel like there has to be some productive way to utilize both feature sets. If not, I can revert to UFS I suppose.

Just out of curiosity, can you explain why the write speed was acceptable with a small file if ZFS isn't taking advantage of the hardware RAID caching? I would think it would either work or not work if that is in fact the case.

 

[cyberjock]

So would you ever recommend you run a RAID array inside of a RAID array? That's exactly what you are doing if you are trying to run ZFS on a hardware RAID.

I'll tell you that I have an enterprise class RAID controller with 4GB of RAM (and formerly a BBU). And the comments about not running ZFS on a true RAID controller are 100% serious and not a joke at all. Even with JBOD on my controller, if I use the on-card read ahead and write cache I can barely get 200MB/sec from my 18 drive array. Disable it and it instantly goes over 1GB/sec.

So yes, ditch the RAID controller or disable all the read and write cache features it has. They compete(not complement) ZFS' own read and write optimizing systems and kill your performance.

I'm betting the sole reason why the small write gave you a good speed is because you didn't write enough data to your drives to cause the RAID controller's read/write cache's to clash with ZFS'. ZFS optimizes its reads and writes which are butchered all to hell if your RAID controller caches a write for "later" when ZFS is then trying to do its own read. When ZFS writes to the disks it had better write to the disks and not some RAM cache on your controller. Same for reads. Don't do the standard Windows-ism and think that more hardware must be better. It's so far from the truth its disgusting.

Additionally, ZFS' recommended RAM needs are 6GB of RAM + 1GB for each TB of disk space. Considering you have 12x3TB drives you should be looking for a system with around 42GB of system RAM. You may be able to get by with less. I can tell you that my 30TB array had absolutely horrible performance with less than 20GB of system RAM. Now that I've added 12TB of additional storage I've found that I need over 32GB of RAM. I know because I had performance problems that were instantly solved with an upgrade. This also means your Q6600 is NOT suitable for ZFS for your application, so you are looking at more serious issues and solutions than just "more RAM" if you want to use ZFS. You can use your hardware RAID with UFS with your system though. But more RAM won't help your UFS performance.

 

[jgreco]

I'm betting the sole reason why the small write gave you a good speed is because you didn't write enough data to your drives to cause the RAID controller's read/write cache's to clash with ZFS'. ZFS optimizes its reads and writes which are butchered all to hell if your RAID controller caches a write for "later"

I'm going to emphasize this. For writes, ZFS may try to write a staggering amount of data as a transaction group, which defaults to 1/8th your system memory in most cases. If you happened to have a 32GB RAM system, that translates to a 4GB transaction group. Anything smaller is soaked up by the ZFS write cache and will get written out as a single transaction group.

In the case of an HBA, basically that 4GB of transaction group data gets split up amongst the drives of the pool and written out in parallel to a bunch of drives. If you have a 24 drive array that writes at 150MB/sec, then your array is 100% busy for a second or two.

But in the case of a RAID controller, even if you have 4GB of cache on it, it is unlikely that the RAID controller is making it all available as write cache, and basically you're just flooding the "hardware" RAID's cache with data at full speed.

On the flip side, under-resourcing your system in terms of memory makes other bad things happen. ZFS is kind of annoyingly complicated to learn and successfully predict.