Bad disk performance after upgrade.

[Anders7]

Hello,

We recently upgraded our Freenas server from version 8.0.2 to version 8.3.0, 32-bit. After performing the upgrade we upgraded the zfspools from version v15 to v28.

The system is a single Xeon 3.40Ghz with 2GB of ram, not sure about the details since i only recently took over this remote server but i think it's an old HP proliant with 4 scsi 320 disks, probably running HW raid 10, not ideal for ZFS i know.

All that set aside, after the upgrade we noticed the write performance on the zfs partitions to be very poor, getting a steady 8mb/s with no peaks up or down when benchmarking locally. It´s not a netowrk problem, we are on a fibre connection and have a steady 90mbs to the server.

Before the upgrade the server was acting "normaly", don't have any benchmarks but a previous 3h backup job is now taking over 11h.

Thankful for any ideas before or clues before i take a deep-dive, i know the server ain´t exactly a powerhouse when it comes to RAM and ZFS, but like stated, it was running much better before the 8.3 upgrade.

Cheers

 

[cyberjock]

If you look in the manual it says:

The best way to get the most out of your FreeNAS® system is to install as much RAM as possible. If your RAM is limited, consider using UFS until you can afford better hardware. ZFS typically requires a minimum of 8 GB of RAM in order to provide good performance. The more RAM, the better the performance, and the FreeNAS® Forums provide anecdotal evidence from users on how much performance is gained by adding more RAM. For systems with large disk capacity (greater than 6 TB), a general rule of thumb is 1 GB of RAM for every 1TB of storage. This post describes how RAM is used by ZFS.

NOTE: by default, ZFS disables pre-fetching (caching) for systems containing less than 4 GB of usable RAM. Not using pre-fetching can really slow down performance. 4 GB of usable RAM is not the same thing as 4 GB of installed RAM as the operating system resides in RAM. This means that the practical pre-fetching threshold is 6 GB, or 8 GB of installed RAM. You can still use ZFS with less RAM, but performance will be effected.

I'm not sure who exactly built the system. I get the impression either you didn't, you were forced to without the proper hardware, or you did it and then realized the error of your ways.

In any case, you are really in a bind since you upgraded the zpool version. My first recommendation would be MORE RAM. I didn't know you could even boot up and not have a kernel panic with 2GB of RAM with ZFS. I know lots of people that had performance and stability issues with just 3GB.

As the ZFS versions increase there is some increased load on the system due to the added features. I'd start with upgrading your RAM to at least 4GB. If you can't upgrade your RAM(your system is pretty old to have scsi 320) I'd look at just replacing the whole system and migrating your data to the new server.

Since your drives are in a RAID you can't even do a SMART check to see if there are problems. You could try doing a scrub, but if you are having problems it could make the system unable to serve files and possibly crash.

I'm really not sure who had it in their head that using just 2GB would be "good". I started playing with FreeNAS just before 8.0.4 came out and I believe the manual said to have a minimum of 6GB for best performance.

 

[Anders7]

Thanks for the answer.

Yeah i didn´t setup this server it´s been in use for quite some time as a backup, got dumped on me after the upgrade / performance failed :) . I got plenty of Linux and storage experience but not much at all on BSD and ZFS so having to rack my brain a little about how it works.

The server is remote without any iKVM / ipmi interface so i´m currently a bit in the dark. The way i see it my problem is most likely :

1. RAM
2. Failed BBU on the RAID turning my Write cache off.
3. Degraded RAID
4. ZFS running on a RAID setup, not sure how much the impact of this is rather then running "raw" disks?
5. Configuration error or ZFS tuning.

So, the only thing i can test remotely for now is the configuration and tuning of Freenas / ZFS.

The main reason I think there might be something up with my configuration / tuning is the fact that it was working so much better prior to the 8.3 upgrade. I read on the forums that v28 can suffer about 15% performance decrease compared to v15 on Freenas, not sure this is a fact though, plus this box suffers alot more then 15% performance penalty after the upgrade.

Apart from that though, never had a look at freenas before, nice bit of software.

 

[cyberjock]

Actually:

2. ZFS does some cool tricks to optimize writes into sequential writes even if you might not expect them to be. I've found that the RAID write cache does not significantly increase performance. Additionally, disabling it would definitely NOT give you 8MB/sec.
3. ZFS is constantly checking against itself, so I'm not sure if that's a problem either. But you may have a drive that is starting to fail and having problems that you can't see because you are using a hardware RAID controller.
4. ZFS can fix alot of problems with direct disk access. Without it some of the reliability and performance optimizations won't work as well.
5. Possibly. But before you start messing with the configuration I'd try more RAM first.

I've never heard of the 15% thing. Both of my systems were completely unaffected. The bottleneck is the hard disks. Depending on how loaded your CPU is I think that the consequences of upgrading could be anywhere from 0% to severe. Not to mention that any kind of performance enhancements with v28 likely will use more RAM. Something you just don't have much of.

Since you are new to FreeNAS, I'd go with RAM first. That may not give you back full speed performance, but there's no reason to be choking your system like that. Do you have good backups? If not, you should start considering it in case things go bad.