ESXi 5.5 ISCSI FreeNAS 9.2.0 slow random write speed (read is ok)

[Thomymaster]

Hi everybody


I have the problem, that with my ESXi 5.5. (with 5.1U1 the problem is the same) setup (ISCSI storage via FreeNAS) the random write speed is very slow, see attached screenshot (Bench1).

The unit consists of 4 2TB HDDs together with a SSD as L2ARC, 8GB RAM and a XEON E5345.

The ESXi is attached via 2 different gigabit ethernet links, however enabling/disableing MPIO doesn't change anything.

When running 8.3.1, i had no seen this behaviour. However i had to enable the "vfs.zfs.no_write_throttle" tuneable, and this only took effect when the hardware had 4 CPU cores (see the 2 additional screenshots).

Does anybody of you know what causes this behaviour in 9.2.0?


Cheers
Thomy

 

[dlavigne]

When running 8.3.1, i had no seen this behaviour. However i had to enable the "vfs.zfs.no_write_throttle" tuneable, and this only took effect when the hardware had 4 CPU cores (see the 2 additional screenshots). Does anybody of you know what causes this behaviour in 9.2.0

I don't know the solution but I do know that that tunable is now deprecated. From http://doc.freenas.org/index.php/Tunables:

The ZFS version used in 9.2.0 deprecates the following tunables:
vfs.zfs.write_limit_override
vfs.zfs.write_limit_inflated
vfs.zfs.write_limit_max
vfs.zfs.write_limit_min
vfs.zfs.write_limit_shift
vfs.zfs.no_write_throttle

 

[jgreco]

8GB is far too little RAM to support L2ARC.

write_limit_shift got deprecated? Damn! I haven't looked at 9.2.0's code so I can only pray that txg sizing has somehow been repaired...

 

[Thomymaster]

Yes, but even if i have not enough RAM, only the read performance should be affected and it is now the write performance which suffers.

 

[jgreco]

I agree, but I felt the observation was needed. Write speeds have a variety of factors and I haven't played with 9.2.0 anyways.

 

[cyberjock]

WOW.. 8GB of RAM and an L2ARC. Impressive there. NOT!

You should remove that L2ARC as performance will probably improve slightly. Then, go and get yourself a system with at least 64GB of RAM. Then(and only then) try to use an L2ARC again.

 

[Thomymaster]

Its about the write performance NOT the read performance!!

 

[Thomymaster]

But in any case i can try, what is the CLI command i need to use to remove the SSD (ada4) from the zpool? And the command to add it againg :)

 

[cyberjock]

Your point? Here's mine:

Using L2ARC steals RAM from the ARC for the index. This hurts performance overall. In fact, if you read around, you shouldn't really even be trying to use an L2ARC of 60GB until you hit something like 55GB oF RAM if you do the appropriate math for best case scenario for how an L2ARC works.

When people just start blindly adding L2ARCs or ZILs without know how this stuff works they end up with poor performing, sometimes unreliable, sometimes data eating pools. You're here complaining about performance.

Your #1 solution per the manual, per experience on this forum, and per my personal experience(and many other admins here, but I won't speak for them) is to add more RAM. You're shooting yourself in the foot by taking what little RAM you have and using it to index your L2ARC. RAM does two things very well...it increases read performance of your pool... and it increases write performance of your pool.

In fact, I can quote the manual for you:

ZFS uses a RAM cache to reduce read latency. If an SSD is dedicated as a cache device, it is known as an L2ARC and ZFS uses it to store more reads which can increase random read performance. However, adding a cache device will not improve a system with too little RAM and will actually decrease performance as ZFS uses RAM to track the contents of L2ARC. RAM is always faster than disks, so always add as much RAM as possible before determining if the system would benefit from a L2ARC device.

 

[titan_rw]

Its about the write performance NOT the read performance!!

And where do writes get cached? Ohh yea, RAM.

So have the bare minimum amount of ram combined with a ram stealing l2arc isn't doing you any favors.

 

[jgreco]

And where do writes get cached? Ohh yea, RAM.

So have the bare minimum amount of ram combined with a ram stealing l2arc isn't doing you any favors.

The ARC and the transaction group write buffers are separate as far as I recall. Not debating that it is a bad setup though :)

 

[titan_rw]

The ARC and the transaction group write buffers are separate as far as I recall. Not debating that it is a bad setup though :)

I didn't mean to imply they were the same, only that with the severe pressure the arc would be under, there might be a less than ideal amount of memory for txg's. Unless the 'txg' buffer takes precedence over arc, but that would only stress the arc even more on such a ram starved machine. I wouldn't be surprised if a severely stressed arc could also cause write issues, as the arc also is responsible for caching metadata and such that is probably required for dumping txg's to disk.

 

[jgreco]

Need a code refresher I don't have time for, but I think the space for the txg buffers doesn't really compete. Definitely pressure on the ARC is bad overall though.

 

[cyberjock]

They are separate, but they do somewhat work together to kill performance on starved machines. It's like "the perfect storm" when your ARC is pressured.

 

[Thomymaster]

Hi

i removed the l2arc from my zpool (and restarted), but the write performance is not going up. Using "arcstat.py" i can see that the adaptive size for the arc is still the same.

However when i read/write data on my esxi, i can see that the iscsi daemon on the FreeNas is at 25% CPU usage:

last pid: 4574; load averages: 0.27, 0.40, 0.40 up 0+00:28:23 01:07:27
28 processes: 1 running, 27 sleeping
CPU: 3.9% user, 0.0% nice, 4.5% system, 0.4% interrupt, 91.2% idle
Mem: 133M Active, 71M Inact, 6138M Wired, 76M Buf, 1565M Free
ARC: 5639M Total, 1334M MFU, 4277M MRU, 272K Anon, 22M Header, 5902K Other
Swap: 8192M Total, 8192M Free

PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND
2246 root 7 20 0 61848K 19768K uwait 2 4:58 22.46% istgt

i have a quad-core cpu, does this mean that (because the iscsi daemon isn't multi-threaded) it uses one of the 4 cores fully? Or am i mistaken?

 

[cyberjock]

Maybe. It could be the iscsi daemon is using up one core while trying to get stuff to the zpool and the zpool is actually holding up progress.

With 8GB of RAM, there's virtually nothing you're going to do to impress me with performance. You need something with a lot more RAM... say 32GB is a good start. Not to mention your CPU is quite old, and FSBs don't exactly make things go speedy fast.

 

[ZFS Noob]

N00b checking in.

I've recently been disappointed with the default NFS settings with XenServer (I'm unique - yay!), so I've been doing some reading about issues that might pop up if I just go iSCSI instead...

Have you looked at fragmentation as an issue here? If you've been running this a while, and the volume is heavily fragmented, might that contribute to the write slowness you're seeing (regardless of TXG sizing)?

Have you considered moving all your VMs from that iSCSI share to another then back? If it's fragmentation, this might resolve the issue.

(Note: not a VMWare or FreeNAS expert...)

 

[jgreco]

Yes. The answer is to keep at least 50% of the pool space free to reduce fragmentation effects, or less than 10% full to largely eliminate fragmentation performance issues even under severe conditions.

 

[Thomymaster]

Hi

I have more than 10% free to reduce fragmentation. I upgraded to 12GB RAM which didn't change anything (the benchmark looked exactly the same).
The ESXi datastore i tested with was based on a ZFS dataset.

Now i created a Zvol (with the default cluster size of 8KB) and made another datastore.
When i test on this, i get a quite better result, see screenshot attached.

Is there any explanation for this?

 

[jgreco]

I have more than 10% free to reduce fragmentation.

I read this as "My pool may be as much as 89% full." Which is way too full. You really need at least 50% free space in the pool, quite possibly more, to reduce fragmentation effects.

Basically there seem to be a lot of things wrong with what you're doing. I don't really have a lot of time to guess at which thing is going most wrong for you. For VM storage, the recipe I've been looking at is documented here and elsewhere.