bad performance with mirrored SSDs as SLOG

[Thomymaster]

Hi

Yesterday i installed my mirrored SSDs (ADATA SX900 64GB) to enably "sync=always" in my ZFS dataset which is used to provide and ESXi datastore.
Before this, i could get approx. 100MB/s with CrystalDiskMark on a VM, but now i am stuck to a maximum of 25MB/s. The command "gstat" showed me the same. The SSDs are connected to the onboard S-ATA2 controller.

I have 16GB of RAM and 4 1.5TB drives in a striped-mirror config running the latest FreeNAS. Further i have enabled these kernel tune values:

vm.kmem_size 14336M
vfs.zfs.arc_max 12288M
vfs.zfs.arc_min 12288M
vfs.zfs.prefetch_disable 0
vfs.zfs.zil_disable 0
vfs.zfs.txg.timeout 5
vfs.zfs.vdev.max_pending 10
vfs.zfs.vdev.min_pending 4
vfs.zfs.write_limit_override 0

What causes this issue?

i created the mirror using

Code:

zpool add zpool1 log ada4 ada5

zdb shows ashift=9 for the 2 SSD devices. Could it be that i have to create a partition and align it to use 4K sectors?
If so, how do i do that?

Best,
Thomas

 

[Yatti420]

Need specs.. Do you have hardware that even requires slog or tunables? 95% of users I've seen don't have the hardware and majority shouldn't probably be using slogs/l2arcs or tuning of any kind.. I believe slogs are always ashift=12 4k?..

 

[cyberjock]

First, post the output of "zpool status". Put it in code blocks please. I'm not sure what you have is a mirrored slog, but rather a striped slog on 2 SSDs.

If you had wanted a mirror I think the appropriate command would have been:

# zpool add zpool1 log mirror ada4 ada5

Let's verify this with your zpool status though.

Secondly, elaborate a little. Was it 100MB/sec before you added the ZIL, then 25MB/sec after you added the ZIL or 25MB/sec after you added the ZIL and sync=enabled? It's best to change only 1 thing at a time to keep track of where things are going and what works and what doesn't. Also, how are you getting these numbers? Doing a file copy in the VM, dd, etc? How is your VM being used? iSCSI or NFS? I'm presuming iSCSI because you want sync=enabled. Details are key here.

I will say that with 16GB of RAM, expecting high speeds is going to require some work on your part. You'd be in a different ballpark in terms of 'ease of tuning' if you had 32GB of RAM.

Add to that what Yatti420 said. SLOGs aren't always good for users, and they can hurt you if you don't use them right. This is why my noobie guide says not to add them unless you are sure you need them. And then 99% of users that claim to need them don't need them at all.

You don't have to create a partition, I think the GUI will let you do mirrored slogs(and if it does you should have used it). Normally I just partition off like 8GB for the slog and leave the rest unallocated so that the SSD can do its thing. There's serious questions on whether TRIM works or not for slogs, if it matters or not, etc. For Intel SSDs unpartitioned space is used for self garbage collection. I can't make the same claim with 100% certainty for Adata. Based on what I've read about them from users in the forum in the last 48 hours they do things totally different from Intel.

 

[Thomymaster]

Hello

Sorry the CLI command i posted was wrong, i created a mirrored SLOG, i am sure :)

100MB/s was with sync=standard and 25MB/s was after adding the 2 SSDs and setting sync=always on the zVol (which is used via ISCSI for an ESXi datastore). The performance was measured with CrystalDisk info, but i can confirm that the speed was the same when transferring files (the VM is used as a fileserver).

Can you tell me the commands to partition the SSD to 8GB and use this partitions for SLOG?

I don't think its because of my 16Gb RAM, because when i don't have a SLOG vDev then the performance is like i said approx. 100MB/s write.

 

[cyberjock]

Hello

Sorry the CLI command i posted was wrong, i created a mirrored SLOG, i am sure :)

100MB/s was with sync=standard and 25MB/s was after adding the 2 SSDs and setting sync=always on the zVol (which is used via ISCSI for an ESXi datastore). The performance was measured with CrystalDisk info, but i can confirm that the speed was the same when transferring files (the VM is used as a fileserver).

Oh, zvols huh? The zvols don't perform as well as file based iSCSI. I'd recommend you switch to file based.

Can you tell me the commands to partition the SSD to 8GB and use this partitions for SLOG?

Ug, can you PM me tomorrow for this? I'm kind of in a rush today and I don't have it handy.

I don't think its because of my 16Gb RAM, because when i don't have a SLOG vDev then the performance is like i said approx. 100MB/s write.

That's not a good justification in my opinion. Normally sync writes will be immediately committed to your pool. This means the amount of data "flying" to the pool is going to be extremely small at any given time. If you add a ZIL the amount of data "flying" will be much more. The more data you have flying in RAM that needs to be written to the pool(notice I said pool and not the ZIL/SLOG) the less RAM you have for your ARC. Less RAM for your ARC means slower performance. And it doesn't take stealing much RAM to hurt performance for some people.

So literally, when you add ZILs, you are stressing your RAM and ARC. How much it is stressed depends on many factors. I typically tell people that want to run more than 1 or 2 VMs to go to 32GB of RAM minimum. I do 1 VM with 24GB of RAM, and it is horribly slow. I run it only for testing and I'd never even think about doing it in production. It's just much much too slow for that.

The same applies for L2ARCs. Using an L2ARC stresses your ARC, but for different reasons than the ZIL/SLOG does.

So yeah, sorry but I have to disagree with your assessment.

 

[HoneyBadger]

Can you tell me the commands to partition the SSD to 8GB and use this partitions for SLOG?

I believe those would be:

Code:

#create an 8GB partition on each disk, starting at 1MB
gpart add -t freebsd-zfs -l slog1 -s 8G -b 1M ada4
gpart add -t freebsd-zfs -l slog2 -s 8G -b 1M ada5
 
#add them as log mirror
zpool add zpool1 log mirror /dev/gpt/slog1 /dev/gpt/slog2

You might want to remove all partitions and TRIM both SSDs first though.

 

[Thomymaster]

ok and how to i disable TRIM?

 

[HoneyBadger]

ok and how to i disable TRIM?

Technically you want to enable TRIM, but you probably don't need to do that step; FreeNAS should have TRIM support in 9.2.0. Just make sure you delete all other partitions before creating the single 8GB one.

 

[Thomymaster]

OK thanks.

@Cyberjock
Why do you think that file-based extents work better than vDevs for ISCI?

 

[cyberjock]

Why? I have no clue. I just know that the developers ahve said that it is the case. As far as I know they don't know precisely why either. They(and I) believe that it has something to do with how blocks are handled in zvols, but we're not ZFS experts, so its hard to understand the magic smoke that ZFS operates with.

I also know that after reading that I changed to a file-based and my performance increased.

I also know that other people have switched to file based extents and their performance has also increased, sometimes by huge amounts.

It's merely observed behavior and not really understood. At least not yet. If you drop in IRC and ask the devs which is better, they'll tell you file based seems to be better. You'd be a hero if you could explain the difference. Check out this sentence from the FreeNAS manual...

In theory, a zvol and a file extent should have identical performance. In practice, a file extent outperforms in reads/writes but this is only noticeable at 10 GB Ethernet speeds or higher. For high performance, file extents are recommended at this time. Future changes to FreeBSD's zvol code will increase its performance.

Personally, I don't think it really matters why as much as the fact that it does. Don't get me wrong, I love understanding things and I'd love to know why. Weird obscure tidbits like this always interest me. But nobody seems to have the answer. :(

 

[cyberjock]

Also, I'd recommend you make your SLOGs using these command over what HoneyBadger provided. His will work, but I will always recommend people format their disks identically to how FreeNAS' GUI does it. It ensure future compatibility(which is always a good thing!)

gpart create -s GPT daXX
gpart add -t freebsd -a 4k -s 8G daXX (for 8G SLOG)
zpool add pool log daXXp1

If you have two disks, then you gpart 2 disks and then do the mirror. I highly recommend mirrors.

 

[HoneyBadger]

Also, I'd recommend you make your SLOGs using these command over what HoneyBadger provided. His will work, but I will always recommend people format their disks identically to how FreeNAS' GUI does it. It ensure future compatibility(which is always a good thing!)

gpart create -s GPT daXX
gpart add -t freebsd -a 4k -s 8G daXX (for 8G SLOG)
zpool add pool log daXXp1

If you have two disks, then you gpart 2 disks and then do the mirror. I highly recommend mirrors.

Thanks, I forgot that -a 4K will handle the start offset as well ...

Disclaimer, I've been managing Linux, BSD, and Solaris recently, and haven't been sleeping nearly enough, the following may be off my rocker;

... But I would still suggest using -l "label" instead of a hard mapping to daXpY for the gpart/zpool add.

If for whatever reason you end up switching SATA/SAS ports on your motherboard/HBA and they get reordered differently - for example, your L2ARC was da4p1, your SLOG was da5p1, you have an "incident" and they get their plugs/trays swapped. You'd really rather FreeNAS be able to sort out where they landed on its own rather than mounting your SLOG as your L2ARC and your L2ARC as your SLOG.

 

[cyberjock]

FreeNAS works by gptid. When you reboot it'll self-correct to the gpt-id and you'll be all better.

 

[Thomymaster]

So these commands are OK?

Code:

gpart create -s GPT daXX
gpart add -t freebsd -a 4k -s 8G daXX (for 8G SLOG)
zpool add pool log daXXp1

 

[HoneyBadger]

FreeNAS works by gptid. When you reboot it'll self-correct to the gpt-id and you'll be all better.

Too many POSIX OSes on the brain. Thanks for clarification.

So these commands are OK?

Code:

gpart create -s GPT daXX
gpart add -t freebsd -a 4k -s 8G daXX (for 8G SLOG)
zpool add pool log daXXp1

That would create a single SLOG device, you'll want to do the first two gpart commands for each disk and then add them both as a mirror. Again, might be my Solarisms coming out but you could add one and then the other later - but if you've got both ready, why not add both at once?

Code:

gpart create -s GPT ada4
gpart create -s GPT ada5
gpart add -t freebsd -a 4k -s 8G ada4
gpart add -t freebsd -a 4k -s 8G ada5
zpool add zpool1 log mirror ada4p1 ada5p1

 

[Thomymaster]

I tried adding the 2 partitions for a mirrored SLOG, but still max. 25MB/s :(

 

[cyberjock]

Have you ever considered that your system is not adequately balanced to support high speeds? I mean, 16GB of RAM is not a lot of RAM. And as I said above you'd be much better off with 32GB of RAM before you do an SLOG. To be frank, I think you are practicing an exercise in futility until you up your RAM.

 

[HoneyBadger]

I tried adding the 2 partitions for a mirrored SLOG, but still max. 25MB/s :(

Can you post some more detail in the benchmarks? Screenshot perhaps?

25MB/s is phenomenally good if it's random 4K write at low queue depth, but abysmal if it's sequential read.

 

[Thomymaster]

I added another 4GB of RAM so that i have 20Gb in total but the speed didn't improve.

So the traffic created by the SLOG is sequential 4K write or, so the speed should be more fast??

 

[cyberjock]

You don't seem to understand what the SLOG does. The SLOG doesn't change the writes at all. It's nothing more than a temporary non-volatile storage for information that needs to be committed to the pool in the future.

It does NOTHING to change writes to sequential(whatever you think that means.... ZFS has no definition for sequential).

And I have no clue why you thought that adding 4GB of RAM was going to make a difference. Remember when I said:

Have you ever considered that your system is not adequately balanced to support high speeds? I mean, 16GB of RAM is not a lot of RAM. And as I said above you'd be much better off with 32GB of RAM before you do an SLOG. To be frank, I think you are practicing an exercise in futility until you up your RAM.

Notice that I said 32GB, not 20GB. There's a big difference for ZFS in terms of 20GB of RAM versus 32GB of RAM.