Rapidly outgrown my first NAS

[leonroy]

So, feeling a bit stupid.... but here goes:

I built 3 ZFS servers with the following spec.

Hardware:

• Motherboard: Supermicro X9SCL+-F
• CPU: Intel Xeon E3-1220L V2
• RAM: 16GB ECC
• HBA: 2x IBM M1015 in IT mode
• Case: 16 bay SATA/SAS hotswap chassis (Supermicro SC836)
• HD (Data): 16x 1TB disks in a single RAID-Z3 pool
• SLOG: Intel S3700 100GB SSD (8GB partition)
• L2ARC: 256GB Samsung 840 Pro (128GB partition)

We have been using these three machines as primary, secondary and backup servers respectively for all our office storage needs.

The past two years have seen a large growth in our ESXi usage and we've gone from 8 VMs to 40+.

Latency however is ridiculous (100ms on virtual disk I/O) and we've hit the limits with number of VMs we've been able to load per box.

First thing we've done is remove the L2ARC. To be honest, now that we understand ZFS a little better we're appreciating that everything needs to be sized carefully and with 16GB of RAM an L2ARC is useless.

We're looking at improving performance, which either entails upgrading to 32GB RAM per box (the limit for the E3 platform) or throwing out the RAM, CPU and MB on the primary and secondary servers and upgrading to an E5 CPU and MB which will allow us to fit 96GB RAM per box (with 16GB RDIMMs).

Any thoughts gratefully appreciated!

 

[anodos]

I suppose it depends on your budget. If you don't have much money to throw at the problem right now, you can upgrade RAM (get something that is on the HCL for the current mobo and the on that you might upgrade to). See if it helps, then look into upgrading CPU/MB.

How much of your zpool's storage capacity are you using? Maybe look at changing your zpool configuration from 16 disk raidz3 to raid-10 (or striping 2 8-disk raidz2 vdevs).

 

[leonroy]

Budget is roughly £1k to upgrade one of the boxes. I'm looking at 32GB RAM for £250 or just over £1k for an E5 board, CPU and 96GB ECC RAM. Just wondering whether the 32GB will suffice for our load, or if ZIL speed is also holding us back.

 

[anodos]

Okay, so you are considering the following:
Option 1 - purchase 32GB RAM for your current server with an E3 Board
Option 2 - purchase 96GB RAM, E5 board, and cpu

Option 1 is a subset of Option 2. Go with Option 1 and then do testing to see if you need to purchase an additional 64GB RAM, board, CPU.
16GB RAM is an obvious candidate for a bottleneck.

Your storage pool may also be a bottleneck with your current workflow. You should probably experiment with increasing the number of vdevs in your zpool. Probably best to compare performance with a zpool of 8 mirrors (assuming that it won't be more than 80% full). If storage space will be an issue, look into other ways of creating more vdevs (two 8-disk raidz2 vdevs is an obvious candidate).

 

[Nick2253]

One concern I have is that you are not mirroring your SLOG. This means your SLOG becomes a single point of failure for you data. Something you'll want to correct moving forward.

Have you verified that you are not limited by your network bandwidth? Perhaps an upgrade to a 10GB network would be more prudent.

If anything, get more RAM. RAM is always a good thing :)

 

[gpsguy]

You might want to follow this thread - Growing pains with ESXi.

As anaodos said, striped mirrors will probably give you better performance. And, you don't want to use more than 60% of your storage.

 

[leonroy]

You might want to follow this thread - Growing pains with ESXi.


As anaodos said, striped mirrors will probably give you better performance. And, you don't want to use more than 60% of your storage.

Thanks, useful thread! Can't believe how much ZFS performance tanks with only 50% space usage!

One concern I have is that you are not mirroring your SLOG. This means your SLOG becomes a single point of failure for you data. Something you'll want to correct moving forward.

Besides mission critical 100% uptime stuff a striped SLOG is pretty unnecessary since ZFS v28. It was only versions prior which would trash the pool when the SLOG died. Granted a mirror will avoid performance tanking if using sync writes but I don't think there's any need for it outside of super mission critical stuff.

Have you verified that you are not limited by your network bandwidth? Perhaps an upgrade to a 10GB network would be more prudent.

If anything, get more RAM. RAM is always a good thing :)

Not even hitting 300Mb on the wire and we're using LACP on each server so I don't think we'd need 10GbE anytime soon. That said it'd make my life much easier when putting a datastore into maintenance and vMotioning everything off it!

Indeed, it certainly looks like RAM is something I need more of! :)

Okay, so you are considering the following:
Option 1 is a subset of Option 2. Go with Option 1 and then do testing to see if you need to purchase an additional 64GB RAM, board, CPU.

Sadly the 32GB of RAM would be UDIMMs whereas the E5 platform to go beyond the 64GB limit will only accomodate RDIMMs so spending the money to buy 4x8GB UDIMMs for the existing server would be a dead end. If however it solves my problem then it's worthwhile. Trying to see if anyone here has any input on whether 32GB is enough.

Your storage pool may also be a bottleneck with your current workflow. You should probably experiment with increasing the number of vdevs in your zpool. Probably best to compare performance with a zpool of 8 mirrors (assuming that it won't be more than 80% full). If storage space will be an issue, look into other ways of creating more vdevs (two 8-disk raidz2 vdevs is an obvious candidate).

Good point, I'll look at striping at least two vdevs, although if anyone here has any rough quantifiable figures that'd save me the effort!

Just for reference, what's the consensus on benchmarking tools here?

I know how to run the dd and iozone benchmarks but wondering if setting up an iometer test is a better gauge of performance.

 

[Ericloewe]

If you believe you might keep growing quickly, the more comprehensive upgrade might be in order, as it wouldn't make sense to spend the extra cash only to have to do the real upgrade soon after.

 

[cyberjock]

I'd go with the E5 system. You're very limited in your expandability with SLOGs/ZILs and L2ARCs until you get lots of RAM. And once you get lots of RAM and can actually use both of those, your performance will just explode before your very eyes. Personally, I'd go with 128GB of RAM. That seems to be the sweet spot where you can add about 500GB of L2ARC and a ZIL/SLOG and see some damn amazing performance.

Besides mission critical 100% uptime stuff a striped SLOG is pretty unnecessary since ZFS v28. It was only versions prior which would trash the pool when the SLOG died. Granted a mirror will avoid performance tanking if using sync writes but I don't think there's any need for it outside of super mission critical stuff.

Yeah, incorrect. Your SLOG is a single point of failure in the event that you have data that isn't committed to the pool but is in the SLOG on a loss of power, kernel panic, etc.

With v15 it was a requirement for you to have mirrors because if you lost the SLOG you lost the entire pool. There was no recovery.

After that bug was fixed you could run with a single disk, but any data in your SLOG had only 1 copy. So you have 2 copies of all of the data in your pool, but one copy of the data in the SLOG. Don't you see the problem?

So don't be fooled. If you want to do it, go for it. But I've already seen plenty of people that ended up with unbootable VMs because of incomplete transactions lost because of a bad SLOG. You learn this the hard way once and you'll realize that buying the second SLOG was worth the man-hours you paid those technicians to handle the restore from backups.

 

[leonroy]

I'd go with the E5 system. You're very limited in your expandability with SLOGs/ZILs and L2ARCs until you get lots of RAM. And once you get lots of RAM and can actually use both of those, your performance will just explode before your very eyes. Personally, I'd go with 128GB of RAM. That seems to be the sweet spot where you can add about 500GB of L2ARC and a ZIL/SLOG and see some damn amazing performance.

Fortunately we have 8x 16GB sticks of the memory I need gathering dust on our parts shelf, so that's doable. I think I'll look at upgrading only one box and running all 40+ VMs of it. I assume a system with that sort of spec can handle that?

So you have 2 copies of all of the data in your pool, but one copy of the data in the SLOG. Don't you see the problem?

*sigh* guess I do see the problem! The mirror must slow the SLOG down a little I'd imagine?

Here's the SLOG SSD I have:
http://ark.intel.com/products/71913...-100GB-2_5in-SATA-6Gbs-25nm-MLC?q=s3700 100gb

Is that suitable performance wise and moreover would an 8GB partition on that still be sufficient with 128GB of RAM and 32TB of storage?

Thanks!

 

[ser_rhaegar]

You want double the txg size for your slog. Txg is 1/8 of your RAM, so your slog should be 1/4 or 32GB.

 

[cyberjock]

Fortunately we have 8x 16GB sticks of the memory I need gathering dust on our parts shelf, so that's doable. I think I'll look at upgrading only one box and running all 40+ VMs of it. I assume a system with that sort of spec can handle that?

There's no way to know. Like I have said before around here, you use the system and when it can't handle the load you figure out what the limiting factor is and do what is necessary to make it no longer the limiting factor. Also your VMs could be very busy all day long and only be able to do 10 VMs with given hardware while I can run 100 VMs since mine are all idle. I have helped people do 40 VMs with 128GB of RAM.

Is that suitable performance wise and moreover would an 8GB partition on that still be sufficient with 128GB of RAM and 32TB of storage?

Again, there's no way to know. That depends on your usage. But if you assume 1Gb throughput to your server and you saturate it for 6 seconds(which is the default time for transaction holding), it's impossible for you to have more than 1Gb * 6 seconds, or approximately 798MB. So any 8GB is massive overkill, but for an SSD of that size I'd consider that appropriate.

Mirrors do slow down the SLOG slightly, but don't be fooled. SLOGs provide such an amazing increase in performance when used in proper situations that slight penalty is mute. It's like trying to argue that you pissed on the ocean and therefore the level must be higher. Yes, you are technically correct, but who is going to actually be upset about it.

 

[leonroy]

Alright, thanks for the replies guys.

I plan on doing the following:

1. Upgrade the secondary ZFS server to 32GB and all else the same see the change in throughput.
3. Upgrade the primary box to 128GB, X9SRL-F and E5 1620V2
3a. Add a second SLOG (mirrored) and second L2ARC SSD (already a 256GB SSD in there so will stripe the L2ARC to give 512GB)

ser_rhaeger love your build, based our new spec around the motherboard you chose. Curious though, why did you go with the more expensive E5-2620V2 for your build over the E5-16XX range? It only seems to offer two more cores than the 1620V2 and at a much lower clock speed...?

 

[ser_rhaegar]

ser_rhaeger love your build, based our new spec around the motherboard you chose. Curious though, why did you go with the more expensive E5-2620V2 for your build over the E5-16XX range? It only seems to offer two more cores than the 1620V2 and at a much lower clock speed...?

It will become an ESXi box. Not that I cap the CPU anyways with the lower clock. Was testing ram options, swapped from 24GB to 136GB and had no speed bumps (gigabit links not 10Gb).

If I needed more than 32GB I was going to get a 16XX processor for the second X9SRL board I bought (Microcenter sale for 130 each). And load up the RAM.

However I am instead going to use the X9SRL boards for ESXi and move my E3 v3 motherboard with 32GB of ram into a supermicro case and use that for FreeNAS. Then split my registered ram between the ESXi boxes.

Then sell the Microserver, Cisco blades and leftover parts as I will be done with testing.