24 bays, 2 pools

[MtK]

Hey,
I have a server with 24 bays, were 23 of them are populated with 300Gb 10K SAS drives and one with an SSD.
At the moment, I have 2 pools:

1. LARGE: 2 x 6 RaidZ2 (= 12 drives) and the SSD for cache.
2. SMALL: 5 x 2 mirror vdev (= 10 drives)

and one drive is there as a warm spare, mainly because the SSD was there before I could put its pair for another mirror vdev, but also, cause it's not always possible to get to the data-centre within 24 hours.
(I named them SMALL/LARGE for simplicity)

The reason I have 2 pools, is because I host 2 types of XenServer VMs (~20 are ON at any given moment) with it, while some are more "mission critical" than others. Another reason is the nice Supermicro-Bridge-Bay, which actually allows me to have a dedicated host for each pool (with its own quad-core and 24Gb of RAM, which is plenty).

Both capacity and performance are sufficient at the moment to host *all* VMs in either pool (specially if the SSD stays on the LARGE pool). As an example, I can (tested) moved all VMs to the SMALL pool, and I can do a scrub and a backup (snapshot) at the same time, without overloading anything. but this is obviously not the desired state.

For those curious, at peak I can get from the SMALL pool: >3000 IOPs and > 400 Mb/s.
but this is very far from the average (current) usage and was never tested for more than a few seconds of burst.

What I would like to know, is if there is a better layout of the pools, taking into consideration future expansions like:

• more storage needed - which can be achieved by upgrading to bigger disks.
• adding SSDs for both L2ARC (1 bay) and/or ZIL (2 bays)
• spare drives - though I have one warm (in a bay inside the server) and several cold (out of the server, but in a short distance)

appreciate any input or ideas on this...

Thanks!

 

[zambanini]

l2arc and only 24gb of ram? not good.
a pool with only two vdevs..your raidz2 will give you low IOs.

 

[MtK]

l2arc and only 24gb of ram? not good.
a pool with only two vdevs..your raidz2 will give you low IOs.

Read the post again, like I wrote the pool giving the high IO was the SMALL one with the 5 mirror vdevs, and again like I wrote, it works fine for what it needs to do...

 

[MtK]

Any suggestions?

 

[jgreco]

l2arc and only 24gb of ram? not good.

Could be worse, it will probably be okay at warming over time as long as the workload isn't demanding. However, this seems like you're trying to make a slow pool that will always be slow marginally faster with the SSD, while it seems to me that you could instead use it for the faster pool for a better speed increase and make your moderately fast pool a good bit faster.

 

[MtK]

Could be worse, it will probably be okay at warming over time as long as the workload isn't demanding. However, this seems like you're trying to make a slow pool that will always be slow marginally faster with the SSD, while it seems to me that you could instead use it for the faster pool for a better speed increase and make your moderately fast pool a good bit faster.

I see your point, but having 2 pools is not a bad thing in this case, since some VMs require much more than others...
And that is mainly why I'm trying to keep my future option open

 

[jgreco]

I wasn't suggesting eliminating a pool. I was suggesting focusing on making a faster fast pool and letting the slower one be sucky.

 

[MtK]

I wasn't suggesting eliminating a pool. I was suggesting focusing on making a faster fast pool and letting the slower one be sucky.

Sorry, don't think I understood what are you suggesting...

 

[jgreco]

Take the SSD off of the slow pool. Put it on the fast pool.

Right now you have one faster pool that isn't real fast, and one slower pool that's been made a little faster. On a scale of 1...10 the slow is about a 3 or 4 and the fast is maybe a 6.

Let the slow pool be a solid 3 and get the fast pool to be maybe a 6 or 7.

 

[MtK]

Take the SSD off of the slow pool. Put it on the fast pool.

Right now you have one faster pool that isn't real fast, and one slower pool that's been made a little faster. On a scale of 1...10 the slow is about a 3 or 4 and the fast is maybe a 6.

Let the slow pool be a solid 3 and get the fast pool to be maybe a 6 or 7.

So let me get it right:

• keep the existing 12-drive 2 x RAIDz2, just without the SSD.
• keep the 5 x mirror-vdev, but with the SSD.

would that make it a solid 3 and a solid 7?

 

[MtK]

@jgreco, Would the SSD actually help here?

 

[jgreco]

So let me get it right:

• keep the existing 12-drive 2 x RAIDz2, just without the SSD.
• keep the 5 x mirror-vdev, but with the SSD.

would that make it a solid 3 and a solid 7?

Yeah, that's the idea. You currently have a sucky pool you're trying to make faster and a fastish pool you're crippling.

Hard to say what the actual numbers would be ... try it and find out...! The L2ARC can always be moved without fuss.

 

[MtK]

Yeah, that's the idea. You currently have a sucky pool you're trying to make faster and a fastish pool you're crippling.

Hard to say what the actual numbers would be ... try it and find out...! The L2ARC can always be moved without fuss.

I can think of several other options.
How about these:
* adding all the drives into one big 2-way mirror (I rather keep my 2 pools)
* making a second mirror vdevs pool (out of the 12 raidz drives (2-way or even 3-way)

 

[jgreco]

Of course, there's lots of options. I don't know what's more useful for you. It was mostly a commentary on what I saw as a slight irrationality. Mirrors are fast and RAIDZ is slow for block storage. If you felt the need for a fast and a slow pool then I think it strange to cripple the fast pool and try to speed up the slow pool.