Help with building a new TrueNAS Server for VMs

[uberwebguru]

Err, the QVO are crap SSD's and definately NOT what you want to be using for a VMStore

How can they be crap though?
I mean did you read the TDW numbers???
What makes it crap?

 

[jgreco]

Getting 870 QVO since it got 8TB and has better endurance of 2880TB (TDW) than 870 EVO which maxes at 4TB

Samy 870 EVO 4TB is 2400TBW $450 avg retail
Samy 870 QVO 4TB is 1440TBW $400 avg retail

The QVO doesn't have better endurance. It looks like it's about 2/3rds the drive for 8/9ths the price.

Now if you have an application that isn't going to be heavy on the writes, then, perhaps, the higher density available in the 8TB QVO might make sense. However, people have reported trouble with QVO devices and heavy usage. So, ...??!?

 

[uberwebguru]

Samy 870 EVO 4TB is 2400TBW $450 avg retail
Samy 870 QVO 4TB is 1440TBW $400 avg retail

The QVO doesn't have better endurance. It looks like it's about 2/3rds the drive for 8/9ths the price.

Now if you have an application that isn't going to be heavy on the writes, then, perhaps, the higher density available in the 8TB QVO might make sense. However, people have reported trouble with QVO devices and heavy usage. So, ...??!?

870 QVO 8TB is 2880TB TDW
$700-750

I get the performance might be lower than 870 EVO but i will not ignore the 2880TB TDW and 8TB capacity

 

[uberwebguru]

@NugentS

Looking at your Primary TrueNAS Storage specs

What are these and do i need these also?

LSI 9305-16i IT Mode SAS HBA
SAS3008 on board in IT Mode

and lastly, is this typo? 512TB? and 2 x 512TB?

NVMEPool: 2 * 512TB + 2 * 2GB NVMe in same Bifurcated X16 slot as striped mirrors in Bifurcated X16 slot + Mirrored Optane 900P (Partition) as SLOG

 

[NugentS]

@NugentS

Looking at your Primary TrueNAS Storage specs

What are these and do i need these also?



and lastly, is this typo? 512TB? and 2 x 512TB?

yup - its a typo - thank you

The LSI & SAS ports are because I don't have a SAS expander backplane - but I do have a 24 slot chassis so I need 24 SAS lanes for 24 slots. I use the motherboard SATA ports for boot disks and such

 

[uberwebguru]

yup - its a typo - thank you

The LSI & SAS ports are because I don't have a SAS expander backplane - but I do have a 24 slot chassis so I need 24 SAS lanes for 24 slots. I use the motherboard SATA ports for boot disks and such

So i dont need the LSI and SAS right?
Planning on getting the Dell R730xd and i think with RAID controller H730P

Electronics, Cars, Fashion, Collectibles, Coupons and More Online Shopping | eBay

Buy and sell electronics, cars, fashion apparel, collectibles, sporting goods, digital cameras, baby items, coupons, and everything else on eBay, the world's online marketplace

www.ebay.com

Just want to make sure, i am not forgetting anything

I plan to use 2 x 40G card
I think most pools will be like max 8 disks together
Will mostly likely use the 870 EVO 4TB in my pools
The 870 QVO 8TB will be used as local storage for Database servers
Not going to use NAS for database storage yet till i think i can, if at all
And will all be SATA SSD, no NVMe for me

 

[HoneyBadger]

with RAID controller H730P

Sure, if your plan is to rip it out and install an HBA330 instead.

870 QVO 8TB is 2880TB TDW
$700-750

I get the performance might be lower than 870 EVO but i will not ignore the 2880TB TDW and 8TB capacity

Hat tip to Anandtech for the graph: https://www.anandtech.com/show/15887/the-samsung-870-qvo-1tb-4tb-ssd-review-qlc-refreshed/2

You're potentially going to give up a lot of performance if you use the QVOs (yes, that's a spinning WD Red nipping at its heels!) and I certainly wouldn't subject them to sustained writes long-term for the sake of their endurance/write amplification, including as local DB drives.

QLC NAND isn't quite mature enough for this use case IMO - I'd spend the extra few bucks and buy EVOs.

 

[uberwebguru]

Sure, if your plan is to rip it out and install an HBA330 instead.

What do i need the HBA330 for? And what do i replace it with?
sorry if question seem dumb, just want to confirm

Yeah i decided to return the 870 QVO 8TB and get the 870 EVO 4TB instead

 

[HoneyBadger]

What do i need the HBA330 for? And what do i replace it with?
sorry if question seem dumb, just want to confirm

You replace the H730P with the HBA330, because you should not use a RAID card with TrueNAS.

 

[uberwebguru]

You replace the H730P with the HBA330, because you should not use a RAID card with TrueNAS.

So why do i need to replace and not remove?
Again sorry if this sounds dumb, just tryna learn
Never used hardware RAID ever(because i really dont believe in them)

Also can i just not use the RAID controller and leave it in? Must i remove it to use TrueNAS?
No way to ignore it?

 

[uberwebguru]

yup - its a typo - thank you

The LSI & SAS ports are because I don't have a SAS expander backplane - but I do have a 24 slot chassis so I need 24 SAS lanes for 24 slots. I use the motherboard SATA ports for boot disks and such

Looking at your SSD pool

Code:

SSDPool: 6 * DC S3610 1.6TB SSD in 3 * mirrored pairs + Mirrored Optane 900P (Partition) as SLOG

What are these? + Mirrored Optane 900P (Partition) as SLOG
Are those on same pool or a separate pool?

 

[NugentS]

What I am doing here is non-standard and not supported.
Because I use iSCSI & NFS - as ESXi datastores - I am using sync write. For safety I am using sync=enabled (which is slow)

I have taken 2 Optane 900P's and manually partitioned them into 3 * 20GB partitions, leaving the rest unallocated. I have then, for each pool that uses a dataset that is sync=enabled added 1 partition from each drive as a mirrored SLOG/ZIL. Thus the optanes are running as a SLOG for three different pools

If I was putting a heavy load on the optanes I probably wouldn't do this - but I am not, so I can

 

[Ericloewe]

Also can i just not use the RAID controller and leave it in?

You can also burn a pile of money, it's about as useful. And you'd need an additional HBA, so there really is no benefit to it.

 

[jgreco]

870 QVO 8TB is 2880TB TDW
$700-750

Yes, but its endurance is crap. Just because it's available as a larger drive doesn't make it better, it just makes it a larger pile of low endurance crap. Most people don't build VM environments to survive in low endurance storage, because it's a bit difficult.

 

[NugentS]

A friend of mine built an array on QVO drives and used them as a iSCSI store for a 3 server cluster of ESXi hosts.
I think the drives shat themselves after about 3 months - total loss of the array.

 

[uberwebguru]

Yes, but its endurance is crap. Just because it's available as a larger drive doesn't make it better, it just makes it a larger pile of low endurance crap. Most people don't build VM environments to survive in low endurance storage, because it's a bit difficult.

Returning for the 870 EVO 4TB

 

[uberwebguru]

A friend of mine built an array on QVO drives and used them as a iSCSI store for a 3 server cluster of ESXi hosts.
I think the drives shat themselves after about 3 months - total loss of the array.

No RAID? did he setup similar to RAID 0 or something?
Or we can lay blame completely on the storage drives

 

[uberwebguru]

@jgreco @NugentS

#1
What is best way to setup a pool for 4 x 870 EVO 4TB?
Or best to to go 6 x 870 EVO 4TB in a pool?

I am tending towards starting with 4 and then can expand the pool with more discs later
Wanted to get your advice

#2
Also what i be restricted to get the speed from the pool? What will be my constraints and what is max speed i can get?
I am trying to put 2 x 40G card and wanted to know if i can get anything close to that
Anything else i need to buy here, or am all set?

At stage where i can still change things now
So whatever i need to decide will be best now

 

[NugentS]

No RAID? did he setup similar to RAID 0 or something?
Or we can lay blame completely on the storage drives

It was a synology NAS

 

[HoneyBadger]

Forewarning - this post is pretty much a bunch of Resources links.

So why do i need to replace and not remove?
Again sorry if this sounds dumb, just tryna learn
Never used hardware RAID ever(because i really dont believe in them)

Also can i just not use the RAID controller and leave it in? Must i remove it to use TrueNAS?
No way to ignore it?

Well you'll need some sort of storage controller in order to access the 24x drive bays in the front of your server. You just can't/shouldn't use a RAID controller and instead use an HBA.

What's all the noise about HBAs, and why can't I use a RAID controller?

1) An HBA is a Host Bus Adapter. This is a controller that allows SAS and SATA devices to be attached to, and communicate directly with, a server. RAID controllers typically aggregate several disks into a Virtual Disk abstraction of some sort...

www.truenas.com

Generally there's a limited number of PCIe or expansion slots in a server. The RAID controller can be kept as part of a Very Highly Available Boot Pool strategy, see below, but you need to be able to separate it from the rest of the pool and make sure its filthy RAID-ness doesn't touch your precious zpool.

Highly Available Boot Pool Strategy

I've pounded a few versions of this out over the years, but I hate explaining over and over. TrueNAS allows a ZFS boot mirror pool to be created to increase the reliability of the NAS. This sounds great in theory, but there's a flaw. Due to...

www.truenas.com

And since this is designed to host VMs, have a look at the justification for mirrors over RAIDZ:

Some differences between RAIDZ and mirrors, and why we use mirrors for block storage

ZFS is a complicated, powerful system. Unfortunately, it isn't actually magic, and there's a lot of opportunity for disappointment if you don't understand what's going on. RAIDZ (including Z2, Z3) is good for storing large sequential files...

www.truenas.com

As well as the fact that you'll need to devote a PCIe slot or two for mirrored SLOG devices similar to what @NugentS has done to ensure fast/safe writes:

Resource - Sync writes, or: Why is my ESXi NFS so slow, and why is iSCSI faster?

This post is not specific to ESXi, however, ESXi users typically experience more trouble with this topic due to the way VMware works. When an application on a client machine wishes to write something to storage, it issues a write request. On a...

www.truenas.com

Given that you want to link this via 40GbE you will need a seriously speedy SLOG device if you want to keep up - you're well into the point where genuine NVDIMMs would be required but you can get a decent chunk of speed by using a large-capacity Optane DC device (P4800X, P5800X) - a budget SSD will not cut it here!