Disable Jumbo frames on a switch

[acp]

Do you have to have a external storage for the VMs ? Can you just do ssd inside the hypervisor or raid with sdd for redundancy still inside the hypervisor and call it a day ?

I tried that but the array kept getting corrupted. Could be because it was software raid. After the second failure I went back to using freenas for storage.

Sent from my Nexus 5X using Tapatalk

 

[Black Ninja]

Truth is I don't know xen. I am a vmware ESXI fan, and esxi don't support software raid. But that's fine cause I like hardware raid better anyways and that works and it's rock solid. Tons of IOPS and speed from just 2 SSDs in RAID1

 

[jgreco]

With ESXi, it's really hard to beat something like an LSI 9270CV-8i (available with flash and supercap on eBay for $100-$150 often enough) as a local storage controller. It does take some research, and you can definitely run into issues if you're not careful, but a pair of SSD's in RAID1 on a 9270 is about the fastest non-NVMe storage available and it is wicked fast. 1GB cache. You can move on up to the 12Gbps stuff and get a bit of a bump but that's like $700 for a fully outfitted controller.

The great thing is that it's a Real RAID Controller, used in enterprise environments, and it can easily be set up so that stuff like "pull failed drive, insert new drive" automatically does the rebuild, or that you can have a warm spare drive, and it will automatically rebuild on failure, and the idiot lights light, and there's no "oops I forgot to configure the disk in the host platform" so often associated with software RAID platforms. My favorite configuration is 3 x WD Red 1TB 2.5", 2 in RAID1/1 hotspare, 5 x ~500GB SSD (Intel 535, Samy 850, etc) in two separate RAID1 and a hotspare. You can actually make an I/O subsystem with tiers, including HDD, a high endurance SSD tier, and a light-use SSD tier, protected with hotspares, very inexpensively. The downside is that as a Real RAID Controller, you have to make sure you do things like disable WDIDLE on those Reds, and expect to have to do some learning about how to make the thing work.

 

[acp]

With ESXi, it's really hard to beat something like an LSI 9270CV-8i (available with flash and supercap on eBay for $100-$150 often enough) as a local storage controller. It does take some research, and you can definitely run into issues if you're not careful, but a pair of SSD's in RAID1 on a 9270 is about the fastest non-NVMe storage available and it is wicked fast. 1GB cache. You can move on up to the 12Gbps stuff and get a bit of a bump but that's like $700 for a fully outfitted controller.

That is a sweet looking board. Perhaps in the future, but I will keep this on the radar.

 

[Black Ninja]

I do agree with Resident Grinch, as a matter of fact that's how I setup my hypervisor base on his advice over a year ago. Perhaps the only thing I would change(and I did change) is the choice of drives. Like getting radi1 with Intel S3700 800GB or 400GB and for mechanical drive I would got with HGST 1.2TB SAS.

Make sure I have a proper cooling for the raid card, it has crucial importance for you system reliability. That's the hardest thing for me to do when planing a new system, and I do use only server chassis.

 

[jgreco]

Yeah, a lot of the environments I work with are heavy on the reads and not too stressy on the writes, plus modest budgets, so I like to optimize for inexpensive, but you can definitely beef up the storage devices. Eight are directly supported, but you can also toss an SAS expander into the mix.

 

[Black Ninja]

I expect also in your case (your customers) wouldn't go for used ones, which in my case is not a problem to do cause I am getting them for myself.

Ohhh I forgot ! There is something I always wanted to ask you about your LSI 9631 (in your hypervisor 1U, WIO,1650v3) what is the temp of the raid card in this configuration ?

P.S. I assume since mine is 9271 and show the temp, yours being newer model will show too ? Unlike LSI 9260 that don't show.

 

[jgreco]

I expect also in your case (your customers) wouldn't go for used ones, which in my case is not a problem to do cause I am getting them for myself.

I have customers that are subject to significant budget pressures. With two of the 9270CV's going for less than one brand new LSI HBA 9240/9340/whatever on the retail market, and the performance difference being so big, I've got a system being built for a customer right now on the bench as an ESXi hypervisor where I flat out said "we can do the new HBA, or a RAID controller at a price you've turned down in the past, or I can source a pair of 9270CV's on the used market so you have a spare, it's the cheapest AND the fastest option."

Ohhh I forgot ! There is something I always wanted to ask you about your LSI 9631 (in your hypervisor 1U, WIO,1650v3) what is the temp of the raid card in this configuration ?

P.S. I assume since mine is 9271 and show the temp, yours being newer model will show too ? Unlike LSI 9260 that don't show.

/opt/lsi/storcli/storcli /c0 show all|grep -i "ROC temp"
ROC temperature(Degree Celsius) = 58

/opt/lsi/storcli/storcli /c0 show all|grep -i "ROC temp"
ROC temperature(Degree Celsius) = 56

But these are 1U WIO with "all da fanz" so they don't run the stupid-hot temps so common when running in a standard PC.

It looks like the LSI 2108 did not have the temp sensor but the 2208 does based on a very small sample set, so if you have a 9265 or 9270 or other 2208-based, you should have temp out. The 9260 is based on the older 2108 so it doesn't. I suspect everything newer does but I don't plan to go check right now.

 

[Black Ninja]

I wouldn't put raid card in standard PC(without active cooling) and insult you by asking about the your temps.:)

I have temp sensor on my 9270. Mine is 64-67 Celsius in SC113MTQ-R400CB chassis also with all the fans (but these are single fans unlike yours). I keep the fans on "standard". I would assume you don't have this great temps cause you are running them on "full" more right ?

 

[jgreco]

I wouldn't put raid card in standard PC(without active cooling) and insult you by asking about the your temps.:)

I have temp sensor on my 9270. Mine is 64-67 Celsius in SC113MTQ-R400CB chassis also with all the fans (but these are single fans unlike yours). I keep the fans on "standard". I would assume you don't have this great temps cause you are running them on "full" more right ?

No, just standard. When replacing a fan involves a thousand-dollar site visit and a hard service downtime, there's more value in going easy on the hardware. Unlike most sites where they just unbox and install the things, though, we ran them hard in the shop for awhile and did the little extras like replacing the fan blanks with actual fans, and probably did some airflow management. I'm very into optimizing these sorts of things.

We've been working our way through refurb work on a few dozen systems that are essentially 1027B-URF, which are 1356 based E5-2400's with Supermicro's UIO form factor, which is basically a regular PCIe slot and a reversed PCIe slot ("UIO") for which you have to buy an unobtanium UIO card. So we've been sticking in a more standard non-UIO riser, and subbing in a slotted blank on the top PCIe slot to allow a modest airflow, and then a full Supermicro "hardware cloth" style slot blank on the bottom to encourage maximal airflow there, and putting the RAID controller in the bottom slot. This allows for easy addition of a 10G card at a later date. Basically these cards are very good in a 1U environment as long as you're flowing air over them.

The 56mm 0086L4 counter-rotating fan probably pushes a ton more air than the 0065L4 fan in that SC113 of yours, but the M's come with the short depth fan tray and the regular fans. Looking a bit at the picture, though, it looks like you really might be able to sub in the 56mm fan tray, which you can probably get through WiredZone etc. Part number is CSE-PT0135 for the 56mm tray. No promises that it would actually fit, though, it's just a crazy idea. Maybe you can pick one up on eBay cheap. But don't even bother unless you open up your chassis and you think there's a chance it might fit. I could just be incredibly optimistic - that's how I get most of my crazy done.

 

[Black Ninja]

The 56mm 0086L4 counter-rotating fan probably pushes a ton more air than the 0065L4 fan in that SC113 of yours, but the M's come with the short depth fan tray and the regular fans. Looking a bit at the picture, though, it looks like you really might be able to sub in the 56mm fan tray, which you can probably get through WiredZone etc. Part number is CSE-PT0135 for the 56mm tray. No promises that it would actually fit, though, it's just a crazy idea. Maybe you can pick one up on eBay cheap. But don't even bother unless you open up your chassis and you think there's a chance it might fit. I could just be incredibly optimistic - that's how I get most of my crazy done.

Daaaaamn you are good.:)

Let me research and think...

,but in a mean time do you think 67 Degree Celsius is too hot. I would think up to 70 should be fine, but I could be wrong. I did purchased this LSI 9271 (with super-cap as you suggest long time ago) for around $700 so I don't want to wrong and not realizing it and screw my card.

 

[jgreco]

Eh. I err on the side of cool where possible.

Wait, wait, I got it... better answer...

I'm actually just seeing if I can get you to blow more money on more silly hardware. ;-)

 

[Black Ninja]

I did want to get a cocked raid card that some kid put it it's desktop and after few crashed from overheating then put it on ebay for me to get.:)

Anyways,

the idea about faster fans is goo if they can fit but, there is a better solution:

If I could just give a little bump in it fan speed from "standart" to little faster they will cool the card better and still be in a quiet zone. And that should be piece of cake if supermicro did put their shit together:

For example on X9SRW-F you can switch from "standard" to "heavy-io" mode and it will do just that , cause that is what is suppose to do.

But on any X9 (X9SRL, X9SCM,....) that has the options: STANDARD, OPTIMAL or FULL you can only do FULL or STANDARD cause the optimal never worked, they just put the option there but it didn't do anything.:(

I learn this the hard way.:-((

P.S. If my had "hevy-IO" instead of "optimal" I could just switch to that and be perfect.

 

[Black Ninja]

I'm actually just seeing if I can get you to blow more money on more silly hardware. :) ;-)

That you can do very well. But I can't blame you cause I know I am asking for it.:)

P.S. Perhaps now is good time when I am building a new ESXI that I really need to be universally great.

 

[acp]

P.S. Perhaps now is good time when I am building a new ESXI that I really need to be universally great.

You mean awesome. [emoji39]

Sent from my Pixel 2 using Tapatalk

 

[Black Ninja]

You mean awesome. [emoji39]

Sent from my Pixel 2 using Tapatalk

Yeah, I meant awesome and also to be multipurpose(universal). But I combined them both too fast and lost the meaning. It's interesting how one's mind work, now when I read back what I have said.

Thanks for correcting me.