New NAS build ZIL drive 100% used poor performance

[robertmehrer]

If you plan to keep this system around and want an actual service contract with an SLA, reach out to iX here and see if they can facilitate a contact.

https://www.ixsystems.com/freenas-commercial-support/

ill get this over to the product guys to reach out to them, thank you!

 

[robertmehrer]

I am done criticizing, if you didn't realize.

I realize that you can't destroy the one you are using. That is why I said it would be building a new system to do it right.

Yes when the box gets back and data is offloaded its being rebuilt. Ill have enough time to redo everything and properly test it.

 

[HoneyBadger]

Yes i was questioning the build as well. I didnt realize the Beta was the downloaded version until after i had everything setup and it was being delivered.

You've got the makings of a solid build there; replacing the H700 with a flashed H200 and reverting to 11.1-U6 should get you most of the way, the rest you can do with tunables.

When your timeline isn't as crunched, pop back into the "New to FreeNAS" subforum and we'll see about getting you up and running on a solid platform suited for your needs.

Edit: For giggles you could leave the EVO 960 in there long enough to do the SLOG benchmark in my signature.

 

[robertmehrer]

Storage -> Volumes -> Click your volume -> Volume Status (at bottom) -> Select your actual log/cache drive -> Offline/Remove

See 8.1.10.2 Removing a Log or Cache Device

http://doc.freenas.org/11/storage.html#removing-a-log-or-cache-device

Removed the cache drives and now sustaining 9.3Gbps network traffic to the box...

 

[Chris Moore]

Hey now, don't hate on the R510!

Not hate. There is nothing inherently wrong with it. It just needs more drive bays, for more vdevs, for a higher combined data rate.

 

[robertmehrer]

You've got the makings of a solid build there; replacing the H700 with a flashed H200 and reverting to 11.1-U6 should get you most of the way, the rest you can do with tunables.

When your timeline isn't as crunched, pop back into the "New to FreeNAS" subforum and we'll see about getting you up and running on a solid platform suited for your needs.

Edit: For giggles you could leave the EVO 960 in there long enough to do the SLOG benchmark in my signature.

Yea ill do that. Again Thank you so much for the help!

 

[kdragon75]

If there was paid support for FreeNAS on dated hardware and we could have paid for service contracts we would.

Don't tell us, Tell iXsystems! Your far from the first.

and why do you think im on here asking the experts?

You mean random people on the internet?

Im not too proud to say it wasnt done 100% properly based on the 12 hours i had to pull hardware together and throw FreeNAS on it...

Fire whoever took on this client and didn't do a proper assessment! Seriously it sounds like my old MSP. I would have a detailed PO and when the boxes showed up half the stuff was missing, the servers were configured wrong, and I was dead in the water having to order network cards, cables, mounts, etc.. The reason? The manager responsible didn't see that it was a two page order... They were a bunch of hacks.

 

[robertmehrer]

Not hate. There is nothing inherently wrong with it. It just needs more drive bays, for more vdevs, for a higher combined data rate.

We have a 36Drive 943 supermicro but shipping that beast is a pain, we got better performance out of the r510 with 12 drives based on the drives specs.

 

[Chris Moore]

Removed the cache drives and now sustaining 9.3Gbps network traffic to the box...

That is really good, but you should still be using RAIDz2 to have some level of redundancy. Safety first.

 

[robertmehrer]

Don't tell us, Tell iXsystems! Your far from the first.

You mean random people on the internet?

Fire whoever took on this client and didn't do a proper assessment! Seriously it sounds like my old MSP. I would have a detailed PO and when the boxes showed up half the stuff was missing, the servers were configured wrong, and I was dead in the water having to order network cards, cables, mounts, etc.. The reason? The manager responsible didn't see that it was a two page order... They were a bunch of hacks.

Yea from what im seeing its a large complaint!!

Usually you boys on the opensource forums are better experts than the dev team who develops the stuff!!

This was a recover from a failed system, a BU was lost on their end and had to be reseeded. Their apollo had a space issue and lost the BU... we had to re-backup and reseed. So it was a wild fire!

 

[kdragon75]

Their apollo had a space issue

You mean the admin wasn't doing proper cap planning.
the good news is that once this is all sorted, you will be ready for next time!

 

[robertmehrer]

You mean the admin wasn't doing proper cap planning.
the good news is that once this is all sorted, you will be ready for next time!

Yea they dont really have an admin... lmao...

Its hard for them to cap plan, it really depends on how much crime happens lol Its been a struggle with them!

 

[HoneyBadger]

Removed the cache drives and now sustaining 9.3Gbps network traffic to the box...

Lovely results, suggests that the 960 EVO was just overwhelmed. Was it a PCIe card or an M.2 on an adapter?

(Don't let this dissuade you from getting the H200 and recreating the setup properly though!)

As suggested previously, you really need some manner of redundancy, even if it was only RAIDZ1 (Z2 strongly preferred given your drive sizes) because the time lost in having to re-copy all of these files is non-trivial. Safety first (and sometimes second!)

Usually you boys on the opensource forums are better experts than the dev team who develops the stuff!!

We've got OpenZFS devs lurking around here as well, I sure wouldn't want to get into a pissing match with one of the guys who authored the filesystem.

 

[robertmehrer]

These are the kinds of numbers i was expecting!!! Our seed time went from weeks to 2 days!

 

[robertmehrer]

Lovely results, suggests that the 960 EVO was just overwhelmed. Was it a PCIe card or an M.2 on an adapter?

(Don't let this dissuade you from getting the H200 and recreating the setup properly though!)

As suggested previously, you really need some manner of redundancy, even if it was only RAIDZ1 (Z2 strongly preferred given your drive sizes) because the time lost in having to re-copy all of these files is non-trivial. Safety first (and sometimes second!)



We've got OpenZFS devs lurking around here as well, I sure wouldn't want to get into a pissing match with one of the guys who authored the filesystem.

yea H200 has been ordered!

Its an M.2 on a PCIe 2 x4 adapter, its all the MoBo could handle.

Im sure they would have some insight... but generally you guys are doing what im doing and testing hardware and build-outs for the best performance.

 

[HoneyBadger]

Solid real-world application of the saying "Never underestimate the bandwidth of a station wagon full of tapes barreling down the highway."

 

[robertmehrer]

Yea these are the fastest seed speeds we've seen so far... Looks like we are on the right path, just gotta do some tweaking!

 

[HoneyBadger]

yea H200 has been ordered!

Good. I'd suggest doing the cross-flash in a separate system just to avoid the long server reboot cycle.

Its an M.2 on a PCIe 2 x4 adapter, its all the MoBo could handle.

I'm betting you could fry an egg on it right now, it's probably overheating. Want to do a
smartctl -a /dev/nvme0 (or nvme1, depending on which one it was) and post the "Temperature" result?

Im sure they would have some insight... but generally you guys are doing what im doing and testing hardware and build-outs for the best performance.

ZFS might be my job, but I wouldn't dream of putting myself on the level of a dev. Once I've got a few accepted commits under my belt I'll talk a bigger game. ;)

 

[robertmehrer]

Good. I'd suggest doing the cross-flash in a separate system just to avoid the long server reboot cycle.



I'm betting you could fry an egg on it right now, it's probably overheating. Want to do a
smartctl -a /dev/nvme0 (or nvme1, depending on which one it was) and post the "Temperature" result?



ZFS might be my job, but I wouldn't dream of putting myself on the level of a dev. Once I've got a few accepted commits under my belt I'll talk a bigger game. ;)

That command doesnt work, its stating unable to detect device type

 

[robertmehrer]

has anyone seen an issue with the Reporting on the iSCSI port doesnt show any traffic within the GUI?