New NAS build ZIL drive 100% used poor performance

[robertmehrer]

Part of the problem. Probably memory bandwidth. That is an old / slow processor, LGA 1366, for trying to do 10Gb networking.

What is the network card?

ixl0@pci0:3:0:0: class=0x020000 card=0x00018086 chip=0x15728086 rev=0x01 hdr=0x00
vendor = 'Intel Corporation'
device = 'Ethernet Controller X710 for 10GbE SFP+'
class = network
subclass = ethernet

 

[Chris Moore]

I posted some screenshots of the speeds I get with my NAS here, just a couple days ago:
https://forums.freenas.org/index.ph...-partitioned-for-two-pools.62787/#post-483761

 

[Chris Moore]

All of this advice is, of course, assuming that you take that H700 out and replace it with an H200, flashed with the appropriate LSI IT firmware:

Which may mean recreating the pool entirely because the disks were each setup as RAID-0 devices on the hardware controller.

 

[robertmehrer]

Indeed. Badgering is my job, folks, it's in my username. ;)

To be fair, OP, you did yourself admit to one of the biggest no-no's of ZFS (using a HW RAID controller) so some flak should be expected - but people don't need to be rude about it.



If it's all the same workload of "huge sequential write, physically move the box, huge sequential reads to send it to the final resting place" then yes, ditch the caches altogether. You don't read the same data multiple times (L2ARC benefit) and you aren't writing small blocks and demanding a quick ACK back that it was completed (SLOG benefit) so pull them (and take them home for your gaming machine!)



In your case, if you're sharing ZVOLs via iSCSI, the recordsize value (actually volblocksize) can't be changed after it's initially set. You'd have to make a new zvol/extent/etc and check the "Advanced" options when you make the ZVOL. But see below instead.

Is connecting this server via SMB an option? It's probably better overall as far as performance/tunability.

You'd do zfs set recordsize=1M Poolname/Dataset as well as zfs set sync=disabled Poolname/Dataset to get the two of those, although SMB by default does async writes under Windows so the latter shouldn't be necessary.

All of this advice is, of course, assuming that you take that H700 out and replace it with an H200, flashed with the appropriate LSI IT firmware:

https://techmattr.wordpress.com/201...-flashing-to-it-mode-dell-perc-h200-and-h310/

Thanks!!! this has been very very helpful! But to be clear i cannot remove the cache drives after the Zvol has been created correct? Since we are seeding data currently i would have to destroy it and rebuild it.

 

[Chris Moore]

But to be clear i cannot remove the cache drives after the Zvol has been created correct? Since we are seeding data currently i would have to destroy it and rebuild it.

Not correct. You can remove both the SLOG and L2ARC from the GUI while the pool is active.

 

[kdragon75]

Why don't you give solutions as well as badgering...

Why don't you pay for them?

This was thrown together with gear we had laying around to try and solve a problem quickly. Commvault isnt actually writing the BU data. Its just copying the BU to a second location as a seed. Like VEEAM does, its just an Aux Copy of the already created backup.

If it's worth doing...

Which HBA do you recommend then?

As noted, a crossflash Dell H200. The Dell H310 also works well but the rear card brace needs to be removed. I may have that backwards.... Look at your box and look at some photos of the cards. You will see what I'm talking about. you can also look at the resource section of the forum or searching the internet for FreeNAS HBA.

The underlying drives have a max of 250MBps with 12 drives in a stripe... we should be seeing in thousands range...

I think this is in response to the IOPS comment. You may want to look at how IOPS are calculated for any given drive... It has little to do with throughput.

From the attached screenshot you can see the NAS is capable of performing, it just won't sustain these speeds.

I don't doubt that but benchmarks are funny things.

How do i set those flags?

http://doc.freenas.org/11.2/storage.html?highlight=sync record#adding-datasets
You may also want to read the rest of the user guide.

Just box those up and send them to me, you don't need them... Instead of wasting money like this, you should have come and asked for advice first.

I get one too!

Very short turn time, less than 24 hours to build the box from parts on hand. trying to solve a large issue with a large dataset.

Has your company never had a large client?

Thanks!!! this has been very very helpful! But to be clear i cannot remove the cache drives after the Zvol has been created correct? Since we are seeding data currently i would have to destroy it and rebuild it.

You can remove them any time via CLI. Just don't pull them out.

 

[robertmehrer]

Not correct. You can remove both the SLOG and L2ARC from the GUI while the pool is active.

I dont see that option in the GUI?? Where would it be?

 

[HoneyBadger]

Which may mean recreating the pool entirely because the disks were each setup as RAID-0 devices on the hardware controller.

Covered that point already:

Changing the controller will necessitate destruction and recreation of the pool.

And that's only because you're going from "drives managed by controller" to "drives managed by ZFS" - changing from one IT mode HBA to another will do precisely nothing to the data. IDK if it would remount nicely though.

Thanks!!! this has been very very helpful! But to be clear i cannot remove the cache drives after the Zvol has been created correct? Since we are seeding data currently i would have to destroy it and rebuild it.

As others have stated, you can remove a cache (and log) device from a live pool. There may be a minor pause while ZFS performs these operations but it will resume afterwards.

That said I highly suggest you order the H200 ASAP and make plans for a maintenance window (1hr should do it) to fix the problems.

 

[HoneyBadger]

I don't see that option in the GUI?? Where would it be?

Storage -> Volumes -> Click your volume -> Volume Status (at bottom) -> Select your actual log/cache drive -> Offline/Remove

See 8.1.10.2 Removing a Log or Cache Device

http://doc.freenas.org/11/storage.html#removing-a-log-or-cache-device

 

[robertmehrer]

Why don't you pay for them?

If it's worth doing...

As noted, a crossflash Dell H200. The Dell H310 also works well but the rear card brace needs to be removed. I may have that backwards.... Look at your box and look at some photos of the cards. You will see what I'm talking about. you can also look at the resource section of the forum or searching the internet for FreeNAS HBA.

I think this is in response to the IOPS comment. You may want to look at how IOPS are calculated for any given drive... It has little to do with throughput.

I don't doubt that but benchmarks are funny things.

http://doc.freenas.org/11.2/storage.html?highlight=sync record#adding-datasets
You may also want to read the rest of the user guide.

I get one too!

Has your company never had a large client?

You can remove them any time via CLI. Just don't pull them out.

If there was paid support for FreeNAS on dated hardware and we could have paid for service contracts we would.

I understand the term... time crunch... and why do you think im on here asking the experts? Now that we are seeing a problem we are reaching out for assistance. Im not too proud to say it wasnt done 100% properly based on the 12 hours i had to pull hardware together and throw FreeNAS on it...

Im ordering an H200 now

i used the FreeNAs calculator for RAID and those are the numbers it gave me.

We have very large clients, we just never physically seed this much data. Well lets restate this and say we dont seed this large of a data set at once. Usually smaller IXSytems NAS's are sent to clients to seed a server or two at a time. However when you have a single server with over 60TBs and mount points that grow 3TB every 4 weeks, we had to come up with a solution to seed that much data without shipping 3PAR array to their site. a 144TB R510 seemed like a viable solution if we can tune the performance where it needs to be.

In an effort to speed this can you direct me to the commands or where i would remove the drives in the GUI?

Im currently seeding the data and we need to get back in SLA as soon as possible.

Thanks for your help.

 

[robertmehrer]

Covered that point already:



And that's only because you're going from "drives managed by controller" to "drives managed by ZFS" - changing from one IT mode HBA to another will do precisely nothing to the data. IDK if it would remount nicely though.



As others have stated, you can remove a cache (and log) device from a live pool. There may be a minor pause while ZFS performs these operations but it will resume afterwards.

That said I highly suggest you order the H200 ASAP and make plans for a maintenance window (1hr should do it) to fix the problems.

Thank you, sorry im trying to read replies in order but they are flying in too fast and im seeing them out of order. Again all of this help is greatly appreciated!!

 

[Chris Moore]

Why don't you give solutions as well as badgering...

Would you like to know how to do this the right way? It will be a whole new build to take the place of this one.

 

[robertmehrer]

Storage -> Volumes -> Click your volume -> Volume Status (at bottom) -> Select your actual log/cache drive -> Offline/Remove

See 8.1.10.2 Removing a Log or Cache Device

http://doc.freenas.org/11/storage.html#removing-a-log-or-cache-device

Perfect got it!!! Thank you again, You are awesome!

 

[kdragon75]

Storage -> Volumes -> Click your volume -> Volume Status (at bottom) -> Select your actual log/cache drive -> Offline/Remove

See 8.1.10.2 Removing a Log or Cache Device

http://doc.freenas.org/11/storage.html#removing-a-log-or-cache-device

The OP is on the BETA... I won't go down that rabbit hole..
The remove cache/SLOG section is missing from the 11.2 manual. You can still use the old UI by going to http://<FreeNASIP>/legacy/

 

[robertmehrer]

Would you like to know how to do this the right way? It will be a whole new build to take the place of this one.

I know youre trying to be helpful... youre not... Having actual technical conversations and offering assistance would be great. We are currently seeding the data and cannot destroy the array.

 

[kdragon75]

Would you like to know how to do this the right way? It will be a whole new build to take the place of this one.

Hey now, dont hate on the R510!

 

[Chris Moore]

changing from one IT mode HBA to another will do precisely nothing to the data. IDK if it would remount nicely though.

I have had very good fortune with that. I was able to move pools from SATA controllers to SAS controllers and from BSD to Linux. ZFS rocks.

 

[HoneyBadger]

If there was paid support for FreeNAS on dated hardware and we could have paid for service contracts we would.

If you plan to keep this system around and want an actual service contract with an SLA, reach out to iX here and see if they can facilitate a contact.

https://www.ixsystems.com/freenas-commercial-support/

 

[robertmehrer]

The OP is on the BETA... I won't go down that rabbit hole..
The remove cache/SLOG section is missing from the 11.2 manual. You can still use the old UI by going to http://<FreeNASIP>/legacy/

Yes i was questioning the build as well. I didnt realize the Beta was the downloaded version until after i had everything setup and it was being delivered.

 

[Chris Moore]

I am done criticizing, if you didn't realize.

We are currently seeding the data and cannot destroy the array.

I realize that you can't destroy the one you are using. That is why I said it would be building a new system to do it right.