10GBE maximizing performace

[pjr_atl]

Just added a direct 10GBE connection to my freenas 11.2 beta 2 N40L Microserver with 16 GB RAM and 5 WD red 4TB drives in raidz. JUmbo frames are enabled on the 10 gig link

using crystaldiskmark I see 428Mb/s read and 395 MB/s write using a SMB3 connection. CPU utilizatoin is under 15% during the test.

I am surprised by the write speed as I expected the write to be to the RAM...

what should my next steps be to further tune this system?

Thanks in advance

 

[HoneyBadger]

what should my next steps be to further tune this system?

Knee-jerk reaction is "more vdevs"

Those drives have a 150MB/s internal transfer speed according to their spec sheet:

https://www.wdc.com/content/dam/wdc/website/downloadable_assets/eng/spec_data_sheet/2879-800002.pdf

Taking into account network overhead, that's about the level of performance I'd expect.

The N40L is also a pretty weak system by current standards. Is that 15% the CPU utilization of the client or the FreeNAS machine?

 

[pjr_atl]

I actually am happy with the read speed 400 out of a theoretical 600 seems about right I see that limited by drive transfer rates

Its the write side I was hoping for bigger numbers on do to caching

 

[Chris Moore]

Its the write side I was hoping for bigger numbers on do to caching

What cache do you have that you are expecting to make the drive mechanically faster?

 

[pjr_atl]

snarky a bit Chris? :)

I expect no cache to make a drive faster Chris I don't believe in magic, I expect a write cache to buffer writes to the slower device, you know what all caches do :)

so back to the real question in order to get effective write cache on ZFS for SMB access what are the recommended steps?

Thanks in advance

 

[HoneyBadger]

Again, this is likely the maximum level of performance you're going to get from those drives in a RAIDZ1, considering the overhead of crossing the network and doing the parity.

You could have an infinitely fast "write cache" in the middle, but unless your back-end drives can keep up, eventually you'll slow down to the speed of your vdev(s).

https://forums.freenas.org/index.php?threads/when-to-upgrade-ram.68890/#post-472865

The only other explanation would be that you measured CPU usage at the client side and it's the poor dual-core 1.5GHz AMD in that N40L that's running out of steam. But I don't know how much more you'd expect from those drives ... close to 400MB/s write is pretty good for a 5-drive Z1.

 

[pjr_atl]

I guess a little more explanation of my use case is warranted.

This is a home lab/media server. Maximum users at one time is 3 (and thats rare) primary situation I am optimizing for is transfer of 3 to 6 gig files to the freesias box. 3 of these transfers an hour would be a lot. Performance is acceptable but I am a geek and want t do more if I can at a reasonable cost. This is me playing with Tech not me designing of a DC with millions of users and 10K + servers (I do that too but not with this little freenas box). The CPU utilization was on the freenas server I am surprised how this old box is holding up but again the actual utilization on this box is extremely low

So yes I want an extremely fast write cache just for the grins and giggles and to see if I can max out the 10GBE link

reading your linked to thread now good stuff thanks

I think this https://www.ixsystems.com/blog/o-slog-not-slog-best-configure-zfs-intent-log/ is the answer to my question

 

[diskdiddler]

Let me tell you, I literally get one tenth your performance and I have the N54L and 6 drives.
This is what happens when your FreeNAS machine is full right up.

 

[pjr_atl]

Let me tell you, I literally get one tenth your performance and I have the N54L and 6 drives.
This is what happens when your FreeNAS machine is full right up.

ouch I got 85% before I started swapping out my 2TB for 4TB :)

 

[diskdiddler]

I'm at 94% right now and waiting to get some money in. Replacing my server is going to be, very, very expensive for my budget and country.
Will be nice to see in excess of 200MB R/W but I don't actually need it, that badly.

 

[Chris Moore]

Replacing my server is going to be, very, very expensive for my budget and country.

Are you going to build a new server or just put larger drives in this one?

 

[Chris Moore]

snarky a bit Chris? :)

Sorry. I sometimes fail to do what I should. Nobody is perfect.

I expect no cache to make a drive faster Chris I don't believe in magic, I expect a write cache to buffer writes to the slower device, you know what all caches do :)

The only thing you could do to add cache that would work the way you want it to work would be to add RAM to the system. Still, ZFS must flush the transaction group on schedule, so if the drives can't keep up, eventually you are going to slow down to the speed of the drives. I have been working on issues like that at my job and I have to repeatedly explain that the number of drives is needed to support the data rate. I need double the number of drives per server to be able to satisfy the users need for speed, but management doesn't want to spend that much money.

 

[pjr_atl]

No worries Chris I have seen how much help you have given others here and your knowledge is impressive. I can get my hands on a Intel S3700 I'm going to install it as a SLOG just to see what happens.

I am staring to think about what I'd really like to build Xeon-d , bunch -o-ram,. Optane Slog just need that money tree to bare fruit :)
also wondering about lots of 2.5 drives vrs 3.5 spinners... so many options

 

[HoneyBadger]

I can get my hands on a Intel S3700 I'm going to install it as a SLOG just to see what happens.

You'll lose speed.

Right now you are writing to RAM and only RAM - adding a slow-by-comparison SATA SSD and forcing the writes to be acknowledged there will only cause them to slow down.

 

[pjr_atl]

You'll lose speed.

Right now you are writing to RAM and only RAM - adding a slow-by-comparison SATA SSD and forcing the writes to be acknowledged there will only cause them to slow down.

I am basing my decision on this https://www.ixsystems.com/blog/o-slog-not-slog-best-configure-zfs-intent-log/ which states the Zil is actually written to the disk pool not held in memory....

 

[diskdiddler]

Are you going to build a new server or just put larger drives in this one?

Still deciding, it's awfully expensive. I'd prefer new disks and ship my existing system to someone else to use as an offsite backup.

I would've liked to try the disk import functionality though, I've never used it - but I do get a full defrag if I move it all off the system.

 

[UDO]

hi @pjr_atl - Out of interest, which 10gbe adapter did you select ?
Can you share an overview of your network setup ?

 

[Chris Moore]

I am basing my decision on this https://www.ixsystems.com/blog/o-slog-not-slog-best-configure-zfs-intent-log/ which states the Zil is actually written to the disk pool not held in memory....

Unless you're doing synchronous writes, it doesn't really matter


Sent from my SAMSUNG-SGH-I537 using Tapatalk

 

[pjr_atl]

Unless you're doing synchronous writes, it doesn't really matter


Sent from my SAMSUNG-SGH-I537 using Tapatalk

ah good catch Chris thanks

 

[pjr_atl]

hi @pjr_atl - Out of interest, which 10gbe adapter did you select ?
Can you share an overview of your network setup ?

I used these
https://smile.amazon.com/gp/product/B016OYD0D4/ref=oh_aui_detailpage_o02_s00?ie=UTF8&psc=1

and its just a peer to peer connection install and configuration was easy (other than a stupid used error involving net masks :) )