My new NAS box, running on 8.0.2

[ProtoSD]

It's under the System section, there is Sysctls & Tunables (Loaders).

 

[Daisuke]

I only have General, Advanced, Email and SSL, under Settings GUI tab.

 

[ProtoSD]

Sorry, on the left side in the Tree menu.

 

[Daisuke]

Thanks, I had no idea about those options. Could you give me an example how I enter a tunable setting into GUI? I want to make sure I don't break anything. :)

 

[ProtoSD]

If you look at my earlier post from yesterday you can see the different variable names you can use (there are lots more). If you click on "Tunable" in the pop-up enter the name of the variable on the first line, and then the value I posted to the right of that variable name in the the field below the variable name. Make sure the "Enable" checkbox is clicked, then click OK. Most of the variable require you to reboot before they take effect.

In the worst case you could enter a value too large for kmem_size or some other that could prevent you from booting. Keep a list of the variables you change because if it fails to boot there is a way to fix it if you know the variable names you changed.

What values did you have for AIO read & write size? When I enable AIO things are worse. It's pretty frustrating to keep trying changes and rebooting. Did you check that prefetch = 0?

 

[Daisuke]

I'm confused what and where to add the vars. For example:
vfs.zfs.prefetch_disable=0

I add it where? Tunables or Sysctls?
All these settings you posted earlier go where?

vfs.zfs.prefetch_disable=0
vfs.zfs.arc_max=5074190464
kern.ipc.nmbclusters=10000
vfs.zfs.txg.write_limit_override=1073741824
vfs.zfs.vdev.max_pending=1
vfs.zfs.vdev.min_pending=1
kern.ipc.somaxconn=384
kern.ipc.nsfbufs=150000
net.inet.tcp.inflight.enable=1
net.inet.tcp.recvbuf_auto=1
net.inet.tcp.sendbuf_auto=1
net.inet.tcp.delayed_ack=0

I used 4096 as AIO value in both cases.

 

[ProtoSD]

You can put them in either place, but I prefer to put them in Tunables.

You have to enter them one at a time, it's a pain in the ass, but once they're entered it's easier to enable/disable them for testing.

AIO doesn't do anything for me, it seems to make things worse when it's on.

 

[Daisuke]

Maybe because I have prefetch enabled (related to AIO)?
I'm trying now the prefetch disabled and see what it does...

Edit: That does not do anything. When you have 8GB of RAM, the prefetch is set automatically to 0. This is what it showed on my boot screen:
vfs.zfs.prefetch_disable 0 -> 0

 

[Daisuke]

OK, I compared your settings with the default values in FreeNAS:
vfs.zfs.prefetch_disable: 0
vfs.zfs.prefetch_disable: 0 (default)

vfs.zfs.arc_max: 5074190464
vfs.zfs.arc_max: 7217082368 (default)

kern.ipc.nmbclusters: 10000
kern.ipc.nmbclusters: 50000 (default)

vfs.zfs.txg.write_limit_override: 1073741824
vfs.zfs.txg.write_limit_override: 0 (default)

vfs.zfs.vdev.min_pending: 1
vfs.zfs.vdev.min_pending: 4 (default)

vfs.zfs.vdev.max_pending: 1
vfs.zfs.vdev.max_pending: 10 (default)

kern.ipc.somaxconn: 384
kern.ipc.somaxconn: 128 (default)

kern.ipc.nsfbufs: 150000
kern.ipc.nsfbufs: 0 (default)

net.inet.tcp.inflight.enable: 1
net.inet.tcp.inflight.enable: 0 (default)

net.inet.tcp.recvbuf_auto: 1
net.inet.tcp.recvbuf_auto: 1 (default)

net.inet.tcp.sendbuf_auto: 1
net.inet.tcp.sendbuf_auto: 1 (default)

net.inet.tcp.delayed_ack: 0
net.inet.tcp.delayed_ack: 1 (default)

In my books those settings are all related to sysctl so they should be all added into Sysctls section. I don't know if it matters, but this is where I will place them if I decide to change any default values.

On default settings and AIO disabled I lose about 10MB/sec in speed.
Right now I have enabled:
Large RX support
sendfile(2)
Support DOS File Attributes
AIO

 

[ProtoSD]

I used the autotune feature in 8.2 Beta to get some different values, like arc_max.

Let me add some comments to your list below:

OK, I compared your settings with the default values in FreeNAS:

vfs.zfs.arc_max: 5074190464 [AUTOTUNE]
vfs.zfs.arc_max: 7217082368 (default)

kern.ipc.nmbclusters: 10000 [AUTOTUNE]
kern.ipc.nmbclusters: 50000 (default)

vfs.zfs.txg.write_limit_override: 1073741824
vfs.zfs.txg.write_limit_override: 0 (default)


kern.ipc.somaxconn: 384 [TCP LISTEN BUFFERS]
kern.ipc.somaxconn: 128 (default)

kern.ipc.nsfbufs: 150000 [THESE ARE BUFFERS FOR SENDFILE / CIFS]
kern.ipc.nsfbufs: 0 (default)

net.inet.tcp.inflight.enable: 1 [ONLY AFFECTS OUTGOING DATA, TCP BANDWIDTH DELAY]
net.inet.tcp.inflight.enable: 0 (default)

net.inet.tcp.delayed_ack: 0
net.inet.tcp.delayed_ack: 1 (default)

In my books those settings are all related to sysctl so they should be all added into Sysctls section. I don't know if it matters, but this is where I will place them if I decide to change any default values.

It doesn't make any difference, do a search and you'll see. I've tested it myself and they work the same in either place. It makes it easier for testing to see them all in one place, at least for me. There are a couple of other Autotuned settings I didn't see above. They must have been in the Sysctl section, let me put them here too.

Code:

net.inet.tcp.sendbuf_max	2097152	Generated by autotune	

kern.ipc.maxsockbuf	        2097152	Generated by autotune	

net.inet.tcp.recvbuf_max	2097152	Generated by autotune

 

[Daisuke]

OK, some settings tested:
vfs.zfs.prefetch_disable=1
vfs.zfs.txg.write_limit_override=1073741824
vfs.zfs.txg.timeout=5

The results are not fantastic, AIO Off 40MB/sec and AIO On 45MB/sec.
Reverting to stock values with AIO On gives me about 50MB/sec.

Edit: vfs.zfs.arc_max should be technically about 1.5 times bigger than actual memory, from what I read. I don't know if is worth to fiddle with it.

I'm going to try the kern.ipc.nsfbufs setting now with AIO Off (don't say I don't listen to you). :)

 

[ProtoSD]

I'm going to try the kern.ipc.nsfbufs setting now with AIO Off (don't say I don't listen to you). :)

Never ;)

I've been digging through all kinds of stuff that people have tried to see if it makes a difference. Nothing magic so far.

What size test file are using? I'm using 2 different 20GB datafiles.

Anyway, it's time for bed here. Hope you have better luck.

 

[Daisuke]

I'm using 3 files: 4GB, 10GB file and 14GB.
The kern.ipc.nsfbufs setting did not do anything fantastic, I see no difference. I'm done fiddling... Obviously something changed drastically from 8.0.2 to 8.0.4 version. I'm not sure I want to revert back to 8.0.2, I'm afraid about my data.

 

[arryo]

How do you add cache disk? I look every where in the webui but I couldn't find the cache option

 

[Daisuke]

Create a ZFS RaidZ2 volume, you will see the option to add the SSD disk as cache.

It is explained into documentation.

 

[arryo]

thank you. I thought I could just add a cache drive to an existing pool from GUI. I guess I have to do command line way

 

[Daisuke]

You can do that also, it is called Extend a volume.

 

[Antioch18]

Hi, I am considering setting up a FreeNAS box with the same set of hardware you have in your build. However, I was reading through your thread and noticed that you took a SEVERE performance hit using a newer version of FreeNAS. Were you ever able to fix this? Did you file a bug against FreeNAS (it seems like they changed something that impacted IO on RAIDZ).

Thanks for the follow up.

 

[Daisuke]

Personally, I don't really care anymore about the performance loss. At 50MB/sec, you transfer a large file very fast. I would rather benefit from the new features available into new software.

 

[Antioch18]

Another quick question:
Does your board support 3TB+ HDs (I believe that requires UEFI BIOS).

Thanks,
Antioch