Fractal Node 304
SeaSonic SSR-450RM (modular, 80plus Gold)
ASRock FM2A85X-ITX (7 internal SATA3 ports)
AMD A4-4000 (65W TDP)
2x4GB non-ECC Corsair PC3-10600 (1333MHz)
6x2TB WD Red
ZFS raidz2 (mostly empty)
booting from internal USB stick (had to keep USB 3.0 disabled due to crashes)
All testing using 100GB transfers via "dd if=/dev/zero of=... bs=2048k count=50k” and "dd if=... of=/dev/zero bs=2048k count=50k”, with compression disabled.
Most testing performed with rudimentary sysctl tuning:
Code:
net.inet.tcp.sendbuf_max 16777216
net.inet.tcp.recvbuf_max 16777216
kern.ipc.maxsockbuf 16777216
net.inet.tcp.sendbuf_auto 1
net.inet.tcp.recvbuf_auto 1
net.inet.tcp.sendbuf_inc 16384
net.inet.tcp.recvbuf_inc 524288
net.inet.tcp.inflight.enable 0
Later testing performed with powerd enabled, and more sysctl/tunable tuning after some research:
Code:
tunables:
kern.hz 250 Reduce kernel interrupt frequency for power savings
sysctl:
net.inet.tcp.recvbuf_max 2097152 Recommended for up to Gigabit Ethernet speeds.
net.inet.tcp.sendbuf_max 2097152 Recommended for up to Gigabit Ethernet speeds.
kern.ipc.maxsockbuf 2097152 Recommended for up to Gigabit Ethernet speeds.
net.inet.tcp.sendbuf_auto 1
net.inet.tcp.recvbuf_auto 1
net.inet.tcp.sendbuf_inc 65536 1/4 of what’s suggested for Gigabit Ethernet (doing WiFi here)
net.inet.tcp.recvbuf_inc 65536 1/4 of what’s suggested for Gigabit Ethernet (doing WiFi here)
net.inet.ip.forwarding 1
net.inet.ip.fastforwarding 1
net.inet.tcp.mssdflt 1460 MTU size
net.inet.tcp.rfc1323 1 Allow send/recvspace to exceed 64KB
net.inet.tcp.sendspace 411720 Congestion window: 80Mbps * 40ms
net.inet.tcp.recvspace 411720 Congestion window: 80Mbps * 40ms
net.inet.tcp.slowstart_flightsize 141 (sendspace / MTU) / 2 - Bet that the connection is good, ramp up quickly.
net.inet.tcp.delayed_ack 1
net.inet.tcp.delacktime 100 Piggyback ack’s onto the return packet in most cases.
net.inet.tcp.inflight.min 6144 Suggested by tuning(7)
net.inet.tcp.inflight.debug 0
net.inet.tcp.inflight.enable 1 Dynamically limit throughput to optimize buffer usage on variable speed connections (WiFi).
net.inet.tcp.hostcache.expire 3900 Don’t expire the cache right when a host does its hourly checkin.
Code:
GELI disk encryption disabled (for testing only - I want it on):
direct (no network):
write: 317.353848 secs (338342147 bytes/sec) = 322.67MiB/s
read: 347.829313 secs (308697911 bytes/sec) = 294.40MiB/s
Gigabit Ethernet + afp:
write: 1291.743512 secs (83123454 bytes/sec) = 79.27MiB/s (664.99mbps)
read: 1476.271611 secs (72733352 bytes/sec) = 69.36MiB/s (581.87mbps)
WiFi + afp (802.11n, channel 149, RSSI -32 to -37, transmit rate 216-300):
write: 5825.926545 secs (18430404 bytes/sec) = 17.57MiB/s (147.44mbps)
read: 6404.202416 secs (16766207 bytes/sec) = 15.99MiB/s (134.13mbps)
final settings (powerd, more sysctls), via Wifi + afp:
write: 4920.086985 secs (21823635 bytes/sec) = 20.81MiB/s (174.59mbps)
read: 6226.676063 secs (17244222 bytes/sec) = 16.45MiB/s (137.95mbps)
GELI disk encryption enabled:
direct (no network):
write: 1137.761125 secs (94373221 bytes/sec) = 90.00MiB/s
read: 704.188801 secs (152479253 bytes/sec) = 145.42MiB/s
Gigabit Ethernet + afp:
write: 2032.278082 secs (52834395 bytes/sec) = 50.38MiB/s (422.68mbps)
read: 1615.509503 secs (66464593 bytes/sec) = 63.38MiB/s (531.72mbps)
final settings (powerd, more sysctls), via Wifi + afp:
write: 10753.461960 secs (9985080 bytes/sec) = 9.52MiB/s (79.88mbps)
read: 6542.687941 secs (16411326 bytes/sec) = 15.65MiB/s (131.29mbps)
So, regrettably, the maximum throughput in my case has very little to do with reality. I want full disk encryption, and I’ll primarily be using the NAS via WiFi (our household is nearly free of cluttery cables).
Performance is seriously impacted by all factors shown above: underpowered CPU for encryption and compression (though hardware encryption was found and used), slow Gigabit ethernet, even slower 802.11n WiFi (with a very open 5GHz spectrum and while sitting in the same room as the base station)...
But ultimately, I’m getting what I wanted: a storage solution that’s hardened against bit rot, and reasonably fast, large, and WiFi-accessible.
By the way, power consumption is reported by my Kill-a-Watt as follows:
Code:
without powerd:
~40-42 watts idle, drives spun down
~53-55 watts idle, no drive spin down
~61-67 watts under full load (no encryption, no compression)
with powerd:
~35 watts idle, drives spun down, powerd enabled
~50 watts idle, no drive spin down
~59-69 watts under full load (no encryption, no compression)
~59-71 watts under full load (with encryption, no compression)
~55-74 watts under full load (with encryption, compression)