ZFS (RAIDZ-1) write: 22MB/s ...over 10GbE (SFP+) to 8x HGST 7200 with SM951 L2ARC . ¯\_(ツ)_/¯

[TrumanHW]

Networking Beginner -- I'll do my best to provide the relevant.

Once 10Gb was configured I tested R/W via AJA on MacOS
I'm clumsy at terminal or I'd have tried: Bonnie++ / DD would be preferable

Write - 230 MB/s
Read - 350 - 400 MB/s

...thus I felt confident it was configured 'correctly'. The initial transfer speed was faster ... but as time went on (I'm transferring 800 GB) it got slower and slower. :-(

NETWORKING
Layer 2: D-Link DXS-1210-12SC
Layer 3: Airport Extreme (Routing)


SOURCE
15in MacBook Pro, Retina (2015) with 2TB NVMe SSD
ATTO ThunderLink NT 2102 (Thunderbolt to 10GbE SFP+)


FreeNAS Machine

System: FreeNAS-11.2-U4.1

Dell T320 Tower (8-bay)
CPU . Intel Xeon CPU E5-2403
CPU 4 cores x 1.80GHz
RAM 48 GB DDR3 ECC 1333MHz
NIC Mellanox ConnectX-2 SFP+
HBA LSI H220 - SAS 9205-8i
HDD . 10TB 7200 RPM SAS - Ultrastar He10 IBM HGST HUH721010AL4200
L2ARC . 128GB - L2ARC -- SM951 PCI SSD (Apple SSUBX)

 

[JaimieV]

You haven't mentioned what network sharing protocol you're using - SMB, AFP, NFS, iSCSI? SMB (aka CIFS) particularly can take some tuning. There's a guide to 10gigE usage stickied over in the Networking forum, plus some 'slow?' posts over in Sharing.

Have you verified you're getting full 10gigE link speed? My switch wasn't reliable until I updated its firmware (Netgear XS708T) and kept dropping the link down to 1gig. You can check real network speed using iperf (install it via Homebrew on the Mac side, and it's shipped with FreeNAS).

You can check actual pool speed locally on the FreeNAS host, examples here alongside speed comparisons for different configs. RAIDZ1 isn't ideal for absolute speed, but you should be getting more than you are.

Last, a note that using RAIDZ (giving one redundant drive) with an 8x10TB pool is pretty risky. Ensure you have full backups for the almost inevitable second drive failure while resilvering at some point down the line.

 

[TrumanHW]

Also - bc I use this via Mac / Ubuntu ... I setup sharing for AFP, SMB and am working (hard) to try to make NFS work at the moment... (I was updating that when you posted it) ... but I'm at the stage of getting

mount_nfs: could not parse file system specification

mount: realpath /nfs: No such file or directory

..errors ... :-/



Re: RAIDZ-1 ... I'm actually using RAIDZ-2 .. sorry. :) I agree with you. (and own a data recovery company) ...

Thanks!

 

[TrumanHW]

Have you verified you're getting full 10gigE link speed? My switch wasn't reliable until I updated its firmware (Netgear XS708T) and kept dropping the link down to 1gig. You can check real network speed using iperf (install it via Homebrew on the Mac side, and it's shipped with FreeNAS).

Yes. That was what I'd hoped to convey by saying that my original AJA speed tests were:

WRITE: 250MB/s
READ: ~375MB/s

RAID is often slower than people expect/acknowledge.
But, I both TESTED it and got MUCH higher (as in, Gig-E CANNOT transfer a single file above 125MB/s)...
Generally ... the vol shouldn't be slower than the speed of the stripes of which it's comprised unless limited by the NIC/switch/etc.

I need to tune the L2ARC and will probably bump RAM to 128GB.

I will deal with the test examples later.

For now I'm going to wait for this stupid transfer to finish ... reboot, and check some transfers that are smaller ... as well as tuning my L2ARC.

REMAINING ISSUES:
Figure out why the xfer speed is 1/5 of Gig-E (which really makes it irrelevant as to what I'm connecting through)...
Tune L2
Get Windows 10 Connecting to FreeNAS

Thanks

 

[JaimieV]

You can't verify your network speed using file transfers, there are too many other things involved. AJA writes test files, so it's a cumulative test of everything in the path.

With 10gigE there's no point at all in trying to tune other things until you get your network speed validated.

Have a look at the output of ifconfig on both client and NAS. Find the right interface (by the IP address) and then the "media:" line. What you want to see is something very like "media: 10Gbase-T <full-duplex>" on the Mac, and "media: Ethernet autoselect (10Gbase-T <full-duplex,rxpause,txpause>)" on the NAS.

Test the network in isolation using iperf, it's on FreeNAS already and you need to install it on the Mac using Homebrew (install as per that page, then 'brew install iperf', then 'iperf -c 192.168.1.2' except with whatever the IP of the NAS is. Or whatever you do to install under Ubuntu.

On the NAS, run 'iperf -s' for it to listen, leave it going in its own terminal window.

On the client, you should get something like this:

Code:

Anaximander:~ jaimie$ iperf -c 10.0.0.3
------------------------------------------------------------
Client connecting to 10.0.0.3, TCP port 5001
TCP window size:  131 KByte (default)
------------------------------------------------------------
[  4] local 10.0.0.10 port 55943 connected with 10.0.0.3 port 5001
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.0 sec  11.5 GBytes  9.88 Gbits/sec

Obviously the nearly-10 Gbits/sec is what you want to see - if you do *then* you can dig into storage, L2ARC, sharing types and whatnot.

 

[TrumanHW]

If people provided Terminal examples/syntax like the below, people'd have a clear idea of what it did & how to tweak it.
Others (who already understood it) could easily say 'bullshit' if it's separated out as such ... with ease.


command Client IP Server streams Refresh Port No. Metric Transmit TTL
iperf -c 10.0.184.15 -P 1 -i 1 -P 5001 -f M -t 15 -T 1



command -- terminal cmd name
Client -- server / client
IP Server - IP address to connect to
streams -- Simultaneous data streams
Refresh -- Freq. to update
Port No -- Address
Metric -- Kb, KB, Mb, MB, etc. ...
Transmit -- Quantity to transfer
TTL -- IP Layer Option


[SkITTLES]



iperf -c 10.0.184.15 -P 1 -i 1 -P 5001 -f M -t 15 -T 1

[ 4 ] local 10.0.184.17 port 59382 connected with 10.0.184.15 port 5001

[ ID] Interval Transfer Bandwidth

[ 4] 0.0- 1.0 sec 193 MBytes 193 MBytes/sec
[ 4] 1.0- 2.0 sec 291 MBytes 291 MBytes/sec
[ 4] 2.0- 3.0 sec 275 MBytes 275 MBytes/sec
[ 4] 3.0- 4.0 sec 278 MBytes 278 MBytes/sec
[ 4] 4.0- 5.0 sec 269 MBytes 269 MBytes/sec
[ 4] 5.0- 6.0 sec 263 MBytes 263 MBytes/sec
[ 4] 6.0- 7.0 sec 196 MBytes 196 MBytes/sec
[ 4] 7.0- 8.0 sec 268 MBytes 268 MBytes/sec
[ 4] 8.0- 9.0 sec 206 MBytes 206 MBytes/sec
[ 4] 9.0-10.0 sec 278 MBytes 278 MBytes/sec
[ 4] 10.0-11.0 sec 272 MBytes 272 MBytes/sec
[ 4] 11.0-12.0 sec 261 MBytes 261 MBytes/sec
[ 4] 12.0-13.0 sec 190 MBytes 190 MBytes/sec
[ 4] 13.0-14.0 sec 251 MBytes 251 MBytes/sec
[ 4] 14.0-15.0 sec 270 MBytes 270 MBytes/sec
[ 4] 0.0-15.0 sec 3762 MBytes 251 MBytes/sec
Done

 

[TrumanHW]

The other things I still need help with though .... [if] i can test the speed from FreeNAS to Client ... which machine do I run iperf on to test that -- and how does the syntax differ ?? (People tend to overlook that ambiguity)..

But the other big problem I have is front to get NFS to work. :-o

On one thread someone said:

The NFS share is mounted at /mnt/nfs on the Mac client.
sudo mount -t nfs 192.168.1.2:/srv/nfs /mnt/nfs

WHAT does that guy mean??
/mnt/nfs ... is a 'location' (whether it exists in the filesystem or not) whereby he's going to
...mount the NFS share: 192.168.1.2:/srv/nfs /mnt/nfs ??

But what does the colon mean? Is it referring to a location on the server? On the Mac? What the F!?


Breaking this string apart...
sudo mount -t nfs 192.168.1.2:/srv/nfs /mnt/nfs

-t . - [external type]
IP address - [location of share]
:/srv/nfs - [WHAT does this refer to? Where? does it already exist?]
I'm assuming the shared location has a folder of /srv/nfs ..? Or do I make it?

/mnt/nfs
is this a wildcard? Can I put whatever I want there? If not, why does it require it? lol.



Then, more importantly, if you've tried these things and they STILL don't work -- wtf mate? What do I do then?

 

[JaimieV]

which machine do I run iperf on to test that -- and how does the syntax differ ?? (People tend to overlook that ambiguity)..

After you got nowhere with my original suggestions, I literally gave you the iperf command line to run on the NAS (iperf -s), an explanation of how to install iPerf on your client Mac and then the command line to run the test from there (iperf -c 192.168.1.2) - with a note to use the NAS's IP not the one I put in, and a quote of output on the client side to show what it actually looks like.

What you haven't done is say whether the iPerf trace you've pasted in are yours, or from some other example somewhere. If they're yours, they clearly show that something on your network between the client and the NAS is running at 2.5Gig. Could be a switch, could be a dodgy cable, could be the ethernet adapter on the Mac (only the latest Mini and the iMac Pro can come with a 10gigE port).

You're getting arsy at the people who you are asking for help, and you're not apparently even trying to learn anything along the way. Lol mate.

 

[TrumanHW]

THANK YOU!! I wish others were half as clear as you. Truly, thanks!

I have 2x ATTO ThunderLinks -- so I tried the second one to rule it out
(it was equally temperamental/variable in performance)

Transferring 900MB MP4 files
- Uploads: 120 - 200 MB/s
- D-loads: 300-500MB/s (20GB folders)

I used large files bc even NVMe's slow to ~100MB/s or less -- when copying small files...
i.e.: emails / chat transcripts / internet cache, etc.

 

[TrumanHW]

One down (macOS via NFS -- just in case I have to stick to one protocol) ...
I also have gotten Windows to mount it - but not read-write ... but I expect I'll have that solved soon too.

Thanks