What's my bottleneck? Help with Performance tuning 101

Status
Not open for further replies.

Mike Bruns

Dabbler
Joined
Dec 9, 2015
Messages
21
This is as much for my curiosity as anything, as I'm trying to improve my understanding of FreeNAS performance tuning. It's been a LONG time since I've been involved in Unix performance tuning, and I'm sure I'm missing something simple. Maybe this exercise can help others.

I'm replicating about 7TB of data from my production pool to a staging/temporary pool on a separate server. I'm going to rebuild my production pool & disks, so I need a place temporarily to store the data. Both servers are connected to the same unmanaged gigabit switch. I would have expected to max-out my network connection, but it appears to be limited (unless I'm misreading)

Sending Machine: Dell T310 - 5x6TB Disks - RaidZ2 -32GB RAM -
QLogic NetXtreme II BCM5716 1000Base-T Onboard Network Chipset
An older server I had around. Should be fine for normal use for 1-2 Plex users and 4-5 file sharing users. Should be idle outside of the replication task,
Receiving/Staging Machine: Dell T110 - 5x2TB Disks - RaidZ1 - 8GB Ram -
QLogic NetXtreme II BCM5716 1000Base-T Network Chipset
Yes, I know, RaidZ1 = Bad. 8GB = Bad. The data should reside on this server for a day or two max while I rebuild the primary. I also ran the full hard-drive burn-in scripts on the drives before I started. Should be idle outside of the replication tasks.​

Here's the data as I see it:
===============

System Info Sender:
Old CPU, not the fastest - 4 Virtual CPUs.​

upload_2016-1-19_20-58-29.png


System Info Receiver:
Old CPU, faster but still old - 4 Virtual CPUs​

upload_2016-1-19_20-59-3.png


CPU Sender:
2/3s idle. Is something single-threaded and clock-speed bound?​
upload_2016-1-19_20-47-51.png


CPU Receiver
Also 2/3s idle. Something single-threaded?​

upload_2016-1-19_20-48-9.png


Network Sender:
Almost exactly 400Mb/s
upload_2016-1-19_20-50-52.png



Network Receiver
Ditto
upload_2016-1-19_20-50-14.png


Disks Sender:
No frame of reference if this is fast or slow. Drives are White-Label WD-Reds.​
upload_2016-1-19_20-52-47.png



Disks Receiver:
Drives are 2-4 years old mixed. Seagate and WD Greens mostly.
upload_2016-1-19_20-53-20.png


Top Sender:
upload_2016-1-19_21-13-5.png



Top Receiver:
SSH Using all the CPU
 

Attachments

  • upload_2016-1-19_20-50-4.png
    upload_2016-1-19_20-50-4.png
    46.4 KB · Views: 432

depasseg

FreeNAS Replicant
Joined
Sep 16, 2014
Messages
2,874
Disable the SSH cipher and compression on the replication task and retry. SSH is single threaded and all your replication data travels across it.
 

Mike Bruns

Dabbler
Joined
Dec 9, 2015
Messages
21
That cleaned it up, SSH must have been CPU bound.

Question: If the replicated dataset is already LZ4 compressed on both sides, is ZFS smart enough to send the already-compressed data over the network? Or does he decompress the data then recompress it at the destination?
 

depasseg

FreeNAS Replicant
Joined
Sep 16, 2014
Messages
2,874

SweetAndLow

Sweet'NASty
Joined
Nov 6, 2013
Messages
6,421
That cleaned it up, SSH must have been CPU bound.

Question: If the replicated dataset is already LZ4 compressed on both sides, is ZFS smart enough to send the already-compressed data over the network? Or does he decompress the data then recompress it at the destination?
It decompresses it before sending. This is a feature request for replication that might come when we get compressed arc.
 

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,175
It decompresses it before sending. This is a feature request for replication that might come when we get compressed arc.
I believe zfs send/receive compression between compatible compression settings is on the to-do list for openZFS compression, once compressed ARC/L2ARC is done.
 

DaveY

Contributor
Joined
Dec 1, 2014
Messages
141
Disable the SSH cipher and compression on the replication task and retry. SSH is single threaded and all your replication data travels across it.

Sorry to revive an old thread, but I'm running into the same problem with replication (max out at 400mbps over gigabit link). How does one disable SSH cipher? I thought that's the point of SSH (secure shell) ? In the replication dialog, my optoins are standard and fast ciphers.
 

DaveY

Contributor
Joined
Dec 1, 2014
Messages
141
Disregard, looks like Disable option was available on older versions
 

Stux

MVP
Joined
Jun 2, 2016
Messages
4,367
Also, old CPUs don't have AES-NI, and this means AES encryption/decryption is MUCH more CPU intensive.
 
Status
Not open for further replies.
Top