Hey Guys,
I've been using Freenas at home for at least 5 years and it's been bascially flawless. I am a sysadmin put in charge of setting up storage quite often so I decided to spool up a couple servers at work with the intention of getting rid of our dodgy and inefficient windows servers.
I have two identical server/array setups one at our primary site and one at our secondary site, everything went great as I expected with the install and configuration. I have snapshots/replication and all the shares setup and I'm super happy in that sense.
The only issue is the very poor network performance, CIFS/NFS/FTP etc,. all top out at 40-45MB/s regardless of anything I do (raidz1/raidz2/raidz3/mirror/single disk/zil/no zil/tunables etc,.)
Here is my setup (I have two of these)
--
OS - FreeNAS-9.3-STABLE-201503071634
Host - Dell R510 w/ H700 HW raid (for OS)
- 128GB ECC Ram
Network - Intel X520 10GBe (in failover config)
HBA - LSI 9300-16e
Array - SuperMicro SC847J x 2
Drives - 45 x 4TB drives (mix of WD REDs / Seagate NAS drives)
l2arc - 4 x 512GB SSD (Samsung 850 Pro)
ZIL - 2 x 200GB SSD (Intel S3700)
Zpool config ~ three 12 disk raidz3 @ 130TB Raw / 92TB Avail
--
I've connected them both to AD and they are using windows permissions successfully. I haven't changed a whole lot, it's running SMB3 max, host lookups off, local master off, time server off (on CIFS). What is interesting is raidz3, raidz2, raidz1, mirror, single disk, zil, no zil etc,. have zero affect on performance.. all configurations result in almost identical speeds. I've tried over a dozen sysctls (temporarily) and they all had basically no affect.
So here is the kicker, when I copy from different sources to freenas it breaks through this ceiling. For example:
Fileserver 1 - Initiate copy, transfers at 40MB/s, open up a second tranfer on same server and now both files transfer at 20MB/s. open up two more and they are all 10MB/S.
Fileserver 2 - While first copy is still happening I transfer a file and it hits 40MB/s
Fileserver 3 - While first two copies are happenign I start a transfer and it hits 40MB/s!
Basically through one server/host I can only get a max of 40-45MB/s to Freenas regardless of any tuning or settings, this happens on 10G and 1G network adapters in failover or standalone configuration as well.
Here is a picture of my network interface, You can see the big "steps" where I added a new source, it looks like each one is limited to about 300-400mbit.
I've been using Freenas at home for at least 5 years and it's been bascially flawless. I am a sysadmin put in charge of setting up storage quite often so I decided to spool up a couple servers at work with the intention of getting rid of our dodgy and inefficient windows servers.
I have two identical server/array setups one at our primary site and one at our secondary site, everything went great as I expected with the install and configuration. I have snapshots/replication and all the shares setup and I'm super happy in that sense.
The only issue is the very poor network performance, CIFS/NFS/FTP etc,. all top out at 40-45MB/s regardless of anything I do (raidz1/raidz2/raidz3/mirror/single disk/zil/no zil/tunables etc,.)
Here is my setup (I have two of these)
--
OS - FreeNAS-9.3-STABLE-201503071634
Host - Dell R510 w/ H700 HW raid (for OS)
- 128GB ECC Ram
Network - Intel X520 10GBe (in failover config)
HBA - LSI 9300-16e
Array - SuperMicro SC847J x 2
Drives - 45 x 4TB drives (mix of WD REDs / Seagate NAS drives)
l2arc - 4 x 512GB SSD (Samsung 850 Pro)
ZIL - 2 x 200GB SSD (Intel S3700)
Zpool config ~ three 12 disk raidz3 @ 130TB Raw / 92TB Avail
--
I've connected them both to AD and they are using windows permissions successfully. I haven't changed a whole lot, it's running SMB3 max, host lookups off, local master off, time server off (on CIFS). What is interesting is raidz3, raidz2, raidz1, mirror, single disk, zil, no zil etc,. have zero affect on performance.. all configurations result in almost identical speeds. I've tried over a dozen sysctls (temporarily) and they all had basically no affect.
So here is the kicker, when I copy from different sources to freenas it breaks through this ceiling. For example:
Fileserver 1 - Initiate copy, transfers at 40MB/s, open up a second tranfer on same server and now both files transfer at 20MB/s. open up two more and they are all 10MB/S.
Fileserver 2 - While first copy is still happening I transfer a file and it hits 40MB/s
Fileserver 3 - While first two copies are happenign I start a transfer and it hits 40MB/s!
Basically through one server/host I can only get a max of 40-45MB/s to Freenas regardless of any tuning or settings, this happens on 10G and 1G network adapters in failover or standalone configuration as well.
Here is a picture of my network interface, You can see the big "steps" where I added a new source, it looks like each one is limited to about 300-400mbit.