Need some help with speed of array on GB and 10GB

[Roveer]

I recently re-purposed and older machine for FreeNAS. It's running, but I'm looking at SMB transfer speed.

It's an 8 drive 800GB/drive array that is SATA2 (3Gb/s) on a IBM M1015 flashed to IT. I also threw a 32gb SSD as a zil cache on it as well.

I did some DD testing and got these results: which if I'm reading correctly is 569MB/s? correct?

Connected via 1gb ethernet I'm seeing 90MB/s write speeds. I was hoping to get full saturation 100+. I know I'm nitpicking, but I'm trying get the best performance. In the next few days I'm going to re-build with a 10GB Nic and will be looking for fastest speeds I can get. Mostly on the write side as it's going to be used mostly as a backup / mirror server.

I'm going to be putting 8GB on this system (max for this MB) and trying to upgrade to a xeon X5460 cpu.

Not the fastest by any stretch, but not bad for a bunch of older stuff I had lying around.

Question. Is there anything I should be doing configuration wise to make this thing go as fast as possible? I turned off SMB logging. What else don't I yet know about? Been working with FreeNAS for all of 3 days now.

Thanks,

Roveer

 

[Nick2253]

Connected via 1gb ethernet I'm seeing 90MB/s write speeds. I was hoping to get full saturation 100+.

How are you testing this?

 

[Roveer]

How are you testing this?

That's what Windows 10 is reporting on a file copy.

 

[Nick2253]

So, you're using file copy to test the speeds? What is the HDD/SSD that the data is coming from? Are you writing from FreeNAS to your desktop's drive, or the other way around? 90MB/s may be the maximum you can get from the device you copying from/to.

 

[Roveer]

So, you're using file copy to test the speeds? What is the HDD/SSD that the data is coming from? Are you writing from FreeNAS to your desktop's drive, or the other way around? 90MB/s may be the maximum you can get from the device you copying from/to.

Data is coming from a SSD and when I copy to one of my other servers I get 100+ MB/s

 

[Roveer]

Correction...

Machine copying data is a Win7 machine with SSD.

When it copies (same file) to Server 2008 R2 I get 103-105 MB/s
When it copies (same file) to FreeNAS I get 86-87 MB/s.

The 2008 R2 server is much newer, more ram, faster drives etc so that could explain it, but when I DD test the FreeNAS I'm looking at 500 MB/s writes on the machine. Is it possible the slower processor and 4GB of memory is slowing me down? I'm probably answering my own question here, especially since we are talking about 20 MB/s but that 500+ number is really throwing me.

Roveer

 

[Nick2253]

Well, 4GB is half the minimum memory, and you have well over 4TB of hard drive space. It wouldn't surprise me if your hardware is limiting you. Don't forget that SMB is pretty CPU limited, so that might be exactly what you're running into. You could try NFS, which should give you better performance.

 

[tvsjr]

You're gonna have a bad time with 4GB RAM. As far as your dd tests, do you have the default lz4 compression turned on for the dataset? If so, writing highly-compressible zeroes to it doesn't tell you much about what the real world. You've also got an FSB-based processor there, which is known for being sub-optimal for FreeNAS.

In short... you might learn a bit with it, but you're going to want to do some serious upgrading before trying to use that system in any sort of production role.

 

[Roveer]

You're gonna have a bad time with 4GB RAM. As far as your dd tests, do you have the default lz4 compression turned on for the dataset? If so, writing highly-compressible zeroes to it doesn't tell you much about what the real world. You've also got an FSB-based processor there, which is known for being sub-optimal for FreeNAS.

In short... you might learn a bit with it, but you're going to want to do some serious upgrading before trying to use that system in any sort of production role.

Thanks for the comments. All part of the learning process. That was the purpose of building this system, to learn. I'll try turning lz4 off and test again. "FSB-Based Processor" Can you elaborate a bit more and share what might be a better choice for processors?

This setup will actually work in my current environment. My nightly data requirements are around 300GB which takes an hour or two but my hobbies now include building super fast systems Just finished a GB vpn system between sites and now I'm looking at super fast storage.

 

[Roveer]

A test with lz4 turned off shows 143MB/s (which sounds right for SATA2 3.0 GB/s). So I guess that's what I was looking for an answer to results that seemed way too good to be true. Now I want to build a super fast subsystem!!! Ahh, it's only money.

 

[tvsjr]

The FSB processors don't handle intense IO well, just by nature of the bus design. I'd be careful about your setup... FreeNAS tends to get pissy/unstable if you don't provide it enough memory. Until you get to 8GB, don't trust it too much.

As far as newer solutions, look at all the various build threads. It all depends how much you want to spend and how much power you want to burn. If you can handle a rack-mount chassis, I'm a big fan of buying used Supermicro gear from a trusted buyer. You get a system with server-class reliability, a nice E5 or two, lots of bays, etc. You can also throw lots of RAM at the box for relatively cheap from eBay (I can buy 8GB DDR3 modules for my system - see sig - for $20/ea.) You can buy a turnkey, used system with 36 bays for about $1800 (https://unixsurplus.com/collections...-2x-e5-2680-2-8ghz-192gb-2-port-10gbe-sfp-nic as an example... I've never bought from them, but others have with good success... www.theserverstore.com is another option). Throw drives in, load FreeNAS, and you're golden.

The general preference is to stay with something in the E3/E5 range, or the matching i3 processors if you don't need E3/E5 power (i3-6100 being very common, it seems).

You should probably hang out in the "Will It FreeNAS?" subforum a bit and see what people are building. You'll see plenty of good and bad ideas. And, of course, read through the posts in Resources.

 

[Roveer]

The FSB processors don't handle intense IO well, just by nature of the bus design. I'd be careful about your setup... FreeNAS tends to get pissy/unstable if you don't provide it enough memory. Until you get to 8GB, don't trust it too much.

As far as newer solutions, look at all the various build threads. It all depends how much you want to spend and how much power you want to burn. If you can handle a rack-mount chassis, I'm a big fan of buying used Supermicro gear from a trusted buyer. You get a system with server-class reliability, a nice E5 or two, lots of bays, etc. You can also throw lots of RAM at the box for relatively cheap from eBay (I can buy 8GB DDR3 modules for my system - see sig - for $20/ea.) You can buy a turnkey, used system with 36 bays for about $1800 (https://unixsurplus.com/collections...-2x-e5-2680-2-8ghz-192gb-2-port-10gbe-sfp-nic as an example... I've never bought from them, but others have with good success... www.theserverstore.com is another option). Throw drives in, load FreeNAS, and you're golden.

The general preference is to stay with something in the E3/E5 range, or the matching i3 processors if you don't need E3/E5 power (i3-6100 being very common, it seems).

You should probably hang out in the "Will It FreeNAS?" subforum a bit and see what people are building. You'll see plenty of good and bad ideas. And, of course, read through the posts in Resources.

Thanks for the tips. I'm just playing around at this point so data is not mission critical. Love looking at all that supermicro gear. Could have a ball putting together a real monster but my needs don't really support the cost. Will definitely hang out and continue to learn.

Want to understand the FSB thing a little better. Right now my FN is running on a Intel(R) Core(TM)2 Duo CPU E6750 @ 2.66GHz but I'm trying to get it upgraded to an Intel Xeon X5460 (that will supposedly work with my MB). Both of these are very much legacy. Would you say that i processors (i3,i5,i7) are much better performers and are not FSB limited. I'm assuming by FSB you are referring to "front side bus". Just trying to figure out what to stay away from as I move forward. I know how this is going to go. I'm going to obsess over FreeNAS and end up building something really great, so learning what not to do is as important as learning what to do. Not as easy as it sounds when you are not always using latest and greatest.

My most recent obsession were a pair of i7 pfSense boxes on either side of gig FIOS connections. Getting 100MB/s+ across the vpn, now I have to catch up with the storage and want to push 10GB on and off this box. All a balance of power and most importantly $$$.

As I'm writing this, my 8gb memory just arrived, so I guess that point is moot.

Roveer

 

[tvsjr]

Plenty of threads on it... just use the search. For example:
https://forums.freenas.org/index.php?threads/fsb-bottlenecks.46302/

Wiki lists all of the processors with FSBs. Core i-series is post-FSB, but your C2D and your X5460 are both FSB-based chips.
https://en.wikipedia.org/wiki/Front-side_bus

 

[Roveer]

Plenty of threads on it... just use the search. For example:
https://forums.freenas.org/index.php?threads/fsb-bottlenecks.46302/

Wiki lists all of the processors with FSBs. Core i-series is post-FSB, but your C2D and your X5460 are both FSB-based chips.
https://en.wikipedia.org/wiki/Front-side_bus

As much as I think I know, I'm always learning something new. Thanks for the information.

 

[Roveer]

I hit a wall with this configuration. The MB only has 1 8x slot so I won't be able to add 10GB NIC. I threw in the 8GB of memory (replacing 4gb), and I notice the web interface got zippier. I'll try and add the Xeon when it arrives and this will have to be a 1gb ethernet FreeNAS. It's just got too many old parts to keep trying to upgrade. I'm looking at some Supermicro or Dell stuff to possibly build a righteous FreeNAS in the future but considering that I'm only copying a couple hundred gig per night, this box will suffice for now.