Freenas very slow reads problem ?

[Black Ninja]

I've build Freenas after reading a lot about hardware recommendation and follow all advices from jgreco and cyberjock. My system is:

CHASSIS: Supermicro CSE-836TQ-R800B ( Direct attached backplane)
MB: Supermicro X9SCM-F
CPU: Xeon 1230 v2
RAM: 32 GB ECC Kingston ( from approved list)
HBA: 2xLSI9211 flashed in IT mode (with p16 Firmware)
HDD: 7x2TB HGST and 1 Intel SSD 530 120GB
FreeNAS-9.3-STABLE


I tried every combination , but I can find why I am getting Very fast Write speed and Very slow on read. I tried every Raid setup and copy over the network but to simplify my test and possible network or Raidz overhead I just created 2 pools ans start copy file between then with ssh and cp command.
Pool 1 is SSD single vdev and another pool 3x2tb hdd in RAID0(stripe).

As you can see on the picture when I copy from SSD to HDD ( RAID0) it's fine all hdd are writing with their max speed(over 110mb/s each) , but after the when I send files from HDD ( raid0) to the SSD they can only provide (35mb/s each) struggling. These HDD are capable of read/write over 110mb/s combine they should provide stream of around 300mb/s

Any ideas will be appreciated , I tried everything I can think of.

P.S. All files in copying process are large 3-8gb mkv files of total around 90GB to make sure the cache was flushed.

 

[SweetAndLow]

See how when you are writing to the ssd it's very consistent. That means you have a bottle neck there. I think you have found thewwrite speed of your ssd. Which would look like slow reads from your stripped pool. If you want to get a simple idea if performance use dd to write directly to disk from Dev/zero or read from disk to Dev/null.This will get you much better results. PPerformance is hard and had lots of variables.

 

[Black Ninja]

I am not sure where is the bottleneck , my ssd can do 500/mb/s Read or Write. 3x2tb in raid0 should provide 3 time the speed , not barely make speed of single drive. Something is not right here despite my efforts.

 

[enemy85]

The problem is the SSD. As u can see the real write speed of the ssd is NOT 500mb/s

http://ssd.userbenchmark.com/Compare/Samsung-840-Evo-120GB-vs-Intel-530-Series-120GB/1886vs1934

EDIT.
From here it seems to be stuck between 125-135 mb/s, more or less the results u are having

 

[Black Ninja]

OMG. You are absolutely right. I don't believe this. I guess my SSD can do 500 read and write only on synthetic bench , like ATTO which I often use even now as we speak. I will use mechanical HDD pool of few hdd in raid0 to test other pools speed by copying files between them. I don't trust these synthetic test commands like dd Dev/zero. By the way at least read can do close to 500mb. How could I been so stupid. Thanks a lot again, can I ask you what you are getting from your pool (5x 3TB WD30EFRX RED (RAIDZ2 setup)) for read/write mb/s ?

 

[enemy85]

Thanks a lot again, can I ask you what you are getting from your pool (5x 3TB WD30EFRX RED (RAIDZ2 setup)) for read/write mb/s ?

My results, with COMPRESSION OFF, are:

dd if=/dev/zero of=tmp.dat bs=2048k count=10k
21474836480 bytes transferred in 65.076788 secs (329992262 bytes/sec)
dd if=tmp.dat of=/dev/null bs=2048k count=10k
21474836480 bytes transferred in 60.500421 secs (354953506 bytes/sec)

I can't provide u any real results, because unfortunately the NAS is used as backup solutions of pc without SSD, so i'm nottlenecked by the mechanical hdd they use.

 

[Black Ninja]

So is that represent around 320-350mb/s read/write ? The Compression from my understanding lz4 in particular according to the freenas manuals should make the speed better despite the common preseption about compression , no ?

 

[enemy85]

So is that represent around 320-350mb/s read/write ?

yes, it's the pool read/write speed

The Compression from my understanding lz4 in particular according to the freenas manuals should make the speed better despite the common preseption about compression , no ?

With compression on you would have inaccurate results because that test write "zeroes" to the disk and zeroes are highly compressible. In general compression on helps to improve the pool transfer speeds and to save space

 

[SweetAndLow]

OMG. You are absolutely right. I don't believe this. I guess my SSD can do 500 read and write only on synthetic bench , like ATTO which I often use even now as we speak. I will use mechanical HDD pool of few hdd in raid0 to test other pools speed by copying files between them. I don't trust these synthetic test commands like dd Dev/zero. By the way at least read can do close to 500mb. How could I been so stupid. Thanks a lot again, can I ask you what you are getting from your pool (5x 3TB WD30EFRX RED (RAIDZ2 setup)) for read/write mb/s ?

did isn't a synthetic test command imo, it's a tool used to read and write data. If you want reliable benchmarks you need to be using it otherwise your results are inaccurate. Even with it people could argue your test is inaccurate but you at least get close.

 

[Black Ninja]

Honestly , I never understood how the "dd if=/dev/zero...." command actually works and what parameters to put in order to give me sequential speed reads equal to copy large files.

 

[SweetAndLow]

write test: dd if=/dev/zero of=/mnt/tank/file.data bs=1024k count=25k

That will get you write speeds. It read the data from /dev/zero which is a device that just has zeros in it and won't be your bottle neck. A couple things to keep in mind is compression will give you invalid results and writing data smaller than your total ram will give you invalid results.

read test: dd if=/mnt/tank/file.dat of=/dev/null bs=1024k count=25k

that should get you read speeds if you have a file on your system that is named file.dat. /dev/null is just a system device that does nothing and "of" is short for outfile and that is where it writes to so you are not bottled necked on the dest write location.

 

[Black Ninja]

Thanks you very much for explaining. I the compression is what made me gave up the first time , it gave me some 5000 mb/s unreal results and the file was not even registered to exist on the disk , despite was big file it made. I turn off the compression and now the results are much different. Now it shows that my SSD can Write 407mb/s and Read: 450 mb/s.

I will agree on the read results , but I disagree on the Write results. As you can see from above posts it turns my ssd can Write about 130mb/s. not over 400 mb/s as I also believed based on my previous test with this drive in Windows with ATTO.

So why the write results are not accurate ?

 

[SweetAndLow]

My commands I gave you should have made a 25GB file that file needs to be bigger than the amount of ram you have, otherwise you are just testing your ram speed. How much ram do you have? Also you are using bits to describe your speeds and you should be using bytes(b=bits, B=bytes).

EDIT: OP says you have 32GB of memory so make your file that size or bigger.

dd if=/dev/zero of=file.dat bs=1024k count=40k

 

[Black Ninja]

This exactly what I was thinking. I did change to count=64k to make sure the cache is flushed. What I meant by mb/s was Megabytes per Second. Unfortunately there is not really a standard how should be typed for it's meaning , even if there was everybody still type it the way he feel about it. I should be more specific like mbytes/sec or something like that.

I am not sure but I think maybe the SSD controller is using it's own cache in this test as well and that's why it gave me that result much higher result, perhaps that's why my ATTO benchmark has the same result , unless you put a very large transfer ( which you can monitor in real time) you might end up just testing your SSD cache memory like you said.

I like this command , but I wonder is there something I can use to monitor the transfer in a real time while it's working ?

 

[SweetAndLow]

There is a standard b==bits and B==Bytes, capital letter vs lower case.

http://en.wikipedia.org/wiki/IEC_80000-13

 

[Black Ninja]

I see there is a standard , but most people don't follow it anyway. I took a look I don't see where it says "mb/s" (I wrote early) stays for megabits instead of megabytes ?!
According to what you send me should be MiB for megabits ,but caps or lower case it doesn't say matters ?!
I seen it mostly like Mbps for megabits most officially used, but anyways it's off topic.
Any info is welcome.

 

[enemy85]

I turn off the compression and now the results are much different. Now it shows that my SSD can Write 407mb/s and Read: 450 mb/s.

That's really weird...as you already said, try with a bigger file and see if it changes.
Btw, are you sure you are running the test for your SSD and not for your other pool?
ah, just to better understand, in your first post you said you have:

"HDD: 7x2TB HGST and 1 Intel SSD 530 120GB"

but then you said:
"Pool 1 is SSD single vdev and another pool 3x2tb hdd in RAID0(stripe)"

so you are not using all your disks or i misunderstood something? Could you post the result of "zpool status" just to have a clear situation of your system?
Thanks

 

[Black Ninja]

I ran dd if=/dev/zero of=/mnt/ssdpool/fileto.data bs=1024k count=84k and then read the file and it's showing Write 407mb/s and Read: 450 mb/s.
But,
When I copy 87gb of large files from HDD (Raid0 with 3 drives) to the SSD with cp command in Shell then the SDD showed it's true colors : which was Write: 130mb/s.
Steady stream which seems it's the max capability of the drive. I guess that's real life performance , because on the test with dd command was showing 407mb/s which wasn't real. I think has something to do with SSD internal cache or something. When I test the drive in Windows with ATTO Benchmark it did show around 500mb/s read and Write , for smaller files it might be true , but you can't fill the drive capacity sustaining that speed.

Manufactures are deceiving us with high Read/Write speeds which are not sustainable , at least not Write speed in my case.

 

[SweetAndLow]

What's the read speed of your stripped pool.

 

[bestboy]

I wonder is there something I can use to monitor the transfer in a real time while it's working ?

Code:

zpool iostat -v 1  # 1 being the refresh interval in seconds

Code:

gstat -f da.p2     # assuming your devices are /dev/daX. replace with ada.p2 or whatever you got

Code:

arcstat.py 1       # 1 being the refresh interval in seconds