FreeNas write speed troubleshoot

[woods]

weird, had an error:



but the disk appears to be online. Performance is also the same. Did a scrub, no issues.

 

[woods]

have been doing more testing but I can't get the read speed past 750'ish MB/s

it's not the network, I've seen numbers up to 1.08Gb/s

For some reason, 10 disks in stripe mode, don't want to surpass the 750MB/s mark... I don't know why?


The HBA is LSI 9205-8i, surely that is not a bottleneck? I'm not sure if it's the drives themselves or the server (cpu/ram) but I'm going to try an SSD stripe to see if I can get past the cap.

 

[woods]

this is the ssd stripe,

speeds are about the same (ever so slight increase) but you can clearly see a trend, right? Every benchmark tool caps at the same levels! Is it a coincidence? I wonder what is going on.

Next step is to disable HT. Then start disabling cores.

Disabled HT, no change.

EDIT:

I created a file sequence video file of 48 000 KB per frame. That's a datarate of just below 10Gbps. I play these files from the SSD stripe and the HDD stripe. Both files play back at the EXACTLY equal limit of avg. 16 fps.

My read speeds achieve 10Gbps but all my reads are stuck at 7Gbps.

Below is the graph:
The first part of the graph is the SSD stripe playing the file in Resolve. The second part of the graphs is the HDD stripe playing the file in Resolve.




As you can see, the graph is virtually identical.

Also: when I reboot the NAS, and test the read speed, for about a minute it reaches up to 10Gb/s but then suddenly it drops to 7Gb/s and stays there forever. Also when I unplug the fiber cable from the NIC and plug it back in after some time, the speed is suddently increased for a little moment and then drops back to 7Gbps.

I think something weird is going on that has nothing to do with hardware limitations. I think there is a software issue. I might have to try a unix share and see if the problem is the same there.


EDIT:

Okay, turned on jumbo frames in FreeNAS and Windows (9000) and the same video file went from 18 fps avg. to 21 fps avg. Network monitor shows an increase from 7Gbps to 8Gbps. But unfortunately, there are some little "drops":




EDIT:

so I thiiiiiink I fixed it:



but that massive video file is still stuck at 22 fps... theoretically it should reach stable 25fps, probably I'll have to create a dataset with a record size that is optimal for this type of image sequence video.

This article basically fixed it: http://45drives.blogspot.com/2016/05/how-to-tune-nas-for-direct-from-server.html

EDIT:

so I threw in another 4x 3TB disks (my old client RAID that crashed more than a year ago - I know, unsafe, but I'm still in the testing phase, I'm not yet migrating my data to this pool).

I went for a two-way striped 5x raid z2 configuration. Sector size on the main dataset is 512KB and I did all the tuning as explained in the article above. The benchmarks indicate read speeds that comfortably saturate the 10Gbps limit. Although the write speeds are equally fast, there is something weird happening. It goes up to 10Gbps for a few seconds and then it massively drops to 0 for a few seconds before it goes up to 10Gbps again.

A few of my drives give sector errors. So I tried a full stripe with 10 legit disks. Both write/read are saturating the 10Gbps network. So I'm guessing the bad disks are causing the write drops. My data is currently on just a bunch of loose 4TB disks (I know, VERY unsafe, no doubles) but I can spare a couple of them to replace the sketchy drives. I'll try that today and see how it performs.

If I'm satisfied with the results, I'll copy all of my data to the pool, so now I have doubles. And I'll also start to upload the most vital pieces of data to google cloud if I can manage to create an account. So all in all, I'm slowly getting close to decent data management.

I'll also buy either a couple of 4TB or 6TB NAS drives to have lying around for when the time comes. So only when a drive fails, I'll install the new disks and order a new one. By doing this, I'm effectively creating a scenario in which my future drives won't be failing in batches, I read somewhere this is good practice.

Here is an image of the benchmark tests so far:




UPDATE:

okay, so the write spikes and drops were indeed caused by a bad disk. I had 3 disks with bad sectors so I wasn't sure which one was causing the big problems. Two disks have only a couple of bad sectors, but there was one with a lot of them. As a test, I put the disks again in a full stripe to test whether or not z2 was causing it. But nope, the test gave the same results, drops in write speed. FreeNas even froze at some point and I couldn't get into the GUI anymore. So I knew I had to pull that disk. Luckily I had another spare 4TB disk on hand so I replaced it.

These are the results of the repaired 2 stripe 6x z2:


Watching the graph, the write curve still has some very minor 100Mbps drops but mostly remains stable. I guess it might be a caching thing or perhaps the 2 disks with sketchy sectors are causing it. I'll report back when I replace them this week. Let's see if that fixed it.

I'm ordering 4TB Ironwolfs today. I'm not going to risk it and replace the other two disks with bad sectors and leave some new disks laying around. I'm suspecting another drive to fail quite soon.

If all keeps running well, I'll slowly migrate all of my data to the NAS in a couple of weeks and recycle the loose 4TB drives it was on into the NAS.
____

Jesus, f'ing Christ... this has been an intense couple of weeks. I hate computers. I hate them.

But can't do without them.