Disk thrash causing video stutter.

[diskdiddler]

Hi all,

8gb HP Microserver N54L (2.2ghz AMD dual core)
6x5TB Toshiba 7200 drives
Recently upgraded from 1.5ghz N40L - it's faster.


Anyhow, I use Locate32 under Windows to replace Windows file search, since it's awful.
I'm watching a video and locate32 does a database update, the way it does this is basically find every single file and get the properties then index them in a database. It gives you a cached search like the Mac Finder thing.

It caused video stutter so I set my VLC player to have a 15 second buffer (that's huge, default is 300ms, I set it to 15,000) - this still stuttered.
Will adding 16gb of RAM make the server have better caching in memory of the entire file structure? What _EXACTLY_ does more memory do? People keep saying how important it is but I'm curious what it actually offers?

Like to be able to watch video while disk access occurs or even set a path to have a higher file serving priority?

 

[krikboh]

Considering the recommendation of 1 GB of RAM per TB of storage you should definitely consider additional RAM.

 

[diskdiddler]

I plan to very shortly but not a single person has said precisely how this RAM works better, there is limitations to what a hard disk can do with it's actuator, the amount of them in the array, the CPU and the write speed of the disk itself.
I can peak now with the new CPU at up to 90MB/s writes, which I'm happy with.

Considering my issue here is basically needing the disk head in 2 physical locations at once, I'm curious if it's going to pre-cache ahead 500mb or 1gb chunks at a time to server up to my SMB session, is it going to keep an index in memory of all files on the filesystem (at least what they are called etc? Like logging the disk in Ztree)
People continue to talk about needing X GB for Y TB but some better technical details would be useful in diagnosing performance issues.

 

[krikboh]

The whole point of FreeNAS using this much RAM in a NAS is for cache. You should do some reading on ZFS ARC and L2ARC.

http://www.zfsbuild.com/2010/04/15/explanation-of-arc-and-l2arc/

 

[cyberjock]

What kriskboh said. But you may need 32GB of RAM for a pool that size.

 

[c32767a]

Hi all,
It caused video stutter so I set my VLC player to have a 15 second buffer (that's huge, default is 300ms, I set it to 15,000) - this still stuttered.
Will adding 16gb of RAM make the server have better caching in memory of the entire file structure? What _EXACTLY_ does more memory do? People keep saying how important it is but I'm curious what it actually offers?

Like to be able to watch video while disk access occurs or even set a path to have a higher file serving priority?

Fundamentally, it offers more space for the Cache. ZFS does do some prefetching based on read activity, but in order for the cache to be useful, you need to be accessing the same data repeatedly. In your case, while you're watching a video, Locate 32 goes out and accesses every file on the filesystem. Even if it just touches the metadata, this is pretty much the worst case scenario for a cache algorithm. Even the readahead algorithm may do more harm than good if you're just hitting the metadata and file headers.

Adding more memory just increases the likelihood that everything locate 32 accesses will fit in RAM. If it's just hitting metadata, you might be able to get away with it. If it's actually reading files, you're going to have tons of cache misses, which translates into disk reads, which translates into stutter on your video.

There are lots of internal counters to analyze exactly what's going on with ZFS, but it's typically easier to solve the problem by throwing additional resources at it, since RAM is cheap and most of the problems that show up here can be solved if you make the cache large enough.

 

[Knowltey]

It caused video stutter so I set my VLC player to have a 15 second buffer (that's huge, default is 300ms, I set it to 15,000)

It may be huge in comparison to the default, but VLCs buffer is insufficient as well. 300 leaves barely any breathing room in the event of a network slowdown or minor hiccup. I always have to increase the VLC buffer if I plan to be pulling files over the network into it, and that's with servers that don't have any performance issues. So don't get too wrapped up in that you had to increase the buffer, they set it way too low to begin with.

Just because something is deafult doesn't mean it's actually an ideal setting. Defaults are generally more often than not a "will always work even in worst case scenario" type of settings. Another great example is how DD-WRT's default ACK Timing setting is over a mile.

 

[diskdiddler]

You're right 300ms is pretty small, mind you a reliable LAN (not WAN) link, really shouldn't ever screw up on this.
None the less, we're talking about 15000ms or fifteen full seconds now.
FIFTEEN SECONDS.

That's an exorbitantly bigger buffer than most online streaming services, let alone a local LAN.
I've ordered my 16GB ECC, fingers crossed it's HP Microserver compatible, should arrive sometime late next week.

 

[Knowltey]

Yeah, fifteen seconds is entirely too long, I usually only have to increase mine up to two or three seconds, but put it at 10 just as an extra buffer since it doesn't take up any noticeable extra time to initially build that buffer, so might as well. Since I'm in an apartment the WiFi clients usually will have times where 3 or 4 isn't quite enough. Wired clients are generally fine with as low as a second, but the buffer is still good since you never know when the client itself gets some sort of hiccup in it's connection or decides to do something else heavy with it's connection that would be too much for a 300ms buffer.

More RAM should definitely help, but if the root cause is still the disks not performing well enough you should definitely look into getting that resolved as well.

 

[diskdiddler]

Well like I said, I'm working with the disk by writing to it, perhaps Acronis is doing a backup and it's writing 20 / 30 / 40mb/s - I don't know
Windows, somehow - would be smart enough to send me enough video data to not require buffering, while still accepting the acronis backup from the laptop or whatever.
FreeNAS does not seem able to do this.
Same with When I initiate a ZTree (XTree) full disk log of the drive, I understand this is going to take a while because it's literally reading / accessing every file on the disk - - mind you it's only the name / header / size stuff. This in terms of size is probably only 2 to 10MB of data but requires the disk heads to go everywhere.
I would like to think the disk index would remain (mostly) in memory though with more ram in the system.

 

[Knowltey]

Yeah, if you're doing it over CIFS to the FreeNAS server it's all just files being accessed, it doesn't look at what type of file is being accessed, so everything will have the same priority.

 

[IanWorthington]

Hmm. I set up a 11x4tb z3 system with only the recommended minimum 8GB of RAM to see how it goes and find it can stream 1080 video over a cifs share without any noticeable problem at all.

Reporting says:

so I haven't been in any rush to add more memory. For sequential reading caching shouldn't really make any difference.

The only time I had stuttering was when I tried it over a wifi standard that wasn't up to the job.

 

[diskdiddler]

I don't know what you're doing differently but ALL the lines on my graph are directly bouncing at the top virtually within 15 minutes of booting, with free ram around 500mb at most - with 8gb. That's ZFS2 or whatever it's called.

Also just FYI - the thing is, I'm talking about disk thrash in a second process accessing the filesystem.

 

[c32767a]

I don't know what you're doing differently but ALL the lines on my graph are directly bouncing at the top virtually within 15 minutes of booting, with free ram around 500mb at most - with 8gb. That's ZFS2 or whatever it's called.

Also just FYI - the thing is, I'm talking about disk thrash in a second process accessing the filesystem.

So you can reproduce the behavior at will?

I'm curious if this change has any effect:

zfs set atime=off volume

eg: http://prefetch.net/blog/index.php/...access-time-atime-updates-on-zfs-file-system/

 

[esamett]

I had VLC (default settings) stutter at 1080. 720 much smoother.

2x10 Z2 array, 8, then 16 GB ram.