Need help finding cause

[orsonb]

I'm having a problem with video playback over the network. I'm streaming from my FreeNAS 9.3 box to a MacMini over a gigabit network. Streaming only needs 5 MB/s, which is well below the 50 MB/s I see when doing straight file transfers. Every once in a while the playback will freeze. I'm trying to find out where the bottleneck is. When I look at the reporting graphs I see a gap in disk access at the time of the freeze, included. All 10 drives show the same graph. The memory utilization shows no noticeable change. The CPU spikes, but is still well below the 400% max. The logs I can find show no useable messages at the time of the freeze.

If a disk is dying, how do I find the message that will tell me which disk?
If it isn't a dying disk, how do I figure out what is causing the problem?

 

[Yatti420]

First off what are the specs of your freenas system??If you go storage >> click on pool.. at bottom pick volume status.. you can see read/write/checksum errors per drive.. Also can do smartctl -a /dev/ada# to look at smart data.. Need more info what protocols your using.. SMB? Apple sharing protocol? Is your macmini overheating etc etc etc..

 

[m0nkey_]

Suggest you read the forum rules (the link at the top in red) and include your hardware specifications and configuration detail. You're more likely to get help this way.

 

[depasseg]

Streaming only needs 5 MB/s, which is well below the 50 MB/s I see when doing straight file transfers.

What activity was going on during the network graph (streaming or file transfer)? Because that was only showing 5MB/s (50Mbits/S).

 

[joeschmuck]

So it doesn't appear to be a problem based on the information provided. And I echo the previous response to post your system specs and FreeNAS version. Getting hard drive info is easy enough and you should have frequent SMART testing being run, assuming you set FreeNAS up correctly.

 

[orsonb]

I thought I had edited my signature to include the specs. 9.3.1-STABLE, i3-3220T CPU @ 2.80GHz w/ 16GB, 10x3TB Raidz2
Yatti420, I use AFP.
Depasseg, I was streaming a video. That was all I was doing. The comment about file transfers was to indicate I was nowhere near the bandwidth limit.
Yatti420, the protocal was AFP. Also running smartctl for each drive revealed 1 drive with this result:

Code:

ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       21
  3 Spin_Up_Time            0x0027   222   159   021    Pre-fail  Always       -       7858
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       287
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       2
  7 Seek_Error_Rate         0x002f   100   253   051    Pre-fail  Always       -       0
  9 Power_On_Hours          0x0032   074   074   000    Old_age   Always       -       19193
10 Spin_Retry_Count        0x0033   100   100   051    Pre-fail  Always       -       0
11 Calibration_Retry_Count 0x0032   100   100   051    Old_age   Always       -       0
12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       283
184 End-to-End_Error        0x0033   100   100   097    Pre-fail  Always       -       0
187 Reported_Uncorrect      0x0032   100   092   000    Old_age   Always       -       13
188 Command_Timeout         0x0032   100   089   000    Old_age   Always       -       390847987809
190 Airflow_Temperature_Cel 0x0022   065   045   000    Old_age   Always       -       35
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       247
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       39
194 Temperature_Celsius     0x0022   117   097   000    Old_age   Always       -       35
195 Hardware_ECC_Recovered  0x0036   200   200   000    Old_age   Always       -       0
196 Reallocated_Event_Count 0x0032   198   198   000    Old_age   Always       -       2
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       6
200 Multi_Zone_Error_Rate   0x0009   100   253   051    Pre-fail  Offline      -       0

I'm going to replace that drive and see what happens.

 

[orsonb]

And now I see that the message does not retain my signature when created, but my signature is appended separately.

 

[joeschmuck]

How often do you run a Long SMART test? It would be helpful to see when the last time that was done and if it hadn't, then you should run it, you might find more problems. Still not sure how this figures in to your question.

 

[gpsguy]

Note for the future - users on mobile devices and/or using Tapatalk can't see signatures.

If you ask for help, we need to see the hardware and software details in the body of the message.

I thought I had edited my signature to include the specs

 

[Yatti420]

Note for the future - users on mobile devices and/or using Tapatalk can't see signatures.

If you ask for help, we need to see the hardware and software details in the body of the message.

Tapatalk wasn't working for me.. Use to use it alot..

 

[orsonb]

How often do you run a Long SMART test?

If it should be run periodically why isn't FreeNAS already scheduling it?

 

[depasseg]

If it should be run periodically why isn't FreeNAS already scheduling it?

By that logic, why isn't freenas sharing all our storage?

I agree that the setup wizard could guide the new setup in this regard, but I'm not sure everyone has the same testing needs.

 

[orsonb]

By that logic, why isn't freenas sharing all our storage? :)

I agree that the setup wizard could guide the new setup in this regard, but I'm not sure everyone has the same testing needs.

The implication was that it should be run on a periodic basis. I don't see how the logic gets back to sharing all storage automatically. I didn't have to schedule the automatic zpool scrub when I created the pool, but there it is. If it should be done the same way a scrub should, why isn't FreeNAS doing it for me?

 

[m0nkey_]

The implication was that it should be run on a periodic basis. I don't see how the logic gets back to sharing all storage automatically. I didn't have to schedule the automatic zpool scrub when I created the pool, but there it is. If it should be done the same way a scrub should, why isn't FreeNAS doing it for me?

FreeNAS was never designed to hold your hand configuring it. It gives you the tools to set-up a very powerful file server. Do you expect Windows or Linux to set-up scrubs and SMARTS tests for you? Probably not, so why should FreeNAS do it.

 

[orsonb]

FreeNAS was never designed to hold your hand configuring it. It gives you the tools to set-up a very powerful file server. Do you expect Windows or Linux to set-up scrubs and SMARTS tests for you? Probably not, so why should FreeNAS do it.

If Windows or Linux was running ZFS I would expect them to setup periodic Scrubs for me. If SMARTS tests are that important, I would expect them to do them as well.
FreeNAS lets me configure the automatic scrub, but it doesn't require me to explicitly create it. But where in the Web UI are SMARTS tests? I can see that the drives in a pool have SMARTS enabled, but so what. The implication is that a scrub is much more important than SMARTS tests.

So I again ask the question, if running SMARTS tests periodically are more important than scrubs, why doesn't FreeNAS do them for me?

 

[BigDave]

So I again ask the question, if running SMARTS tests periodically are more important than scrubs, why doesn't FreeNAS do them for me?

While I don't know the answer in regards to the WHY, what I might suggest is you put in a feature request
and ask the developers to add auto SMART testing to FreeNAS.
As an aside, If I paid for FreeNAS, I would expect it to do a few other things for sure, but it's provided at no cost...

 

[joeschmuck]

So I again ask the question, if running SMARTS tests periodically are more important than scrubs, why doesn't FreeNAS do them for me?

There are a few reasons I can think of to be honest...
1) Not all controllers can communicate with the drives properly in order to run a SMART test so doing this automatically would give the use a sense that things are all okay when in fact that is not the case, so manually setting this stuff is required.
2) FreeNAS was not designed as a simple NAS solution, it's a commercial product if you will (just free), being the testing ground for TrueNAS. From day 1 the use of it has been designed for a system administrator not the common home user. However over time the GUI has come along so nicely that people are expecting it to be all inclusive and do everything for the user. Well that isn't the case of course.
3) Every user has a requirement on how their system runs and SMART testing can impact how the system performs. Imagine if you will that you are a user with 50 hard drives. If the system automatically setup each drive to run a long test at 3:15 AM once a week, well that could be devastating to the network if the system was to be high availability because this would slow things down.

So the solution is to have the user read the User Manual and then set up what they desire.

As for Scrubs, who said either a Scrub or SMART test is more important, they are both important. Additionally a Scrub is a function of the file system so it's not based on how the controller communicates with the hard drives which makes it very easy to incorporate. Scrubs are typically once a month by default but keep in mind that your data is checked each time it reads it off the pool.

My recommendation is to establish a once a week SMART Long Test and then a daily SMART Short Test. Ensure your email account is setup correctly as well or the testing does you no good if your system can't tell you about it. And to be honest, I too wish it were a default setup or maybe a step by step setup during installation but as of right now it isn't. You can't argue too much for a free product that kicks butt compared to an expensive commercial NAS solution.

 

[pschatz100]

I'm having a problem with video playback over the network. I'm streaming from my FreeNAS 9.3 box to a MacMini over a gigabit network. Streaming only needs 5 MB/s, which is well below the 50 MB/s I see when doing straight file transfers. Every once in a while the playback will freeze. I'm trying to find out where the bottleneck is. When I look at the reporting graphs I see a gap in disk access at the time of the freeze, included. All 10 drives show the same graph. The memory utilization shows no noticeable change. The CPU spikes, but is still well below the 400% max. The logs I can find show no useable messages at the time of the freeze.

Pardon me, but I thought the OP asked about freezes during video playback...

To the OP: I still don't see complete hardware specs. I presume your core i3 is plugged into a motherboard of some sort? Also, what format video are you streaming? How? What client are you using on the Mac mini? There are lots of places where a bottleneck could occur - most of them not related to FreeNAS itself.

I would think that with a 10 disk Raidz2 configuration, if you had a bad disk the system would report a degraded pool. Also, I don't see why one bad disk in the pool would affect the performance of the pool.

Back to square one...

 

[toadman]

Agreed, it's too early to assume the issue is Freenas. (On the point about a bad disk, it can be an issue. I've had a "bad" disk that didn't report bad, but would take forever to respond. So the pool showed fine, but every I/O was painfully slow. This can be debugged, but I doubt it's the issue as the entire pool would be effectively too slow to do anything.)

When playback issues happen I would think client software, server (media) software, config between those, and then network before I thought Freenas as the issue. (I'm not saying it's not Freenas, just that it's not first in the priority order for debugging, at least for me.)

So how has all of the other stuff been eliminated as potential issues?

 

[orsonb]

Replacing the hard drive, as I said I would, has produced a resilver with multiple checksum errors. Leading me to think I actually had multiple bad drives. I'm waiting for another replacement drive to arrive.

When playback issues happen I would think client software, server (media) software, config between those, and then network before I thought Freenas as the issue. (I'm not saying it's not Freenas, just that it's not first in the priority order for debugging, at least for me.)

Copying the video to the MacMini and playing it directly off the local drive works perfectly, eliminating the MacMini, the video format, and the playback software. My router showed no other network traffic at the time. Leaving me to think it was a problem with the FreeNAS server. When I looked at the server, everything said it was working normally. So I asked here.

I still don't see complete hardware specs. I presume your core i3 is plugged into a motherboard of some sort?

The reported hardware is what FreeNAS reports in the System Information view. Unless you expect me to take the computer apart, I don't remember any of the other specs. Is there a command I can run to get more information?