MasterTacoChief
Explorer
- Joined
- Feb 20, 2017
- Messages
- 67
FreeNAS 11.0U4, system uptime 60 days, 3x300GB SAS 10k disks in a RAID-Z1 configuration + 240GB SAS SSD L2ARC. I have this dedicated to providing a NFS3 share to VMWare to run a variety of VMs. Early this morning one of the drives reported an increase in bad sectors and the performance dropped significantly. When I noticed it this morning, VMs using this store were running slowly, and the VCenter Network Appliance (also stored on this share) had corrupted itself (it appears to not work well with high latency).
I tried a variety of things, including offlining the disk that reported a bad sector and offlining the L2ARC, but nothing has increased the disk access via NFS. Strangely enough I can copy files to and from this array using SMB and get 200MB/s+ performance, so it seems like there's something in particular going on with NFS on this particular share.
I have another NFS share made up of 5x 4TB SATA disks on the same FreeNAS server and this share isn't having the same issues. Once I migrated the VMs off the bad share (very slowly) to this other share then disk access was back to normal for those VMs. There are no errors in the syslog in FreeNAS, nor any errors from VMWare.
Next I will try a restart of NFS, then if necessary a reboot from FreeNAS. I'd hope for better uptime though.
Below is the log of Disk I/O for the three disks in the array, showing a drop in performance when it all went downhill. da10 was the disk with reported bad sectors. You can see its performance go to 0 when I offlined, then return once I brought it back online. The spikes around 1800 were when I was copying files via SMB. The reads around this time were migrating VMs to another share.
I tried a variety of things, including offlining the disk that reported a bad sector and offlining the L2ARC, but nothing has increased the disk access via NFS. Strangely enough I can copy files to and from this array using SMB and get 200MB/s+ performance, so it seems like there's something in particular going on with NFS on this particular share.
I have another NFS share made up of 5x 4TB SATA disks on the same FreeNAS server and this share isn't having the same issues. Once I migrated the VMs off the bad share (very slowly) to this other share then disk access was back to normal for those VMs. There are no errors in the syslog in FreeNAS, nor any errors from VMWare.
Next I will try a restart of NFS, then if necessary a reboot from FreeNAS. I'd hope for better uptime though.
Below is the log of Disk I/O for the three disks in the array, showing a drop in performance when it all went downhill. da10 was the disk with reported bad sectors. You can see its performance go to 0 when I offlined, then return once I brought it back online. The spikes around 1800 were when I was copying files via SMB. The reads around this time were migrating VMs to another share.