Damn, I'm getting the same problem as everyone here and it doesn't seem to be an easy fix to it... I'd hate to have to go back to Freenas7 but if that's what I have to do... i guess that's what I gotta do... Been playing with Freenas8 for awhile now... speedy but once I put up multiple transfers... it hangs.... sigh...
-
Important Announcement for the TrueNAS Community.
The TrueNAS Community has now been moved. This forum will now become READ-ONLY for historical purposes. Please feel free to join us on the new TrueNAS Community Forums
System freezes when writing data
- Thread starter jsmith8954
- Start date
- Status
Tekkie
Patron
- Joined
- May 31, 2011
- Messages
- 353
I've found one thing which has greatly improved my system's stability, disable S.M.A.R.T. (if you have enabled it) in the Storage -> View Disks screen.
I am still npt 100% happy as I need to do a hard reset on my FreeNAS 8 system at least once a week because istgt or smbd become unresponsive and nothing that I do can either bring those process back to life or kill them, even a normal shutdown or reboot command will time out and hang the system in limbo.
I am still npt 100% happy as I need to do a hard reset on my FreeNAS 8 system at least once a week because istgt or smbd become unresponsive and nothing that I do can either bring those process back to life or kill them, even a normal shutdown or reboot command will time out and hang the system in limbo.
- Joined
- May 28, 2011
- Messages
- 10,994
No one has mentioned this (I don't think) but are all your computer settings at the BIOS defaults? What I'm really asking is are there any overclocking or like Tekkie said disable SMART might work, or it could be something else in the BIOS you could change to make your system more stable. For example on my system I was pushing the RAM to it's highest rated speed and it should have worked however I ended up with errors like crazy. When I brought it to it's nominal setting my system was stable. I'm sure I could have used the faster RAM setting if I tweaked the RAM voltage slightly but it wasn't worth the effort for a NAS. Maybe you're running AHCI but turning it off would solve the problem. This is just suggestions to try and there are a lot of BIOS settings depending on the MB. If you have 4GB of RAM, four 2TB drives (or less), and running the 64-bit version your system should be able to run stable without tweaking the FreeNAS settings. If you're running 32-bit you are likely to be tweaking the settings so if you can get away from the 32-bit, do it. If you run more than 4 drives, you will probably need more RAM if using ZFS to keep things speedy. And I'd like to mention that my system is completely stable (right now) and I have been running an automated backup that runs every hour and also validates the backup when complete and this has been running non-stop for 4 days to check for a memory leak which hasn't occurred for me. I only say this because it is possible to have a stable system if you're not going to crazy with this Beta-3 version. I don't know if this posting will help anyone but I hope so.
Just wanted to give a quick feedback: I've added the lines protosd wrote earlier (thanks for that!) and haven't seen any freezes in the last days. Haven't got much time in the next few weeks, so I can't play around with the settings, but for now I'm happy. Getting up to 35 mb/sec, but thats enough as long as nothing freezes.
Tekkie
Patron
- Joined
- May 31, 2011
- Messages
- 353
@Joe
I am running my system 100% stock, no OC, no extra core unlocking etc. and I have say that my system will freeze up after 24+ hrs of operation, foremost the istgt process will become unresponsive, not even a reboot / shutdown can be performed, the istgt process is reported as being unresponsive.
I am running my system 100% stock, no OC, no extra core unlocking etc. and I have say that my system will freeze up after 24+ hrs of operation, foremost the istgt process will become unresponsive, not even a reboot / shutdown can be performed, the istgt process is reported as being unresponsive.
- Joined
- May 28, 2011
- Messages
- 10,994
@ Tekkie
You're running the 64 bit version and have 8GB RAM so I wouldn't think changing the vm.kmem values would help but I'm not an expert in that area, only because mine works fine. If you do that and it works or fails, please post. Even failure is good to know so it's not done again.
My advice is, if you haven't already done so, grab a copy of The Ultimate Boot CD and run MemTest for a day or two. That sounds like a lot but if you have a memory error you need to weed it out. There is also a CPU stress test but please don't run this for a full 24 hours unless you can monitor your CPU temperature.
@ Protosd
What values should Tekkie use, how much of a bump would you suggest? And this is off topic but how warm is your Pico power supply? I'm just curious.
You're running the 64 bit version and have 8GB RAM so I wouldn't think changing the vm.kmem values would help but I'm not an expert in that area, only because mine works fine. If you do that and it works or fails, please post. Even failure is good to know so it's not done again.
My advice is, if you haven't already done so, grab a copy of The Ultimate Boot CD and run MemTest for a day or two. That sounds like a lot but if you have a memory error you need to weed it out. There is also a CPU stress test but please don't run this for a full 24 hours unless you can monitor your CPU temperature.
@ Protosd
What values should Tekkie use, how much of a bump would you suggest? And this is off topic but how warm is your Pico power supply? I'm just curious.
@ Tekkie
Save yourself a little time, it's easier to change the settings than making a CD and waiting for MemTest. @Buhmi seemed reluctant to try what I suggested and it helped him. Just because you have 8GB of RAM doesn't mean it's allocated correctly for ZFS. You probably want to allocate, or at least set the max so it uses as much of your 8GB as possible, with the exception of leaving enough for regular applications/OS or plug-ins to run. I'm not an expert on ZFS tuning, but I have done a little reading. I may not be able to spout of details, but it's a quick change and its a lot faster to try. If it doesn't help give the MemTest a try. I'll look up some look into what those values should be for 8GB and update this post.
UPDATE:
Ok, there are a lot of different opinions on recommended values based on your RAM. If you search google for "vm.kmem_size recommended for 8GB", you can read them yourself, and then there are other settings that should also be adjusted based on what you set you VM.KMEM values at. So for you I would try vm.kmem_size="4096" and for vm.kmem_size_max="6144M". You should also consider adjusting vfs.zfs.arc_min & vfs.zfs.arc_max, but please, I'm not an expert do a little reading. The settings I suggested are just a starting point, but I'll bet you stop seeing your problems.
OH, don't forget to reboot after you make changes!
Having the vm.kmem_size_max at 6GB will leave 2GB for other apps and the OS *IF* ZFS does happen to use that much. I think ZFS will use all the RAM you can give it, especially when you add encryption, compression, deduplication etc. etc. later on.
@joe
You just forced me to open up my case and use my IR temp sensor to find out the answer ;-) It's 100 degrees F.... The disks are another story since they're stacked horizontally (bad case design) the bottom disk is 88 and the top disk is 94. I wanted to mod the drive cage so the drive are oriented vertically/sideways so the temp doesn't accumulate on the top disk, but haven't had time so far.
Save yourself a little time, it's easier to change the settings than making a CD and waiting for MemTest. @Buhmi seemed reluctant to try what I suggested and it helped him. Just because you have 8GB of RAM doesn't mean it's allocated correctly for ZFS. You probably want to allocate, or at least set the max so it uses as much of your 8GB as possible, with the exception of leaving enough for regular applications/OS or plug-ins to run. I'm not an expert on ZFS tuning, but I have done a little reading. I may not be able to spout of details, but it's a quick change and its a lot faster to try. If it doesn't help give the MemTest a try. I'll look up some look into what those values should be for 8GB and update this post.
UPDATE:
Ok, there are a lot of different opinions on recommended values based on your RAM. If you search google for "vm.kmem_size recommended for 8GB", you can read them yourself, and then there are other settings that should also be adjusted based on what you set you VM.KMEM values at. So for you I would try vm.kmem_size="4096" and for vm.kmem_size_max="6144M". You should also consider adjusting vfs.zfs.arc_min & vfs.zfs.arc_max, but please, I'm not an expert do a little reading. The settings I suggested are just a starting point, but I'll bet you stop seeing your problems.
OH, don't forget to reboot after you make changes!
Having the vm.kmem_size_max at 6GB will leave 2GB for other apps and the OS *IF* ZFS does happen to use that much. I think ZFS will use all the RAM you can give it, especially when you add encryption, compression, deduplication etc. etc. later on.
@joe
You just forced me to open up my case and use my IR temp sensor to find out the answer ;-) It's 100 degrees F.... The disks are another story since they're stacked horizontally (bad case design) the bottom disk is 88 and the top disk is 94. I wanted to mod the drive cage so the drive are oriented vertically/sideways so the temp doesn't accumulate on the top disk, but haven't had time so far.
jgreco
Resident Grinch
- Joined
- May 29, 2011
- Messages
- 18,680
I'll just add that I've seen ZFS get piggy with the CPU. On an older Opteron 240EE (single core) CPU, for example, turning on compression caused the system to go catatonic for short periods of time, apparently while it was doing the compression.
Tekkie
Patron
- Joined
- May 31, 2011
- Messages
- 353
So after 6 days of not using the FreeNAS box I came back switched on one of my desktops and tried to copy a 4.5GB file across iSCSI, result istgt process became unresponsive, the report page showed 2GB of free RAM at the moment, tried to shutdown the system but it hung itself because of the istgt process which didn't respond to the kill signal. :(
- Joined
- May 28, 2011
- Messages
- 10,994
Still using Beta-4?
Did you ever run any testing on your RAM or CPU? Not saying those are the cause but it would be nice to rule them out.
Could you list your system parts and the ZFS pool configurations.
Did you ever run any testing on your RAM or CPU? Not saying those are the cause but it would be nice to rule them out.
Could you list your system parts and the ZFS pool configurations.
Tekkie
Patron
- Joined
- May 31, 2011
- Messages
- 353
Hardware spec
FreeNAS 8.0.1-B4
ASUS M4A78LT-M LE
AMD Athlon II X3 415e / 2.5 GHz Energy Efficient
8GB DDR3 RAM
Intel Gigabit CT Desktop Adapter
LaCie MosKeyto 4GB
RAIDZ2:
2 x Samsung F4 EcoGreen 2TB
2x Western Digital Caviar 2TB
2x Seagate ST2000DL003
ZFS mirror:
1x Western Digital Scorpio Black 320GB
1x Spinpoint MP4 320GB
FreeNAS 8.0.1-B4
ASUS M4A78LT-M LE
AMD Athlon II X3 415e / 2.5 GHz Energy Efficient
8GB DDR3 RAM
Intel Gigabit CT Desktop Adapter
LaCie MosKeyto 4GB
RAIDZ2:
2 x Samsung F4 EcoGreen 2TB
2x Western Digital Caviar 2TB
2x Seagate ST2000DL003
ZFS mirror:
1x Western Digital Scorpio Black 320GB
1x Spinpoint MP4 320GB
- Joined
- May 28, 2011
- Messages
- 10,994
I like the fact that you switched up your hard drive selection.
Now for the question... You have too many SATA drives for that motherboard listed. I tried to figure out which drives also came in an EIDE interface but didn't see any of them so how do you connect 8 SATA drives to a motherboard that only has 6 SATA connections? I'm sure you will correct me if I'm wrong.
Also, do you recall if your swap space was used at all?
-Mark
- Joined
- May 28, 2011
- Messages
- 10,994
Bet your extra card is not fully supported. If you can, hook up all the drives you want to the MB connectors, pull the interface card, test your NAS for an extended period of time. Have you checked to see if anyone else has that Dell SAS card here to see if they have any problems with it?
That's good about the swap space. I read of problems when the swap spare starts getting used. Of course I haven't heard anyone say that their swap space it used a lot and it never affected them but that would be nice to hear.
That's good about the swap space. I read of problems when the swap spare starts getting used. Of course I haven't heard anyone say that their swap space it used a lot and it never affected them but that would be nice to hear.
Tekkie
Patron
- Joined
- May 31, 2011
- Messages
- 353
Actually I've done some reading up and did some tweaks to loader.conf which appears to have done the trick, I've been hitting the FreeNAS box with multiple GB transfers in recent days and it stays up.
thanks for the help.
PS. The LSI 1068 which this DELL card is, is fully supported by FreeBSD, a little firmware tuning is required but that's all: http://forums.freenas.org/showthread.php?589
thanks for the help.
PS. The LSI 1068 which this DELL card is, is fully supported by FreeBSD, a little firmware tuning is required but that's all: http://forums.freenas.org/showthread.php?589
- Joined
- May 28, 2011
- Messages
- 10,994
Good to hear your problems may be solved. I have setup two backup and verifications via Acronis to occur every day. The first is about 52GB once a day, the second is about 5GB and occurs every 2 hours. This is how I'm testing to see the stability of the NAS. So far so good. My NAS has been up and down while trying new things but I haven't had a single failure when the NAS is running. I'd like to setup something that copies a few thousand small files periodically to complete my testing properly.
similar problems...
Hello:
I killed my old Freenas 7.x system and made a new Freenas 8.0.2-RELEASE system. I found this thread because this is the problem I'm having as well. System seemed happy, but when I go to copy a bunch of data to the system quickly, it freezes up running both cpus at 100%.
After rebooting (twice, 2nd time it hung during the mounting of the zfs filesystem), I tried the dd to generate some data. Unlike others though, that DID cause the system to lock up also. What would that point to as the cause?
Perhaps I should run a test that pegs the CPU but does not hit the drives to isolate that. How should I do that?
----
edit-
Ran yes > /dev/null in two terminal windows which did run both CPUs at 100% for a few minutes and no problems.
----
edit-
modified loader.conf with values recommended earlier in this thread (values for a 4 GB Ram system), this did not change the symptoms.
----
edit-
Removed Dell hba and put data drives on motherboard SATA ports. dd if=/dev/zero of=testfile bs=8192k count=4000 and it still lock up after running that for about 10 seconds.
----
edit-
Realized that I was attempting the copy and dd test on a dataset with the recommended compression turned on, so I tried it on a dataset with no compression (on the same volume). Results were even quicker crashing! Something locks up and both CPUs stay at 100% and I have to shut down the machine
----
Machine details in sig.
Thanks.
Hello:
I killed my old Freenas 7.x system and made a new Freenas 8.0.2-RELEASE system. I found this thread because this is the problem I'm having as well. System seemed happy, but when I go to copy a bunch of data to the system quickly, it freezes up running both cpus at 100%.
After rebooting (twice, 2nd time it hung during the mounting of the zfs filesystem), I tried the dd to generate some data. Unlike others though, that DID cause the system to lock up also. What would that point to as the cause?
Perhaps I should run a test that pegs the CPU but does not hit the drives to isolate that. How should I do that?
----
edit-
Ran yes > /dev/null in two terminal windows which did run both CPUs at 100% for a few minutes and no problems.
----
edit-
modified loader.conf with values recommended earlier in this thread (values for a 4 GB Ram system), this did not change the symptoms.
----
edit-
Removed Dell hba and put data drives on motherboard SATA ports. dd if=/dev/zero of=testfile bs=8192k count=4000 and it still lock up after running that for about 10 seconds.
----
edit-
Realized that I was attempting the copy and dd test on a dataset with the recommended compression turned on, so I tried it on a dataset with no compression (on the same volume). Results were even quicker crashing! Something locks up and both CPUs stay at 100% and I have to shut down the machine
----
Machine details in sig.
Thanks.
Finally found this thread, http://forums.freenas.org/showthread.php?11-FreeNAS-8-Freezing-On-VMware-ESXi-4.1
Seems to have fixed my problem, by changing to 1 vCPU.
- Status
