Something is full, and its causing problems...

[Hexland]

So, I finally built my FreeNAS server -- installed 6 x 2TB drives, and formatted it to Raid-Z2.

I created a volume 7.1TB in size, and signed it over to iSCSI.

I have a WHS 2011 machine which uses the iSCSI drive as its primary drive, with all of the shares on the iSCSI drive - across a dedicated Gigabit NIC.

I started transferring the data from my old WHS v1 machine over to the new machine combo pair yesterday -- and it got about 3.5TB into a 5TB transfer, before Windows Update installed a bunch of stuff at 3am this morning, and rebooted the WHS 2011 machine (which didn't recover from the reboot until I got up this morning and manually reset it)

At the point of rebooting the WHS 2011 machine, the FreeNAS machine started logging issues with the iSCSI link.

I awoke this morning to about 80 emails about my drive being full, and not being able to draw some graphs... I'm assuming a log file has filled up somewhere, but I don't know enough to clean it out -- and I don't see any GUI options for clearing out log files...

Any help to stop this bombardment of emails would be appreciated...

Thx

Dave

Code:

Subject: Cron <root@freenas> /usr/local/bin/python /usr/local/bin/graph.py
From: root <root@freenas>
To: root
Date: Tue, 27 Mar 2012 17:00:09 -0000
X-Cron-Env: <SHELL=/bin/sh>

Traceback (most recent call last):
  File "/usr/local/bin/graph.py", line 480, in <module>
    GenerateInterfaceGraph()
  File "/usr/local/bin/graph.py", line 135, in GenerateInterfaceGraph
    'GPRINT:global_tot_tx:ca. %5.1lf%s Total\l'
rrdtool.error: Could not save png to '/tmp/graph/if-em0-1h.png'

Code:

Subject: Cron <root@freenas> newsyslog
From: root <root@freenas>
To: root
Date: Tue, 27 Mar 2012 14:00:36 -0000
X-Cron-Env: <SHELL=/bin/sh>


bzip2: I/O or other error, bailing out.  Possible reason follows.
bzip2: No space left on device
	Input file =3D /var/log/messages.0, output file =3D /var/log/messages.0.bz2
bzip2: Deleting output file /var/log/messages.0.bz2, if it exists.
newsyslog: `bzip2 -f /var/log/messages.0' terminated with a non-zero status=
 (1)

Code:

From: root <root@freenas>
To: root
Date: Tue, 27 Mar 2012 19:00:11 -0000
X-Cron-Env: <SHELL=/bin/sh>

newsyslog: log_trim: fclose: No space left on device

 

[joeschmuck]

What version of FreeNAS are you using?

The log files are in RAM and rolled over periodically as I understand it. Rebooting your FreeNAS machine will restore your /var/ area.

It's been a few days since your problem started, hopefully you have fixed it by now.

Also, I'd turn off the automatic updates on your Windows server while you are doing transfers at night while it's unattended.

 

[j2dapo]

Sorry to bump an older thread. Two days ago I updated from 8.0.1 to the latest release 8.3 using the GUI updater. Now my NAS is hanging/unresponsive and I've been receiving these emails every hour from Cron with

"newsyslog: log_trim: fclose: No space left on device"

I believe there is another problem that is filling up the logs and am ready to hard reset the box and see if it will respond. Any suggestions?

 

[tdarsey]

I am needing to bump this. I am getting the same emails from my box and it is running SLOW. How do I find out what is full? I have setup a 6.8TB iSCSI target connected to a Server 2008 R2 server. The server is saying that I am only about 1/2 full. How do I find out what is full?

 

[dlavigne]

If the message is coming from newsyslog, it is /var/ on the OS drive that is filling up (ie. not the storage disks). Take a look at what is in /var/log for the culprit.

If logs are on ongoing issue, create a ZFS dataset called syslog and FreeNAS will move the logs there (after a reboot).

 

[tdarsey]

Well, I tried to create it but it is now giving me this error.

• cannot create 'Volume1/syslog': out of space

 

[jgreco]

Filling your pool is always a recipe for disaster with ZFS. Try "zpool list" from the command line. It'll tell you under "CAP" how full you are. If it's over 80%, you're in for some serious pain, especially with iSCSI.