all space gone in ZFS

[asw2012]

Problem in my Freenas box. It all started about three months ago when my boot USB drive went haywire / corrupted itself. My fault for not creating a backup of the boot drive and/or saving the config file. I did have a config file, but it was from before I installed a few plugins. I restored it which got me some of the way back to where I was.

So I bought a new USB, installed the same version of FreeNAS 9.3 and loaded the config file. I re-installed subsonic and left it off - until I had time to configure. I also installed MineOS and Emby - and left those off as well. There was about 4.5TB of space left at this time.

This past week, when I was trying to transfer a few items through my CIFS share, got an error "drive is full". I have not transferred any files the past three months that would fill up 4.5TB worth of data. I go to my FreeNAS web console to find out there is 0 space left. I found a way to free up 1GB of data by deleting a jail I know I did not need. I then rebooted the server and left it alone until I had time to figure out what happened. This was yesterday. Now, there is 0 space left again... Something is filling it up and I do not know what's going on. I could use some help here. see attached pictures.. they might help..

I checked my snapshots, those are under control.

Another odd thing I noticed, the volume states 21.1TB - but I know it's only ever had 15TB.

 

[m0nkey_]

It looks to me you've been putting everything in your root dataset. Is that correct? If so, it'll probably be tough to figure out exactly what's taking up all the space.

Do you have any snapshots configured?

 

[styno]

What is the output of the following command?

du -sh /mnt/RAIDz2/*

 

[asw2012]

What is the output of the following command?

du -sh /mnt/RAIDz2/*

 

[Spearfoot]

What's the output of ls -l /mnt/RAIDz2?

 

[asw2012]

What's the output of ls -l /mnt/RAIDz2?

 

[asw2012]

I think I might have found the problem. I have an Acronis backup scheduled every Wed. morning at 1AM to save to my FreeNAS box. I went to go turn this off so I don't fill up the little space that's left. I found that Acronis thinks it's trying to backup 7.2TB worth of data from my computer, which I know is not correct - my hard drive is only 250GB before it's formatted. I do not see any massive file where the Acronis backups are stored, it might be in limbo, only saving some of it and taking up all that space and there's no proper "file" yet because the backup is not finished. I'll report back to see if this is the root cause.

 

[danb35]

One way or another, you have 15 TB in /mnt/RAIDz2/Share. What does du -sh /mnt/RAIDz2/Share/* show?

 

[Spearfoot]

One way or another, you have 15 TB in /mnt/RAIDz2/Share. What does du -sh /mnt/RAIDz2/Share/* show?

Yes, and Share doesn't show up in the OP's zfs list -o space output or either of the pool status screenshots. Seems to be simply a directory in /mnt/RAIDz2, much like it was created from the shell and not as a dataset via the WebGUI.

 

[m0nkey_]

This is why datasets are important. You can limit the amount of space they use.

 

[Robert Trevellyan]

Acronis thinks it's trying to backup 7.2TB worth of data from my computer

Is it backing up the contents of the share, including previous backups, to the share?

 

[asw2012]

It looks to me you've been putting everything in your root dataset. Is that correct? If so, it'll probably be tough to figure out exactly what's taking up all the space.

Do you have any snapshots configured?

I do have snapshots configured.

One way or another, you have 15 TB in /mnt/RAIDz2/Share. What does du -sh /mnt/RAIDz2/Share/* show?

I see a max of 9.2TB +/- a few GB

This is why datasets are important. You can limit the amount of space they use.

After this ordeal, I am going to research and implement this.

Is it backing up the contents of the share, including previous backups, to the share?

It's set to only backup the windows hard drive.

 

[danb35]

I see a max of 9.2TB +/- a few GB

Indeed. What about ls -lh /mnt/RAIDz2/Share?

 

[asw2012]

Indeed. What about ls -lh /mnt/RAIDz2/Share?

 

[danb35]

OK, so nothing significant in Share itself (as opposed to its subdirectories). du isn't the best tool for determining space used with ZFS, but I'd expect it to do better than being off by 50+%. Curious.

 

[Spearfoot]

@asw2012: your screenshots are chopping off part of the command output; makes it hard to see what's going on.

The output of du -sh /mnt/RAIDz2/* you posted above shows that the /mnt/RAIDz2/Share directory contains 15TB of data; this is where your problems lie. Unfortunately, this is just a directory on your system; you didn't create it as a dataset using the WebGUI. So it doesn't show up in the WebGUI at all. I mentioned all of this earlier in the thread.

This would be easier to explore if you install WinSCP or a similar tool.

In any case, run du -sh /mnt/RAIDz2/Share/* to see which directories in /mnt/RAIDz2/Share are using the most space. Then go from there to see about deleting unnecessary files.

 

[danb35]

In any case, run du -sh /mnt/tank/RAIDz2/Share to see which directories in /mnt/tank/RAIDz2/Share are using the most space.

He did that above. Strangely, the directories in Share only total a little over 9 TB, and there's nothing significant in the root of Share either.

 

[Spearfoot]

He did that above. Strangely, the directories in Share only total a little over 9 TB, and there's nothing significant in the root of Share either.

No, sir, he ran ls above. Unless I missed something?

We haven't yet seen his results from running du on the Share directory.

 

[danb35]

Unless I missed something?

I think you must have. I refer you again to this post, where, in response to my request, he appears to have run du -sh /mnt/RAIDz2/Share/* (admittedly, it's only "appears to have run", as he cut off the full command). The output from that command is also cut off, presumably to mask directory names, but it's still there.

 

[asw2012]

I think you must have. I refer you again to this post, where, in response to my request, he appears to have run du -sh /mnt/RAIDz2/Share/* (admittedly, it's only "appears to have run", as he cut off the full command). The output from that command is also cut off, presumably to mask directory names, but it's still there.

yes. did not want to show the directory names, that's all. du -sh /mnt/RAIDz2/Share/* was run.