100% used 0 Avail (after 9.2.9 to 9.3)

[bobbob1016]

First, I know, I know, shouldn't be that filled with data. Just times are tight, I don't have the $ to upgrade my storage now.

I upgraded a 6 1.5tb setup from 9.2.9 to 9.3 earlier, and have since been unable to use the pool. I had 160 gig or so free before the upgrade. I am unable to delete files, but I can read them. rm gives me "no space to remove" or whatever. I did "dd if=/dev/null of=3gigfile" and it now has 0 space, but I still can't delete. I've freed up what has to be 20 gig with doing that. Also removed a few snapshots. If anyone could help I'd appreciate it. Don't know if I'm allowed to paste the output from zpool status here, so I've pastebined it.

http://pastebin.com/Yc9gmC7P

 

[gpsguy]

Search the forum for a thread like .. Disk full can't delete ... For a trick on how to delete files, when this happens


Sent from my phone

 

[bobbob1016]

I'll look when I get home, but if it's the dd the file trick, I've done that.

Edit: No, only things I've seen talk about creating another dataset to lessen later. I've deleted jails, and still at 100%, there seems to be something wrong. Not sure if this is a known bug.
Further edit: Shows as 187 gig free under Storage -> Volumes

 

[gpsguy]

See message #3 in this thread: https://forums.freenas.org/index.php?threads/disk-full-cant-delete-any-files-please-help.12252/

 

[bobbob1016]

That doesn't work, kind of the same trick as the dd one. The files are 0 size, and FreeNAS says I have 187 gig free in Settings -> Volumes, but still no space to start a Jail, nor delete even 0 size files.

 

[bobbob1016]

I think I found the issue. For some reason, it's added twice. As in, it's there as full storage, but it seems to be a dataset of itself or something. Here's a pic:
http://picpaste.com/Screenshot_from_2014-12-19_06_56_33-UqB7M4a8.png

 

[dlavigne]

No, that's just the way volumes are displayed in 9.3. See http://doc.freenas.org/9.3/freenas_storage.html#view-volumes for an example.

 

[bobbob1016]

Then, is this issue to be expected? I mean, I have 187 gig free, yet the nested set is saying it's at 100% when it shouldn't be from what I gather.

 

[dlavigne]

Yes, this is to be expected. ZFS gets very unhappy after it reaches 90% capacity.

 

[bobbob1016]

This wasn't the case pre 9.3 though. 9.2.9 was just fine with 187gig free...

Edit: Further, I can't even delete things. Even things of 0 size. Also, I don't mean absolutely fine, but I didn't have this type of an issue.

Again edited: I've gotten to 240gig free, that's about 95/96% free. As I've been seeing people set 1gig datasets to delete in this case, I'm not sure if this is a bug or something. I've saved my settings from before the upgrade, but not sure if that'll help.

 

[SirMaster]

Find a file that is outside of your snapshots. You can use "zfs diff" to do this. Then truncate the file "truncate -s 0 my_large_file". This should allow you to then delete other files as normally without ZFS saying there is not enough free space.

 

[bobbob1016]

Looking in to the syntax for my case. The main issue is I'm at 100% on basically all of my storage. I'm not sure how to use the @ and things. Here's the pic I pasted earlier, but the whole tank seems to think it's filled. I named the drive RAIDZ. So, I have RAIDZ (240gig free now) -> RAIDZ 100% full -> everything else. Here's the pic when I had 187 gig free. I removed a 50 gig file, but it's in all my snapshots.

Edit: Pic didn't paste http://picpaste.com/pics/Screenshot_from_2014-12-19_06_56_33-UqB7M4a8.1419012306.png

 

[bobbob1016]

Anyone else having this issue? It says I have 240gig free on the "main" set, just the one under it says it's 100% full. I just have one more drive to add before it gets to 6*2tb drives, so I'll get more that way, just can't afford the drive at the moment, and would like to get this working.

 

[diedrichg]

How much RAM do to have? What is your CPU and motherboard?

 

[bobbob1016]

16gig, ECC, Asrock mobo, soldered on Intel Ion. "Intel(R) Atom(TM) CPU C2750 @ 2.40GHz" copied from my FreeNAS status page.

 

[diedrichg]

Thank you, I was unable to see that as I am on Tapatalk.

 

[T0M]

Hello, I just upgraded from 8.3 to 9.3 and was experiencing the exact same problem.

I solved it by following this guide: http://www.surlyjake.com/blog/2010/01/19/zfs-cant-rm-no-space-left-on-device/.

Maybe it has something to do with the warnings I'm getting for both my flashed IBM M1015 LSI-controllers:

• WARNING: Firmware version 15 does not match driver version 16 for /dev/mps0
• WARNING: Firmware version 15 does not match driver version 16 for /dev/mps1

But maybe that's just an unharmful warning?

This is what the volume manager looked like before I managed to delete a couple of big files:

 

[bobbob1016]

Thanks, but that's exactly what I've done, (think I mentioned it above, but not sure on mobile and can't really scroll), didn't fix it. Tried doing that two ways actually, same difference...

 

[gpsguy]

TOM, "Firmware version 15 does not match driver version 16 for /dev/mps0" has nothing to do with this problem.

This message is informing you that you need to reflash your M1015 HBA's with the phase 16 firmware. You need to have matched versions of the firmware and driver.

 

[T0M]

TOM, "Firmware version 15 does not match driver version 16 for /dev/mps0" has nothing to do with this problem.

This message is informing you that you need to reflash your M1015 HBA's with the phase 16 firmware. You need to have matched versions of the firmware and driver.

Thank you for clearing that up. I will need a guide to help me on this one as the M1015's will need to stay in IT-mode after updating/flashing the firmware to prevent them from acting as RAID-controllers.