VM storage is now read-only???

[KiloCharlieDelta]

Good afternoon.

I am running TrueNAS-12.0-U7. I have an a small Ubuntu server running in a VM running Pi-Hole. This has been setup and running for ~1.5 years without any issues.

The other day I had gone into my Pi-Hole web interface to whitelist a domain, and encountered some odd errors, so I attempted to restart the VM from the web interface, but nothing happened. I finally decided that something at the OS level of the VM had occurred, so I initiated a restart the VM from the TrueNAS VM interface.

I waited for the reboot to finish, which did take a bit, but it finally started pinging. However, Pi-Hole never loaded. After logging into the Ubuntu VM I received errors indicating the disk was set to read-only.

I attempted to spin up a new VM, but it won't even boot from the ISO properly, because that new disk is also read-only.

Has anyone else encountered this? Any suggestions?

 

[jgreco]

Yeah, Linux sometimes does this crappy thing where it ends up booting a system with its disks read-only because "something bad" happened during boot, and it apparently isn't smart enough to do something intelligent. It's been sometimes referred to as "read-only rescue" mode. Click link for the obvious Google.

Usually this happens to physical machines, in my experience, but can happen to a virtual machine if suddenly shut off or something else happens. This has nothing in particular to do with "VM storage".

 

[KiloCharlieDelta]

Yeah, Linux sometimes does this crappy thing where it ends up booting a system with its disks read-only because "something bad" happened during boot, and it apparently isn't smart enough to do something intelligent. It's been sometimes referred to as "read-only rescue" mode. Click link for the obvious Google.

Usually this happens to physical machines, in my experience, but can happen to a virtual machine if suddenly shut off or something else happens. This has nothing in particular to do with "VM storage".

I'm not sure this is the issue, as your explanation seems to indicate this should not occur on multiple VMs simultaneously. This doesn't explain why the newly created VM is seemingly having a similar issue. This seems like it's a host issue of some sort to me.

 

[jgreco]

It could easily happen to two or more VM's if the hypervisor was powered off. Seen that happen.

I don't know about the "newly created VM". Try installing something like FreeBSD on that VM and see if it still happens. Easy test.

 

[KiloCharlieDelta]

This is what it looks like when the new VM attempts to boot from the Ubuntu ISO. Very first line is a write error. Keep in mind this a newly created VM shell, created after my issue with the first VM occurred. I even recreated the new VM using a disk from a different storage pool, wondering if it was an issue with the pool, but the results were the same. I have other pools that I use for backups and general storage, and those are working fine. So it's not like there's a blanket read-only applied to all of the pools.

 

[jgreco]

"initramfs unpacking failed: write error"?

Bearing in mind that I despise Linux almost as much as Windows, I would note that Linux uses an in-memory RAM filesystem as part of the boot process. Which is why it is called init_RAM_fs. It sounds like it is failing to write to RAM.

I don't see an obvious indication of how much RAM you have configured. I'm going to assume you've read the instructions for whatever you're installing and are aware that the minimum supported RAM is like 1GB for lots of Linux variants. Therefore, I suggest you try giving your VM 2GB of RAM and see what happens.

Incidentally, rewinding to

This has been setup and running for ~1.5 years without any issues.

I would make the pithy observation that many Linux boxes I've seen fall over after a year or two, their root or /boot partitions typically filled up with initrd.img-blabla, vmlinuz-blabla, and various package manager files. This seems to be particularly true of Ubuntu, which apparently assumes that some Linux fanboi is biting their nails waiting to clean up the mess that Ubuntu 66.6 Piddly Puppy has made all over the system. Honestly, sometimes I don't understand how Linux is considered to be a serious operating system. So I would additionally check your two "read-only" boxes "for ~1.5 years" for disk full and fsck related issues. Kinda expect you'll find one of these things to be impacting you.

 

[KiloCharlieDelta]

I suggest you try giving your VM 2GB of RAM and see what happens.

This worked, so I was able to install Ubuntu on my new VM and PiHole installed just fine on it. All is good now. I deleted the problem VM. Hopefully, the new VM does not suffer the same fate down the road.

Thank you for your help.