RancherOS VM isn't fully booting up any more -- No space left on device error(s)

Lastb0isct

Explorer
Joined
Jun 15, 2011
Messages
96
My docker VM wasn't responding well so I decided to reboot it. This seems to have cause an issues. Any help to get my VM back up and running properly would be appreciated.

[ ] init:error: Failed [16/21] preparefs2: mkdir /tmp: no space left on device
[ ] init:error: RancherOS has failed to boot: mkdir /tmp: no space left on device

My system is throwing this error:

Code:
[   19.256631] FS-Cache: Netfs '9p' registered for caching
[   19.259057] 9pnet_virtio: no channels available for device config-2
[            ] init:info: [1/1] [cloud-init]: Started
[            ] init:info: Project [cloudinit]: Project started
[            ] init:info: [10/21] Starting read cfg and log files
[            ] init:error: read cfg file (/var/log/boot) read /var/log/boot: is a directory
[            ] init:info: [11/21] Starting switchroot
[            ] init:info: [12/21] Starting mount OEM2
[            ] init:info: [13/21] Starting write cfg and log files
[            ] init:info: Swizzle: Wrote file to /etc/resolv.conf
[            ] init:error: write /var/log/boot/.cloud-init-save.log645706070: no space left on device
[            ] init:info: Swizzle: Wrote file to /var/log/boot/cloud-init-save.log
[            ] init:error: write /var/log/boot/.init.log193117373: no space left on device
[            ] init:info: Swizzle: Wrote file to /var/log/boot/init.log
[            ] init:error: write /var/lib/rancher/conf/cloud-config.d/.init.yml471862520: no space left on device
[            ] init:info: Swizzle: Wrote file to /var/lib/rancher/conf/cloud-config.d/init.yml
[            ] init:info: [14/21] Starting b2d Env
[            ] init:info: [15/21] Starting hypervisor tools
[            ] init:info: [16/21] Starting preparefs2
[            ] init:error: Failed [16/21] preparefs2: mkdir /tmp: no space left on device
[            ] init:error: RancherOS has failed to boot: mkdir /tmp: no space left on device
[            ] init:info: Launching recovery console
[            ] init:fatal: mkdir /tmp: no space left on device
[   19.973885] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000100
[   19.973885]
[   19.975598] CPU: 1 PID: 197 Comm: init Not tainted 4.9.75-rancher #1
[   19.976885] Hardware name:   BHYVE, BIOS 1.00 03/14/2014
[   19.977788]  ffffa5f143967c40 ffffffffb7328b56 ffff8c5d855d0d00 ffffffffb7a8e4a9
[   19.979256]  ffffa5f143967cc0 ffffffffb71466f7 ffffa5f100000010 ffffa5f143967cd0
[   19.980736]  ffffa5f143967c68 ffffffffb7c63768 0000000000000100 ffff8c5d8a350010
[   19.982196] Call Trace:
[   19.982670]  [<ffffffffb7328b56>] dump_stack+0x61/0x7d
[   19.983589]  [<ffffffffb71466f7>] panic+0xd8/0x221
[   19.984482]  [<ffffffffb706bf1e>] do_exit+0x4d4/0x92c
[   19.985460]  [<ffffffffb70cecf6>] ? __unqueue_futex+0x55/0x58
[   19.986567]  [<ffffffffb70cfbf8>] ? futex_wait+0x192/0x25a
[   19.987641]  [<ffffffffb706c3ed>] do_group_exit+0x41/0xa2
[   19.988722]  [<ffffffffb7075db0>] get_signal+0x472/0x4ee
[   19.989754]  [<ffffffffb70232e2>] do_signal+0x28/0x5d6
[   19.990399]  [<ffffffffb70d0f9b>] ? do_futex+0xc4/0x82b
[   19.991029]  [<ffffffffb70c551c>] ? tk_clock_read+0xc/0xe
[   19.991700]  [<ffffffffb70c5d5f>] ? timekeeping_get_ns+0x1d/0x31
[   19.992727]  [<ffffffffb70c5fd1>] ? ktime_get_ts64+0x47/0x8a
[   19.993720]  [<ffffffffb71c6536>] ? poll_select_copy_remaining+0xeb/0x103
[   19.994925]  [<ffffffffb70036f9>] prepare_exit_to_usermode+0x6f/0xb2
[   19.995983]  [<ffffffffb70037e7>] syscall_return_slowpath+0xab/0xca
[   19.997015]  [<ffffffffb76e61c7>] entry_SYSCALL_64_fastpath+0xc7/0xc9
[   19.998537] Kernel Offset: 0x36000000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
[   20.000147] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000100
[   20.000147]
 
Last edited:

Lastb0isct

Explorer
Joined
Jun 15, 2011
Messages
96
Watchertower docker container is taking up all the space on my system. I'm attempting to go in now and clean space through another VM but i'm finding it difficult to clear up any space at all without breaking things. Expanding a secondary disk in RancherOS seems to be not possible...
 
Top