iSCSI Service crashing rootfs on Virtual Machines after reboot

[viniciusferrao]

Hello guys,

I don't know exactly if this is a bug report or not. We're using FreeNAS-9.2.0-RELEASE-x64 (ab098f4) and iSCSI service fails to restablish connection after a power failure in a clean state.

What really happens. We have some VM's running over iSCSI on a XenServer host. When iSCSI goes down for a long period of time the filesystem in the VM's remounts as read-only after disk corruption is detected.

The problem is when the iSCSI gets back nothing happens, since the rootfs got corrupted.

Here are some dmesg output from the Linux VM:

Code:

[894425.330993] end_request: I/O error, dev xvda, sector 768272
[894425.331019] Buffer I/O error on device dm-0, logical block 33058
[894425.331035] EXT4-fs warning (device dm-0): ext4_end_bio:250: I/O error writing to inode 2999 (offset 0 size 4096 starting block 33058)
[894425.331057] end_request: I/O error, dev xvda, sector 777616
[894425.331067] Buffer I/O error on device dm-0, logical block 34226
[894425.331079] EXT4-fs warning (device dm-0): ext4_end_bio:250: I/O error writing to inode 3010 (offset 0 size 4096 starting block 34226)
[894425.331093] end_request: I/O error, dev xvda, sector 5048224
[894425.331134] JBD2: Detected IO errors while flushing file data on dm-0-8
[894425.331162] Aborting journal on device dm-0-8.
[894425.331221] EXT4-fs error (device dm-0) in ext4_free_blocks:4766: Journal has aborted
[894457.732827] EXT4-fs (dm-0): Remounting filesystem read-only
[894457.732853] EXT4-fs error (device dm-0) in ext4_reserve_inode_write:4499: Journal has aborted
[894457.774125] EXT4-fs error (device dm-0) in ext4_reserve_inode_write:4499: Journal has aborted
[894457.840951] EXT4-fs error (device dm-0) in ext4_ext_remove_space:2742: Journal has aborted
[894457.841677] EXT4-fs error (device dm-0) in ext4_reserve_inode_write:4499: Journal has aborted
[894457.847755] EXT4-fs error (device dm-0) in ext4_orphan_del:2110: Journal has aborted
[894457.857108] EXT4-fs error (device dm-0) in ext4_reserve_inode_write:4499: Journal has aborted
[894457.858184] EXT4-fs error (device dm-0): ext4_journal_start_sb:327: Detected aborted journal
[980980.704070] EXT4-fs (dm-0): error count: 8
[980980.704087] EXT4-fs (dm-0): initial error at 1394156871: ext4_free_blocks:4766
[980980.704097] EXT4-fs (dm-0): last error at 1394156904: ext4_reserve_inode_write:4499
[994860.307102] adclient[2625]: segfault at 7fbfbb454660 ip 00007fbfb969b07d sp 00007fbfb63cc640 error 4 in libc-2.13.so[7fbfb9624000+182000]
[1067488.224066] EXT4-fs (dm-0): error count: 8
[1067488.224083] EXT4-fs (dm-0): initial error at 1394156871: ext4_free_blocks:4766
[1067488.224093] EXT4-fs (dm-0): last error at 1394156904: ext4_reserve_inode_write:4499

I just need some help to debug this. I don't even think if it's a FreeNAS problem. But it should be, since it never happened (perhaps a coincidence) on our FreeNAS 8.3 iSCSI service.

Thanks in advance,

 

[dlavigne]

Were you ever able to figure this out?

 

[viniciusferrao]

Nothing changed yet... I still have this problem.

 

[dlavigne]

Might be worthwhile creating a bug report at bugs.freenas.org. If you do, post the issue number here.