iscsi connection dropping vms read only

[jgreco]

Hello All,

Since last couple of days our iscsi connection getting drop with below erros. Our FreeNAS storage connected to kvm HV's via iscsi. All vms gone into read only mode. Please suggest what could be issue. we have checked there is no network connectivity issue.

WARNING: 10.0.0.0.0 (iqn.1994-05): tasks terminated
WARNING: 10.xx.xx.xx. (iqn.1994-05.com.): tasks terminated
WARNING: 10.xx.xx.xx.xx (iqn.1994-05.com.r): tasks terminated
ropping connection
WARNING: 10.200.5.254 (iqn.1994-05.com.redhat:HV6): connection error; dropping connection
WARNING: 10.200.1.254 (iqn.1994-05.com.redhat:backup): connection error; dropping connection
ctl_datamove: tag 0xbc3c80e on (1:7:7) aborted
ctl_datamove: tag 0xbc3c80f on (1:7:7) aborted
ctl_datamove: tag 0xbc3c810 on (1:7:7) aborted
ctl_datamove: tag 0xbc3c811 on (1:7:7) aborted
ctl_datamove: tag 0xbc3c812 on (1:7:7) aborted
ctl_datamove: tag 0xbc3c813 on (1:7:7) aborted

Please refer to the following article:

https://www.ixsystems.com/community/threads/the-path-to-success-for-block-storage.81165/

The inappropriate introduction of RAIDZ into block storage is a generally catastrophic mistake.

 

[chetan008]

Dear Samuel Tai,

In the below list i'm not able to locate which is new one. Since in gui visk disk showing 0 to 63 disk it should be o to 64 disk. (da0 its OS hardware raid one disk)

root@[~]# camcontrol devlist
<DELL PERC H730P Mini 4.29> at scbus0 target 0 lun 0 (pass0,da0)
<ATA VK1920GFLKL HPG0> at scbus1 target 0 lun 0 (pass1,da1)
<ATA VK1920GFLKL HPG0> at scbus1 target 1 lun 0 (pass2,da2)
<ATA VK1920GFLKL HPG0> at scbus1 target 2 lun 0 (pass3,da3)
<ATA VK1920GFLKL HPG0> at scbus1 target 3 lun 0 (pass4,da4)
<ATA VK1920GFLKL HPG0> at scbus1 target 4 lun 0 (pass5,da5)
<ATA VK1920GFLKL HPG0> at scbus1 target 5 lun 0 (pass6,da6)
<ATA VK1920GFLKL HPG0> at scbus1 target 6 lun 0 (pass7,da7)
<ATA VK1920GFLKL HPG0> at scbus1 target 7 lun 0 (pass8,da8)
<ATA VK1920GFLKL HPG0> at scbus1 target 8 lun 0 (pass9,da9)
<ATA VK1920GFLKL HPG0> at scbus1 target 10 lun 0 (pass10,da10)
<ATA VK1920GFLKL HPG0> at scbus1 target 11 lun 0 (pass11,da11)
<ATA VK1920GFLKL HPG0> at scbus1 target 14 lun 0 (pass12,da12)
<ATA VK1920GFLKL HPG0> at scbus1 target 15 lun 0 (pass13,da13)
<ATA VK1920GFLKL HPG0> at scbus1 target 16 lun 0 (pass14,da14)
<ATA VK1920GFLKL HPG0> at scbus1 target 17 lun 0 (pass15,da15)
<ATA VK1920GFLKL HPG0> at scbus9 target 20 lun 0 (pass16,da16)
<ATA VK1920GFLKL HPG0> at scbus9 target 21 lun 0 (pass17,da17)
<ATA VK1920GFLKL HPG0> at scbus9 target 22 lun 0 (pass18,da18)
<ATA VK1920GFLKL HPG0> at scbus9 target 23 lun 0 (pass19,da19)
<ATA VK1920GFLKL HPG0> at scbus9 target 24 lun 0 (pass20,da20)
<ATA VK1920GFLKL HPG0> at scbus9 target 25 lun 0 (pass21,da21)
<ATA VK1920GFLKL HPG0> at scbus9 target 26 lun 0 (pass22,da22)
<ATA VK1920GFLKL HPG0> at scbus9 target 27 lun 0 (pass23,da23)
<ATA VK1920GFLKL HPG0> at scbus9 target 28 lun 0 (pass24,da24)
<ATA VK1920GFLKL HPG0> at scbus9 target 29 lun 0 (pass25,da25)
<ATA VK1920GFLKL HPG0> at scbus9 target 30 lun 0 (pass26,da26)
<ATA VK1920GFLKL HPG0> at scbus9 target 31 lun 0 (pass27,da27)
<ATA VK1920GFLKL HPG0> at scbus9 target 32 lun 0 (pass28,da28)
<ATA VK1920GFLKL HPG0> at scbus9 target 33 lun 0 (pass29,da29)
<ATA VK1920GFLKL HPG0> at scbus9 target 34 lun 0 (pass30,da30)
<ATA VK1920GFLKL HPG0> at scbus9 target 35 lun 0 (pass31,da31)
<ATA VK1920GFLKL HPG0> at scbus9 target 36 lun 0 (pass32,da32)
<ATA VK1920GFLKL HPG0> at scbus9 target 37 lun 0 (pass33,da33)
<ATA VK1920GFLKL HPG0> at scbus9 target 38 lun 0 (pass34,da34)
<ATA VK1920GFLKL HPG0> at scbus9 target 39 lun 0 (pass35,da35)
<ATA VK1920GFLKL HPG0> at scbus9 target 40 lun 0 (pass36,da36)
<ATA VK1920GFLKL HPG0> at scbus9 target 41 lun 0 (pass37,da37)
<ATA VK1920GFLKL HPG0> at scbus9 target 42 lun 0 (pass38,da38)
<ATA VK1920GFLKL HPG0> at scbus9 target 43 lun 0 (pass39,da39)
<ATA VK1920GFLKL HPG0> at scbus9 target 45 lun 0 (pass40,da40)
<ATA VK1920GFLKL HPG0> at scbus9 target 46 lun 0 (pass41,da41)
<ATA VK1920GFLKL HPG0> at scbus9 target 47 lun 0 (pass42,da42)
<ATA VK1920GFLKL HPG0> at scbus9 target 48 lun 0 (pass43,da43)
<ATA VK1920GFLKL HPG0> at scbus9 target 49 lun 0 (pass44,da44)
<ATA VK1920GFLKL HPG0> at scbus9 target 50 lun 0 (pass45,da45)
<ATA VK1920GFLKL HPG0> at scbus9 target 51 lun 0 (pass46,da46)
<ATA VK1920GFLKL HPG0> at scbus9 target 52 lun 0 (pass47,da47)
<ATA VK1920GFLKL HPG0> at scbus9 target 53 lun 0 (pass48,da48)
<ATA VK1920GFLKL HPG0> at scbus9 target 54 lun 0 (pass49,da49)
<ATA VK1920GFLKL HPG0> at scbus9 target 55 lun 0 (pass50,da50)
<ATA VK1920GFLKL HPG0> at scbus9 target 56 lun 0 (pass51,da51)
<ATA VK1920GFLKL HPG0> at scbus9 target 57 lun 0 (pass52,da52)
<ATA VK1920GFLKL HPG0> at scbus9 target 58 lun 0 (pass53,da53)
<ATA VK1920GFLKL HPG0> at scbus9 target 59 lun 0 (pass54,da54)
<ATA VK1920GFLKL HPG0> at scbus9 target 60 lun 0 (pass55,da55)
<ATA VK1920GFLKL HPG0> at scbus9 target 61 lun 0 (pass56,da56)
<ATA VK1920GFLKL HPG0> at scbus9 target 62 lun 0 (pass57,da57)
<ATA VK1920GFLKL HPG0> at scbus9 target 63 lun 0 (pass58,da58)
<ATA VK1920GFLKL HPG0> at scbus9 target 64 lun 0 (pass59,da59)
<ATA VK1920GFLKL HPG0> at scbus9 target 65 lun 0 (pass60,da60)
<ATA VK1920GFLKL HPG0> at scbus9 target 66 lun 0 (pass61,da61)
<ATA VK1920GFLKL HPG0> at scbus9 target 67 lun 0 (pass62,da62)
<ATA VK1920GFLKL HPG0> at scbus9 target 68 lun 0 (pass63,da63)
<DELL MD1220 1.06> at scbus9 target 69 lun 0 (pass64,ses0)
<DELL MD1220 1.06> at scbus9 target 70 lun 0 (pass65,ses1)

 

[chetan008]

Dear Team,
We are reconstructing our pool from scratch please advice on this. We will create iscsi zvol for vms

pool: Md1220-one--mirror-o1
state: ONLINE
scan: none requested
config:

NAME STATE READ WRITE CKSUM
Md1220-one--mirror-o1 ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
gptid/41afae3c-bafe-11ea-a5f2-1418774298ed ONLINE 0 0 0
gptid/41d09ee5-bafe-11ea-a5f2-1418774298ed ONLINE 0 0 0
mirror-1 ONLINE 0 0 0
gptid/9d88705b-bafe-11ea-a5f2-1418774298ed ONLINE 0 0 0
gptid/9d9c41b6-bafe-11ea-a5f2-1418774298ed ONLINE 0 0 0
mirror-2 ONLINE 0 0 0
gptid/0a7f0b0b-baff-11ea-a5f2-1418774298ed ONLINE 0 0 0
gptid/0a941713-baff-11ea-a5f2-1418774298ed ONLINE 0 0 0
spares
gptid/837a9ccd-baff-11ea-a5f2-1418774298ed AVAIL
gptid/83c829db-baff-11ea-a5f2-1418774298ed AVAIL
gptid/83dcd2c8-baff-11ea-a5f2-1418774298ed AVAIL

 

[Samuel Tai]

Yes, that's a sensible arrangement.

 

[chetan008]

Dear Samuel ,

Thanks

 

[chetan008]

Hi,

We are checking the logs on freenas, we found only below logs getting capture only. other logs such as messages logs not capturing. Please advice, what could ne the issue

root@[/var/log]# ls -la | grep Jul
drwxr-xr-x 7 root wheel 64 Jul 3 03:01 .
-rw-r--r-- 1 root wheel 1 Jul 3 03:01 3ware_raid_alarms.today
-rw-r--r-- 1 root wheel 1 Jul 2 03:01 3ware_raid_alarms.yesterday
-rw------- 1 root wheel 241840 Jul 3 03:01 dmesg.today
-rw------- 1 root wheel 113140 Jul 2 03:01 dmesg.yesterday
-rw-r----- 1 root wheel 68 Jul 3 00:00 maillog
-rw-r----- 1 root wheel 128 Jul 3 00:00 maillog.0.bz2
-rw-r----- 1 root wheel 124 Jul 2 00:00 maillog.1.bz2
-rw-r----- 1 root wheel 125 Jul 1 00:00 maillog.2.bz2
-rw-r----- 1 root wheel 9679477 Jul 3 10:07 middlewared.log
drwxr-xr-x 2 root wheel 4 Jul 3 00:00 nginx
-rw-r--r-- 1 www www 0 Jul 1 07:51 nginx-access.log
-rw-r--r-- 1 www www 0 Jul 1 07:51 nginx-error.log
-rw------- 1 root wheel 11830 Jul 1 07:51 userlog
-rw-r--r-- 1 root wheel 197 Jul 3 06:11 utx.lastlogin
-rw-r--r-- 1 root wheel 12485 Jul 3 06:11 utx.log
-rw-r----- 1 root wheel 17568 Jul 1 07:52 uwsgi.log

 

[Samuel Tai]

Since you restructured your pools, you'll need to point the system dataset under System->System Dataset to one of your pools again.

 

[chetan008]

Dear Samuel,

Not our old box is not reconstructed yet,we have configured new box. However, old box not showing logs. in system dataset its already point to our pool.Does I need to click on save again ?

root@[~]# df -h
Filesystem Size Used Avail Capacity Mounted on

Dell-730-mini-local/.system 19T 475M 19T 0% /var/db/system
Dell-730-mini-local/.system/cores 19T 5.6M 19T 0% /var/db/system/cores
Dell-730-mini-local/.system/samba4 19T 620K 19T 0% /var/db/system/samba4
Dell-730-mini-local/.system/syslog-4a7da323489644858925be23483a2e70 19T 35M 19T 0% /var/db/system/syslog-4a7da323489644858925be23483a2e70
Dell-730-mini-local/.system/rrd-4a7da323489644858925be23483a2e70 19T 117M 19T 0% /var/db/system/rrd-4a7da323489644858925be23483a2e70
Dell-730-mini-local/.system/configs-4a7da323489644858925be23483a2e70 19T 59M 19T 0% /var/db/system/configs-4a7da323489644858925be23483a2e70
Dell-730-mini-local/.system/webui 19T 217K 19T 0% /var/db/system/webui
root@wwstorage-ssd[~]# cd /var/db/system/syslog-4a7da323489644858925be23483a2e70
root@wwstorage-ssd[...slog-4a7da323489644858925be23483a2e70]# ls -la
total 41
drwxr-xr-x 3 root wheel 3 Aug 21 2019 .
drwxr-xr-x 10 root wheel 12 Jul 5 04:57 ..
drwxr-xr-x 7 root wheel 64 Jul 5 03:01 log

cd logs

total 2249
drwxr-xr-x 7 root wheel 64 Jul 5 03:01 .
drwxr-xr-x 3 root wheel 3 Aug 21 2019 ..
-rw-r--r-- 1 root wheel 1 Jul 5 03:01 3ware_raid_alarms.today
-rw-r--r-- 1 root wheel 1 Jul 4 03:01 3ware_raid_alarms.yesterday
-rw------- 1 root wheel 8626 Mar 12 18:24 auth.log
-rw------- 1 root wheel 1176 Jan 1 2020 auth.log.0.bz2
-rw------- 1 root wheel 79564 Mar 20 10:40 cron
-rw------- 1 root wheel 5275 Mar 19 00:00 cron.0.bz2
-rw------- 1 root wheel 5260 Mar 17 00:00 cron.1.bz2
-rw------- 1 root wheel 5362 Mar 15 00:00 cron.2.bz2
-rw------- 1 root wheel 13591 Mar 20 10:23 daemon.log
-rw-r----- 1 root wheel 4207 Mar 20 00:00 daemon.log.0.bz2
-rw-r----- 1 root wheel 3694 Mar 18 00:00 daemon.log.1.bz2
-rw-r----- 1 root wheel 3668 Mar 15 00:00 daemon.log.2.bz2
-rw-r----- 1 root wheel 3581 Mar 12 00:00 daemon.log.3.bz2
-rw-r----- 1 root wheel 2845 Mar 9 00:00 daemon.log.4.bz2
-rw------- 1 root wheel 22187 Mar 20 03:45 debug.log
-rw------- 1 root wheel 8193 Mar 19 00:00 debug.log.0.bz2
-rw------- 1 root wheel 6797 Mar 15 00:00 debug.log.1.bz2
-rw------- 1 root wheel 343042 Jul 5 03:01 dmesg.today
-rw------- 1 root wheel 299082 Jul 4 03:01 dmesg.yesterday
-rw-rw-rw- 1 root wheel 669 Jan 8 20:37 iocage.log
-rw-r--r-- 1 root wheel 69 Aug 21 2019 lpd-errs
-rw-r----- 1 root wheel 68 Jul 5 00:00 maillog
-rw-r----- 1 root wheel 125 Jul 5 00:00 maillog.0.bz2
-rw-r----- 1 root wheel 126 Jul 4 00:00 maillog.1.bz2
-rw-r----- 1 root wheel 128 Jul 3 00:00 maillog.2.bz2
-rw-r----- 1 root wheel 124 Jul 2 00:00 maillog.3.bz2
-rw-r----- 1 root wheel 125 Jul 1 00:00 maillog.4.bz2
-rw-r----- 1 root wheel 127 Jun 30 00:00 maillog.5.bz2
-rw-r----- 1 root wheel 126 Jun 29 00:00 maillog.6.bz2
-rw------- 1 root wheel 31294 Feb 13 08:25 mdnsresponder.log
-rw-r----- 1 root wheel 1 Mar 21 14:08 messages