iSCSI not mounting after reboot

[orddie]

Running TrueNas scale 22.12.0
new setup.
ESXi 7.03d
two hosts
10GBE networking
ESXi management subnet is 172.16.80.0/24
TrueNas management subnet is 172.16.10.0/24
Storage subnet for both TrueNas and ESXi are 192.168.254.0/24

Everyone can ping one another.

I had the iSCSI lun attached to both hosts overnight and moved some data to them. Took down one server to swap hardware and now it will not mount the LUN. It see's it but will not consume it.

what must I do to get this mounted again?

vmkernal logs

Code:

2023-01-07T19:38:41.482Z cpu12:2097655)DMA: 543: Couldn't get PCI device for props device 'iscsivmkDMAEngine'
2023-01-07T19:38:41.482Z cpu12:2097655)DMA: 687: DMA Engine 'iscsivmkDMAEngine' created using mapper 'DMANull'.
2023-01-07T19:38:41.482Z cpu12:2097655)Device: 395: iscsi_vmk:driver->ops.attachDevice :0 ms
2023-01-07T19:38:41.482Z cpu12:2097655)Device: 400: Found driver iscsi_vmk for device 0xfb04307fce0d992
2023-01-07T19:38:41.482Z cpu12:2097655)Device: 685: iscsi_vmk:driver->ops.startDevice:0 ms
2023-01-07T19:38:41.482Z cpu12:2097655)Device: 1586: Registered device: 0x4307fce01220 logical#logical#swdevroot#com.vmware.iscsi_vmk0#0 com.vmware.HBAPort (parent=0xfb04307fce0d992)
2023-01-07T19:38:41.482Z cpu12:2097655)Device: 507: iscsi_vmk:driver->ops.scanDevice:0 ms
2023-01-07T19:38:41.484Z cpu12:2097655)Device: 395: psa:driver->ops.attachDevice :0 ms
2023-01-07T19:38:41.484Z cpu12:2097655)Device: 400: Found driver psa for device 0x5b214307fce0dbd8
2023-01-07T19:38:41.484Z cpu12:2097655)Device: 685: psa:driver->ops.startDevice:0 ms
2023-01-07T19:38:41.484Z cpu12:2097655)Device: 507: psa:driver->ops.scanDevice:0 ms
2023-01-07T19:38:41.495Z cpu0:2097241)ScsiNpiv: 1522: GetInfo for adapter vmhba64, [0x4305cbe40e80], max_vports=0, vports_inuse=0, linktype=0, state=0, failreason=0, sts=bad0020
2023-01-07T19:38:42.492Z cpu2:2100287 opID=c57ceb47)WARNING: VisorFS: 1094: Attempt to setattr non sticky dir/file from tar mount
2023-01-07T19:38:42.617Z cpu9:2100835)SchedVsi: 2083: Group: host/vim/vmvisor/vmkiscsid(692): min=180 max=180, units: mb
2023-01-07T19:38:58.482Z cpu0:2100837)iscsi_vmk: iscsivmk_ConnNetRegister:1898: socket 0x4313ab841d90 network resource pool netsched.pools.persist.iscsi associated
2023-01-07T19:38:58.482Z cpu0:2100837)iscsi_vmk: iscsivmk_ConnNetRegister:1926: socket 0x4313ab841d90 network tracker id 676229854 tracker.iSCSI.192.168.254.1 associated
2023-01-07T19:38:58.772Z cpu1:2100837)WARNING: iscsi_vmk: iscsivmk_StartConnection:919: vmhba64:CH:0 T:0 CN:0: iSCSI connection is being marked "ONLINE"
2023-01-07T19:38:58.772Z cpu1:2100837)WARNING: iscsi_vmk: iscsivmk_StartConnection:920: Sess [ISID: 00023d000001 TARGET: iqn.2005-10.org.freenas.ctl:truenas TPGT: 1 TSIH: 0]
2023-01-07T19:38:58.772Z cpu1:2100837)WARNING: iscsi_vmk: iscsivmk_StartConnection:921: Conn [CID: 0 L: 192.168.254.5:52187 R: 192.168.254.1:3260]
2023-01-07T19:38:59.260Z cpu0:2097772)NMP: nmp_ThrottleLogForDevice:3867: Cmd 0x0 (0x45b8c199cdc8, 0) to dev "Unregistered Device" on path "vmhba64:C0:T0:L0" Failed:
2023-01-07T19:38:59.260Z cpu0:2097772)NMP: nmp_ThrottleLogForDevice:3875: H:0x0 D:0x2 P:0x0 Valid sense data: 0x6 0x29 0x0. Act:NONE. cmdId.initiator=0x45388739bc88 CmdSN 0x0
2023-01-07T19:38:59.260Z cpu6:2097592)ScsiCore: 1823: Power-on Reset occurred on vmhba64:C0:T0:L0
2023-01-07T19:38:59.262Z cpu6:2097592)StorageApdHandler: 966: APD Handle  Created with lock[StorageApd-0x4305cbe43850]
2023-01-07T19:38:59.262Z cpu6:2097592)PsaStorEvents: 676: Event Subsystem: Device Events, Created!
2023-01-07T19:38:59.262Z cpu6:2097592)PsaStorEvents: 676: Event Subsystem: Device Events - Internal, Created!

working host

non working host