braiam
Dabbler
- Joined
- Nov 28, 2021
- Messages
- 10
Everytime I try to push my LSI SAS2008 card for performance, it starts timing out. I already flashed the controller to v20.0.07.00 in IT mode. These are the kind of messages I'm seeing:
Usually, it recovers, but it leaves my applications hanging for IO. I've seen similar issues here https://www.truenas.com/community/threads/timeout-errors-when-mving-files-to-truenas-server.89820/ but nothing that can point to a specific issue that can be investigated.
Code:
(da2:mps0:0:2:0): READ(16). CDB: 88 00 00 00 00 01 c2 cf 97 a0 00 00 00 08 00 00 length 4096 SMID 1656 Command timeout on target 2(0x000a) 90000 set, 90.135617296 elapsed mps0: Sending abort to target 2 for SMID 1656 (da2:mps0:0:2:0): READ(16). CDB: 88 00 00 00 00 01 c2 cf 97 a0 00 00 00 08 00 00 length 4096 SMID 1656 Aborting command 0xfffffe00c511b140 (da2:mps0:0:2:0): WRITE(10). CDB: 2a 00 85 1f 00 08 00 00 50 00 length 40960 SMID 1440 Command timeout on target 2(0x000a) 90000 set, 90.207423994 elapsed (da3:mps0:0:3:0): WRITE(10). CDB: 2a 00 85 1f 00 08 00 00 50 00 length 40960 SMID 1886 Command timeout on target 3(0x0009) 90000 set, 90.269591921 elapsed mps0: Sending abort to target 3 for SMID 1886 (da3:mps0:0:3:0): WRITE(10). CDB: 2a 00 85 1f 00 08 00 00 50 00 length 40960 SMID 1886 Aborting command 0xfffffe00c512e650 (da4:mps0:0:4:0): WRITE(10). CDB: 2a 00 85 1e ff d0 00 00 30 00 length 24576 SMID 1480 Command timeout on target 4(0x000c) 90000 set, 90.398612076 elapsed mps0: Sending abort to target 4 for SMID 1480 (da4:mps0:0:4:0): WRITE(10). CDB: 2a 00 85 1e ff d0 00 00 30 00 length 24576 SMID 1480 Aborting command 0xfffffe00c510c4c0 (da4:mps0:0:4:0): WRITE(10). CDB: 2a 00 85 1e fe d0 00 01 00 00 length 131072 SMID 1435 Command timeout on target 4(0x000c) 90000 set, 90.526706772 elapsed (da4:mps0:0:4:0): WRITE(10). CDB: 2a 00 85 1e fd d0 00 01 00 00 length 131072 SMID 126 Command timeout on target 4(0x000c) 90000 set, 90.587853050 elapsed (da0:mps0:0:0:0): WRITE(10). CDB: 2a 00 01 f0 44 9f 00 00 01 00 length 512 SMID 1140 Command timeout on target 0(0x000b) 90000 set, 90.121922308 elapsed mps0: Sending abort to target 0 for SMID 1140 (da0:mps0:0:0:0): WRITE(10). CDB: 2a 00 01 f0 44 9f 00 00 01 00 length 512 SMID 1140 Aborting command 0xfffffe00c50efbe0 (xpt0:mps0:0:2:0): SMID 13 task mgmt 0xfffffe00c5091178 timed out mps0: Reinitializing controller mps0: Firmware: 20.00.07.00, Driver: 21.02.00.00-fbsd mps0: IOCCapabilities: 1285c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,HostDisc>
Code:
(da3:mps0:0:3:0): WRITE(10). CDB: 2a 00 5b 36 ac 00 00 01 00 00 length 131072 SMID 2083 Command timeout on target 3(0x0009) 90000 set, 90.246071505 elapsed mps0: Sending abort to target 3 for SMID 2083 (da3:mps0:0:3:0): WRITE(10). CDB: 2a 00 5b 36 ac 00 00 01 00 00 length 131072 SMID 2083 Aborting command 0xfffffe00c513ef08 (da3:mps0:0:3:0): WRITE(10). CDB: 2a 00 5b 36 ab 00 00 01 00 00 length 131072 SMID 750 Command timeout on target 3(0x0009) 90000 set, 90.374417806 elapsed (da3:mps0:0:3:0): WRITE(10). CDB: 2a 00 5b 36 aa 00 00 01 00 00 length 131072 SMID 1523 Command timeout on target 3(0x0009) 90000 set, 90.435550608 elapsed (da3:mps0:0:3:0): WRITE(10). CDB: 2a 00 5b 36 a9 00 00 01 00 00 length 131072 SMID 1189 Command timeout on target 3(0x0009) 90000 set, 90.496875977 elapsed (da3:mps0:0:3:0): WRITE(10). CDB: 2a 00 5b 36 a8 00 00 01 00 00 length 131072 SMID 1548 Command timeout on target 3(0x0009) 90000 set, 90.558443568 elapsed (da3:mps0:0:3:0): WRITE(10). CDB: 2a 00 5b 36 a7 00 00 01 00 00 length 131072 SMID 177 Command timeout on target 3(0x0009) 90000 set, 90.620658793 elapsed (da3:mps0:0:3:0): WRITE(10). CDB: 2a 00 5b 36 a6 00 00 01 00 00 length 131072 SMID 941 Command timeout on target 3(0x0009) 90000 set, 90.681728418 elapsed (da3:mps0:0:3:0): WRITE(10). CDB: 2a 00 5b 36 a5 00 00 01 00 00 length 131072 SMID 1951 Command timeout on target 3(0x0009) 90000 set, 90.742576826 elapsed (da3:mps0:0:3:0): WRITE(10). CDB: 2a 00 5b 36 a4 00 00 01 00 00 length 131072 SMID 98 Command timeout on target 3(0x0009) 90000 set, 90.804019079 elapsed (da3:mps0:0:3:0): WRITE(10). CDB: 2a 00 5b 36 a3 00 00 01 00 00 length 131072 SMID 1820 Command timeout on target 3(0x0009) 90000 set, 90.864894103 elapsed (da2:mps0:0:2:0): WRITE(10). CDB: 2a 00 5b 36 ab 00 00 01 00 00 length 131072 SMID 324 Command timeout on target 2(0x000a) 90000 set, 90.926398938 elapsed mps0: Sending abort to target 2 for SMID 324 (da2:mps0:0:2:0): WRITE(10). CDB: 2a 00 5b 36 ab 00 00 01 00 00 length 131072 SMID 324 Aborting command 0xfffffe00c50ab360 (da2:mps0:0:2:0): WRITE(10). CDB: 2a 00 5b 36 aa 00 00 01 00 00 length 131072 SMID 1889 Command timeout on target 2(0x000a) 90000 set, 90.1053828598 elapsed (da3:mps0:0:3:0): WRITE(10). CDB: 2a 00 5b 36 a2 00 00 01 00 00 length 131072 SMID 1600 Command timeout on target 3(0x0009) 90000 set, 90.1115597697 elapsed (da3:mps0:0:3:0): WRITE(10). CDB: 2a 00 5b 36 a1 00 00 01 00 00 length 131072 SMID 150 Command timeout on target 3(0x0009) 90000 set, 90.1177655506 elapsed (da3:mps0:0:3:0): WRITE(10). CDB: 2a 00 5b 36 a0 00 00 01 00 00 length 131072 SMID 1563 Command timeout on target 3(0x0009) 90000 set, 90.1239169550 elapsed (da4:mps0:0:4:0): WRITE(10). CDB: 2a 00 5b 36 aa 80 00 01 00 00 length 131072 SMID 1842 Command timeout on target 4(0x000c) 90000 set, 90.1301496434 elapsed mps0: Sending abort to target 4 for SMID 1842 (da4:mps0:0:4:0): WRITE(10). CDB: 2a 00 5b 36 aa 80 00 01 00 00 length 131072 SMID 1842 Aborting command 0xfffffe00c512ab30 (da4:mps0:0:4:0): WRITE(10). CDB: 2a 00 5b 36 a9 80 00 01 00 00 length 131072 SMID 1736 Command timeout on target 4(0x000c) 90000 set, 90.1430829791 elapsed (da4:mps0:0:4:0): WRITE(10). CDB: 2a 00 5b 36 a8 80 00 01 00 00 length 131072 SMID 1748 Command timeout on target 4(0x000c) 90000 set, 90.1492591283 elapsed (da2:mps0:0:2:0): WRITE(10). CDB: 2a 00 5b 36 a9 00 00 01 00 00 length 131072 SMID 976 Command timeout on target 2(0x000a) 90000 set, 90.1555067219 elapsed (da2:mps0:0:2:0): WRITE(10). CDB: 2a 00 5b 36 a8 00 00 01 00 00 length 131072 SMID 1199 Command timeout on target 2(0x000a) 90000 set, 90.1616485481 elapsed (da2:mps0:0:2:0): WRITE(10). CDB: 2a 00 5b 36 a7 00 00 01 00 00 length 131072 SMID 2007 Command timeout on target 2(0x000a) 90000 set, 90.1678256014 elapsed (da2:mps0:0:2:0): WRITE(10). CDB: 2a 00 5b 36 a6 00 00 01 00 00 length 131072 SMID 1793 Command timeout on target 2(0x000a) 90000 set, 90.1740030896 elapsed (da4:mps0:0:4:0): WRITE(10). CDB: 2a 00 5b 36 a8 00 00 00 80 00 length 65536 SMID 103 Command timeout on target 4(0x000c) 90000 set, 90.1802795652 elapsed (da4:mps0:0:4:0): WRITE(10). CDB: 2a 00 5b 36 a7 00 00 01 00 00 length 131072 SMID 236 Command timeout on target 4(0x000c) 90000 set, 90.1863678186 elapsed (da4:mps0:0:4:0): WRITE(10). CDB: 2a 00 5b 36 a6 00 00 01 00 00 length 131072 SMID 130 Command timeout on target 4(0x000c) 90000 set, 90.1925108355 elapsed (da4:mps0:0:4:0): WRITE(10). CDB: 2a 00 5b 36 a5 00 00 01 00 00 length 131072 SMID 1569 Command timeout on target 4(0x000c) 90000 set, 90.1986720831 elapsed (da4:mps0:0:4:0): WRITE(10). CDB: 2a 00 5b 36 a4 00 00 01 00 00 length 131072 SMID 1571 Command timeout on target 4(0x000c) 90000 set, 90.2048685759 elapsed (da3:mps0:0:3:0): WRITE(10). CDB: 2a 00 5b 36 9f 80 00 00 80 00 length 65536 SMID 146 Command timeout on target 3(0x0009) 90000 set, 90.2114220373 elapsed (da3:mps0:0:3:0): WRITE(10). CDB: 2a 00 5b 36 9e 80 00 01 00 00 length 131072 SMID 192 Command timeout on target 3(0x0009) 90000 set, 90.-2119645584 elapsed (da3:mps0:0:3:0): WRITE(10). CDB: 2a 00 5b 36 9e 00 00 00 80 00 length 65536 SMID 1309 Command timeout on target 3(0x0009) 90000 set, 90.-2057277883 elapsed (da4:mps0:0:4:0): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00 length 0 SMID 2075 Command timeout on target 4(0x000c) 90000 set, 90.-2015517865 elapsed (da2:mps0:0:2:0): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00 length 0 SMID 1559 Command timeout on target 2(0x000a) 90000 set, 90.-1950535916 elapsed (da3:mps0:0:3:0): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00 length 0 SMID 1141 Command timeout on target 3(0x0009) 90000 set, 90.-1885794373 elapsed (da4:mps0:0:4:0): WRITE(10). CDB: 2a 00 5b 36 ad 80 00 00 80 00 length 65536 SMID 2098 Command timeout on target 4(0x000c) 90000 set, 90.-1810982818 elapsed (da4:mps0:0:4:0): WRITE(10). CDB: 2a 00 5b 36 ac 80 00 01 00 00 length 131072 SMID 1712 Command timeout on target 4(0x000c) 90000 set, 90.-1749184747 elapsed (da4:mps0:0:4:0): WRITE(10). CDB: 2a 00 5b 36 ab 80 00 01 00 00 length 131072 SMID 1784 Command timeout on target 4(0x000c) 90000 set, 90.-1687088677 elapsed (da2:mps0:0:2:0): WRITE(10). CDB: 2a 00 5b 36 ad 00 00 01 00 00 length 131072 SMID 1879 Command timeout on target 2(0x000a) 90000 set, 90.-1624601350 elapsed (da2:mps0:0:2:0): WRITE(10). CDB: 2a 00 5b 36 ac 00 00 01 00 00 length 131072 SMID 1917 Command timeout on target 2(0x000a) 90000 set, 90.-1562668760 elapsed (da3:mps0:0:3:0): WRITE(10). CDB: 2a 00 f3 98 3a 60 00 00 40 00 length 32768 SMID 147 Command timeout on target 3(0x0009) 90000 set, 90.-1523446109 elapsed (da2:mps0:0:2:0): WRITE(10). CDB: 2a 00 f3 98 3a 58 00 00 48 00 length 36864 SMID 1851 Command timeout on target 2(0x000a) 90000 set, 90.-1461970659 elapsed (da4:mps0:0:4:0): WRITE(10). CDB: 2a 00 f3 98 3a 58 00 00 48 00 length 36864 SMID 1530 Command timeout on target 4(0x000c) 90000 set, 90.-1399871977 elapsed (da4:mps0:0:4:0): WRITE(10). CDB: 2a 00 5b 36 b3 b0 00 00 80 00 length 65536 SMID 1279 Command timeout on target 4(0x000c) 90000 set, 90.-1364945410 elapsed (da2:mps0:0:2:0): WRITE(10). CDB: 2a 00 5b 36 b3 b0 00 00 80 00 length 65536 SMID 1785 Command timeout on target 2(0x000a) 90000 set, 90.-1302906753 elapsed (da2:mps0:0:2:0): WRITE(10). CDB: 2a 00 5b 36 b2 78 00 00 80 00 length 65536 SMID 300 Command timeout on target 2(0x000a) 90000 set, 90.-1240767054 elapsed (da3:mps0:0:3:0): WRITE(10). CDB: 2a 00 5b 36 b2 78 00 00 80 00 length 65536 SMID 1832 Command timeout on target 3(0x0009) 90000 set, 90.-1179232102 elapsed (da4:mps0:0:4:0): WRITE(10). CDB: 2a 00 5b 36 b2 70 00 00 80 00 length 65536 SMID 1868 Command timeout on target 4(0x000c) 90000 set, 90.-1117157298 elapsed (da4:mps0:0:4:0): WRITE(10). CDB: 2a 00 5b 36 af 80 00 00 80 00 length 65536 SMID 1816 Command timeout on target 4(0x000c) 90000 set, 90.-1055225799 elapsed (da3:mps0:0:3:0): WRITE(10). CDB: 2a 00 5b 36 af 80 00 00 80 00 length 65536 SMID 1316 Command timeout on target 3(0x0009) 90000 set, 90.-993350472 elapsed (da2:mps0:0:2:0): WRITE(10). CDB: 2a 00 5b 36 af 80 00 00 80 00 length 65536 SMID 1654 Command timeout on target 2(0x000a) 90000 set, 90.-931994975 elapsed (da4:mps0:0:4:0): WRITE(10). CDB: 2a 00 5b 36 af 00 00 00 80 00 length 65536 SMID 1639 Command timeout on target 4(0x000c) 90000 set, 90.-869895079 elapsed (da3:mps0:0:3:0): WRITE(10). CDB: 2a 00 5b 36 af 00 00 00 80 00 length 65536 SMID 1171 Command timeout on target 3(0x0009) 90000 set, 90.-808341213 elapsed (da2:mps0:0:2:0): WRITE(10). CDB: 2a 00 5b 36 af 00 00 00 80 00 length 65536 SMID 1438 Command timeout on target 2(0x000a) 90000 set, 90.-746927912 elapsed (da4:mps0:0:4:0): WRITE(10). CDB: 2a 00 5b 36 ae 80 00 00 80 00 length 65536 SMID 1983 Command timeout on target 4(0x000c) 90000 set, 90.-685211235 elapsed (da3:mps0:0:3:0): WRITE(10). CDB: 2a 00 5b 36 ae 80 00 00 80 00 length 65536 SMID 331 Command timeout on target 3(0x0009) 90000 set, 90.-623481053 elapsed (da2:mps0:0:2:0): WRITE(10). CDB: 2a 00 5b 36 ae 80 00 00 80 00 length 65536 SMID 1829 Command timeout on target 2(0x000a) 90000 set, 90.-562415021 elapsed (da4:mps0:0:4:0): WRITE(10). CDB: 2a 00 5b 36 ae 00 00 00 80 00 length 65536 SMID 2092 Command timeout on target 4(0x000c) 90000 set, 90.-501021227 elapsed (da3:mps0:0:3:0): WRITE(10). CDB: 2a 00 5b 36 ae 00 00 00 80 00 length 65536 SMID 1459 Command timeout on target 3(0x0009) 90000 set, 90.-439434170 elapsed (da2:mps0:0:2:0): WRITE(10). CDB: 2a 00 5b 36 ae 00 00 00 80 00 length 65536 SMID 1819 Command timeout on target 2(0x000a) 90000 set, 90.-377839029 elapsed (da0:mps0:0:0:0): WRITE(10). CDB: 2a 00 01 da f2 8a 00 00 10 00 length 8192 SMID 1437 Command timeout on target 0(0x000b) 90000 set, 90.-1050544238 elapsed mps0: Sending abort to target 0 for SMID 1437 (da0:mps0:0:0:0): WRITE(10). CDB: 2a 00 01 da f2 8a 00 00 10 00 length 8192 SMID 1437 Aborting command 0xfffffe00c5108af8 mps0: Calling Reinit from mps_wait_command, timeout=30, elapsed=30 mps0: Reinitializing controller mps0: Firmware: 20.00.07.00, Driver: 21.02.00.00-fbsd mps0: IOCCapabilities: 1285c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,HostDisc> mps0: Reinit success mps0: mps_user_pass_thru: invalid request: error 60 (da4:mps0:0:4:0): WRITE(10). CDB: 2a 00 5b 3a 7b 80 00 00 80 00 length 65536 SMID 1618 Command timeout on target 4(0x000a) 90000 set, 90.238810847 elapsed mps0: Sending abort to target 4 for SMID 1618 (da4:mps0:0:4:0): WRITE(10). CDB: 2a 00 5b 3a 7b 80 00 00 80 00 length 65536 SMID 1618 Aborting command 0xfffffe00c5117e30 (da4:mps0:0:4:0): WRITE(10). CDB: 2a 00 5b 3a 7a 80 00 01 00 00 length 131072 SMID 975 Command timeout on target 4(0x000a) 90000 set, 90.366451062 elapsed (da4:mps0:0:4:0): WRITE(10). CDB: 2a 00 5b 3a 79 80 00 01 00 00 length 131072 SMID 1182 Command timeout on target 4(0x000a) 90000 set, 90.427712162 elapsed (da4:mps0:0:4:0): WRITE(10). CDB: 2a 00 5b 3a 78 80 00 01 00 00 length 131072 SMID 2061 Command timeout on target 4(0x000a) 90000 set, 90.489198138 elapsed (da4:mps0:0:4:0): WRITE(10). CDB: 2a 00 5b 3a 77 80 00 01 00 00 length 131072 SMID 674 Command timeout on target 4(0x000a) 90000 set, 90.550512085 elapsed (da4:mps0:0:4:0): WRITE(10). CDB: 2a 00 5b 3a 76 80 00 01 00 00 length 131072 SMID 1597 Command timeout on target 4(0x000a) 90000 set, 90.611718379 elapsed (da4:mps0:0:4:0): WRITE(10). CDB: 2a 00 5b 3a 75 80 00 01 00 00 length 131072 SMID 1190 Command timeout on target 4(0x000a) 90000 set, 90.673476846 elapsed (da4:mps0:0:4:0): WRITE(10). CDB: 2a 00 5b 3a 74 80 00 01 00 00 length 131072 SMID 1805 Command timeout on target 4(0x000a) 90000 set, 90.735009195 elapsed (da2:mps0:0:2:0): WRITE(10). CDB: 2a 00 5b 3a 7b 80 00 00 80 00 length 65536 SMID 1125 Command timeout on target 2(0x000b) 90000 set, 90.797470002 elapsed mps0: Sending abort to target 2 for SMID 1125 (da2:mps0:0:2:0): WRITE(10). CDB: 2a 00 5b 3a 7b 80 00 00 80 00 length 65536 SMID 1125 Aborting command 0xfffffe00c50ee7b8 (da2:mps0:0:2:0): WRITE(10). CDB: 2a 00 5b 3a 7a 80 00 01 00 00 length 131072 SMID 1149 Command timeout on target 2(0x000b) 90000 set, 90.925468019 elapsed (da2:mps0:0:2:0): WRITE(10). CDB: 2a 00 5b 3a 79 80 00 01 00 00 length 131072 SMID 1238 Command timeout on target 2(0x000b) 90000 set, 90.986876845 elapsed (da2:mps0:0:2:0): WRITE(10). CDB: 2a 00 5b 3a 78 80 00 01 00 00 length 131072 SMID 1469 Command timeout on target 2(0x000b) 90000 set, 90.1048365154 elapsed (da2:mps0:0:2:0): WRITE(10). CDB: 2a 00 5b 3a 77 80 00 01 00 00 length 131072 SMID 1088 Command timeout on target 2(0x000b) 90000 set, 90.1110251467 elapsed (da2:mps0:0:2:0): WRITE(10). CDB: 2a 00 5b 3a 76 80 00 01 00 00 length 131072 SMID 191 Command timeout on target 2(0x000b) 90000 set, 90.1172172441 elapsed (da2:mps0:0:2:0): WRITE(10). CDB: 2a 00 5b 3a 75 80 00 01 00 00 length 131072 SMID 1053 Command timeout on target 2(0x000b) 90000 set, 90.1233532050 elapsed (da2:mps0:0:2:0): WRITE(10). CDB: 2a 00 5b 3a 74 80 00 01 00 00 length 131072 SMID 1586 Command timeout on target 2(0x000b) 90000 set, 90.1295478910 elapsed (da3:mps0:0:3:0): WRITE(10). CDB: 2a 00 5b 3a 82 80 00 01 00 00 length 131072 SMID 1047 Command timeout on target 3(0x0009) 90000 set, 90.1358649757 elapsed mps0: Sending abort to target 3 for SMID 1047 (da3:mps0:0:3:0): WRITE(10). CDB: 2a 00 5b 3a 82 80 00 01 00 00 length 131072 SMID 1047 Aborting command 0xfffffe00c50e7ee8 (da3:mps0:0:3:0): WRITE(10). CDB: 2a 00 5b 3a 7b 80 00 00 80 00 length 65536 SMID 1339 Command timeout on target 3(0x0009) 90000 set, 90.1488324637 elapsed (da3:mps0:0:3:0): WRITE(10). CDB: 2a 00 5b 3a 7a 80 00 01 00 00 length 131072 SMID 2068 Command timeout on target 3(0x0009) 90000 set, 90.1549818199 elapsed (da3:mps0:0:3:0): WRITE(10). CDB: 2a 00 5b 3a 79 80 00 01 00 00 length 131072 SMID 1341 Command timeout on target 3(0x0009) 90000 set, 90.1611778453 elapsed (da3:mps0:0:3:0): WRITE(10). CDB: 2a 00 5b 3a 78 80 00 01 00 00 length 131072 SMID 278 Command timeout on target 3(0x0009) 90000 set, 90.1673807672 elapsed (da3:mps0:0:3:0): WRITE(10). CDB: 2a 00 5b 3a 77 80 00 01 00 00 length 131072 SMID 1335 Command timeout on target 3(0x0009) 90000 set, 90.1735217121 elapsed (da4:mps0:0:4:0): WRITE(10). CDB: 2a 00 5b 3a 70 80 00 01 00 00 length 131072 SMID 1674 Command timeout on target 4(0x000a) 90000 set, 90.1797431970 elapsed (da4:mps0:0:4:0): WRITE(10). CDB: 2a 00 5b 3a 6f 80 00 01 00 00 length 131072 SMID 88 Command timeout on target 4(0x000a) 90000 set, 90.1859284314 elapsed (da4:mps0:0:4:0): WRITE(10). CDB: 2a 00 5b 3a 6e 80 00 01 00 00 length 131072 SMID 1566 Command timeout on target 4(0x000a) 90000 set, 90.1920340995 elapsed (da3:mps0:0:3:0): WRITE(10). CDB: 2a 00 5b 3a 76 80 00 01 00 00 length 131072 SMID 1096 Command timeout on target 3(0x0009) 90000 set, 90.1982289706 elapsed (da3:mps0:0:3:0): WRITE(10). CDB: 2a 00 5b 3a 75 80 00 01 00 00 length 131072 SMID 1684 Command timeout on target 3(0x0009) 90000 set, 90.2044322129 elapsed (da3:mps0:0:3:0): WRITE(10). CDB: 2a 00 5b 3a 74 80 00 01 00 00 length 131072 SMID 1703 Command timeout on target 3(0x0009) 90000 set, 90.2106329308 elapsed (da2:mps0:0:2:0): WRITE(10). CDB: 2a 00 5b 3a 70 80 00 01 00 00 length 131072 SMID 116 Command timeout on target 2(0x000b) 90000 set, 90.-2125949142 elapsed (da2:mps0:0:2:0): WRITE(10). CDB: 2a 00 5b 3a 6f 80 00 01 00 00 length 131072 SMID 1533 Command timeout on target 2(0x000b) 90000 set, 90.-2064194289 elapsed (da2:mps0:0:2:0): WRITE(10). CDB: 2a 00 5b 3a 6e 80 00 01 00 00 length 131072 SMID 1020 Command timeout on target 2(0x000b) 90000 set, 90.-2001672444 elapsed (da3:mps0:0:3:0): WRITE(10). CDB: 2a 00 5b 3a 70 80 00 01 00 00 length 131072 SMID 1791 Command timeout on target 3(0x0009) 90000 set, 90.-1939288718 elapsed (da3:mps0:0:3:0): WRITE(10). CDB: 2a 00 5b 3a 6f 80 00 01 00 00 length 131072 SMID 1016 Command timeout on target 3(0x0009) 90000 set, 90.-1877002065 elapsed (da4:mps0:0:4:0): WRITE(10). CDB: 2a 00 5b 3a 66 80 00 00 80 00 length 65536 SMID 375 Command timeout on target 4(0x000a) 90000 set, 90.-1810509498 elapsed (da4:mps0:0:4:0): WRITE(10). CDB: 2a 00 5b 3a 6a 80 00 00 80 00 length 65536 SMID 1245 Command timeout on target 4(0x000a) 90000 set, 90.-1748772634 elapsed (da0:mps0:0:0:0): WRITE(10). CDB: 2a 00 01 e1 2c c4 00 00 08 00 length 4096 SMID 1078 Command timeout on target 0(0x000c) 90000 set, 90.-1683978868 elapsed mps0: Sending abort to target 0 for SMID 1078 (da0:mps0:0:0:0): WRITE(10). CDB: 2a 00 01 e1 2c c4 00 00 08 00 length 4096 SMID 1078 Aborting command 0xfffffe00c50ea890 (da4:mps0:0:4:0): WRITE(10). CDB: 2a 00 5b 3a 85 80 00 01 00 00 length 131072 SMID 672 Command timeout on target 4(0x000a) 90000 set, 90.-1572716408 elapsed (da4:mps0:0:4:0): WRITE(10). CDB: 2a 00 5b 3a 84 80 00 01 00 00 length 131072 SMID 2019 Command timeout on target 4(0x000a) 90000 set, 90.-1510530563 elapsed (da4:mps0:0:4:0): WRITE(10). CDB: 2a 00 5b 3a 83 80 00 01 00 00 length 131072 SMID 92 Command timeout on target 4(0x000a) 90000 set, 90.-1448234527 elapsed (da4:mps0:0:4:0): WRITE(10). CDB: 2a 00 5b 3a 82 80 00 01 00 00 length 131072 SMID 1187 Command timeout on target 4(0x000a) 90000 set, 90.-1386584359 elapsed (da2:mps0:0:2:0): WRITE(10). CDB: 2a 00 5b 3a 85 80 00 01 00 00 length 131072 SMID 1909 Command timeout on target 2(0x000b) 90000 set, 90.-1323044348 elapsed (da2:mps0:0:2:0): WRITE(10). CDB: 2a 00 5b 3a 84 80 00 01 00 00 length 131072 SMID 1673 Command timeout on target 2(0x000b) 90000 set, 90.-1260991057 elapsed (da2:mps0:0:2:0): WRITE(10). CDB: 2a 00 5b 3a 83 80 00 01 00 00 length 131072 SMID 163 Command timeout on target 2(0x000b) 90000 set, 90.-1198906067 elapsed (da2:mps0:0:2:0): WRITE(10). CDB: 2a 00 5b 3a 82 80 00 01 00 00 length 131072 SMID 1079 Command timeout on target 2(0x000b) 90000 set, 90.-1136901177 elapsed (da3:mps0:0:3:0): WRITE(10). CDB: 2a 00 5b 3a 85 80 00 01 00 00 length 131072 SMID 1350 Command timeout on target 3(0x0009) 90000 set, 90.-1074021622 elapsed (da3:mps0:0:3:0): WRITE(10). CDB: 2a 00 5b 3a 84 80 00 01 00 00 length 131072 SMID 1812 Command timeout on target 3(0x0009) 90000 set, 90.-1011637399 elapsed (da3:mps0:0:3:0): WRITE(10). CDB: 2a 00 5b 3a 83 80 00 01 00 00 length 131072 SMID 1423 Command timeout on target 3(0x0009) 90000 set, 90.-949366878 elapsed (da4:mps0:0:4:0): WRITE(10). CDB: 2a 00 5b 3a 87 00 00 00 80 00 length 65536 SMID 1751 Command timeout on target 4(0x000a) 90000 set, 90.-1905680586 elapsed (da2:mps0:0:2:0): WRITE(10). CDB: 2a 00 5b 3a 87 00 00 00 80 00 length 65536 SMID 1711 Command timeout on target 2(0x000b) 90000 set, 90.-1843789859 elapsed (da3:mps0:0:3:0): WRITE(10). CDB: 2a 00 5b 3a 87 00 00 00 80 00 length 65536 SMID 961 Command timeout on target 3(0x0009) 90000 set, 90.-1782010647 elapsed (da4:mps0:0:4:0): WRITE(10). CDB: 2a 00 5b 3a 86 80 00 00 80 00 length 65536 SMID 1235 Command timeout on target 4(0x000a) 90000 set, 90.-1720243415 elapsed (da2:mps0:0:2:0): WRITE(10). CDB: 2a 00 5b 3a 86 80 00 00 80 00 length 65536 SMID 63 Command timeout on target 2(0x000b) 90000 set, 90.-1658334681 elapsed (da3:mps0:0:3:0): WRITE(10). CDB: 2a 00 5b 3a 86 80 00 00 80 00 length 65536 SMID 66 Command timeout on target 3(0x0009) 90000 set, 90.-1597355036 elapsed (da2:mps0:0:2:0): WRITE(10). CDB: 2a 00 5b 3a 8f 80 00 00 80 00 length 65536 SMID 1353 Command timeout on target 2(0x000b) 90000 set, 90.-1537330062 elapsed (da3:mps0:0:3:0): WRITE(10). CDB: 2a 00 5b 3a 8f 00 00 00 80 00 length 65536 SMID 71 Command timeout on target 3(0x0009) 90000 set, 90.-1475278184 elapsed (da2:mps0:0:2:0): WRITE(10). CDB: 2a 00 5b 3a 8f 00 00 00 80 00 length 65536 SMID 326 Command timeout on target 2(0x000b) 90000 set, 90.-1414218269 elapsed (da4:mps0:0:4:0): WRITE(10). CDB: 2a 00 5b 3a 8a 80 00 00 80 00 length 65536 SMID 779 Command timeout on target 4(0x000a) 90000 set, 90.-1352373423 elapsed (da4:mps0:0:4:0): WRITE(10). CDB: 2a 00 5b 3a 88 00 00 00 80 00 length 65536 SMID 1541 Command timeout on target 4(0x000a) 90000 set, 90.-1290934493 elapsed (da2:mps0:0:2:0): WRITE(10). CDB: 2a 00 5b 3a 88 00 00 00 80 00 length 65536 SMID 1236 Command timeout on target 2(0x000b) 90000 set, 90.-1228937511 elapsed (da3:mps0:0:3:0): WRITE(10). CDB: 2a 00 5b 3a 8a 80 00 00 80 00 length 65536 SMID 1961 Command timeout on target 3(0x0009) 90000 set, 90.-1166956931 elapsed (da2:mps0:0:2:0): WRITE(10). CDB: 2a 00 5b 3a 8a 80 00 00 80 00 length 65536 SMID 1768 Command timeout on target 2(0x000b) 90000 set, 90.-1105114076 elapsed (da3:mps0:0:3:0): WRITE(10). CDB: 2a 00 5b 3a 88 00 00 00 80 00 length 65536 SMID 1464 Command timeout on target 3(0x0009) 90000 set, 90.-1043317587 elapsed (da4:mps0:0:4:0): WRITE(10). CDB: 2a 00 5b 3a 87 80 00 00 80 00 length 65536 SMID 1830 Command timeout on target 4(0x000a) 90000 set, 90.-980900407 elapsed (da2:mps0:0:2:0): WRITE(10). CDB: 2a 00 5b 3a 87 80 00 00 80 00 length 65536 SMID 1650 Command timeout on target 2(0x000b) 90000 set, 90.-919421545 elapsed (da3:mps0:0:3:0): WRITE(10). CDB: 2a 00 5b 3a 87 80 00 00 80 00 length 65536 SMID 733 Command timeout on target 3(0x0009) 90000 set, 90.-857973306 elapsed (xpt0:mps0:0:4:0): SMID 1 task mgmt 0xfffffe00c5090158 timed out mps0: Reinitializing controller mps0: Firmware: 20.00.07.00, Driver: 21.02.00.00-fbsd mps0: IOCCapabilities: 1285c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,HostDisc> (da4:mps0:0:4:0): READ(16). CDB: 88 00 00 00 00 01 c6 fc 06 58 00 00 00 08 00 00 length 4096 SMID 1698 Command timeout on target 4(0x000a) 90000 set, 90.7424792 elapsed mps0: Sending abort to target 4 for SMID 1698 (da4:mps0:0:4:0): READ(16). CDB: 88 00 00 00 00 01 c6 fc 06 58 00 00 00 08 00 00 length 4096 SMID 1698 Aborting command 0xfffffe00c511e9b0 (da0:mps0:0:0:0): WRITE(10). CDB: 2a 00 01 eb a2 88 00 01 00 00 length 131072 SMID 2100 Command timeout on target 0(0x000b) 90000 set, 90.66333841 elapsed mps0: Sending abort to target 0 for SMID 2100 (da0:mps0:0:0:0): WRITE(10). CDB: 2a 00 01 eb a2 88 00 01 00 00 length 131072 SMID 2100 Aborting command 0xfffffe00c51405e0 (da4:mps0:0:4:0): READ(16). CDB: 88 00 00 00 00 01 21 42 82 08 00 00 00 80 00 00 length 65536 SMID 387 Command timeout on target 4(0x000a) 90000 set, 90.212281186 elapsed (da3:mps0:0:3:0): READ(16). CDB: 88 00 00 00 00 01 21 42 82 10 00 00 00 80 00 00 length 65536 SMID 728 Command timeout on target 3(0x0009) 90000 set, 90.279061399 elapsed mps0: Sending abort to target 3 for SMID 728 (da3:mps0:0:3:0): READ(16). CDB: 88 00 00 00 00 01 21 42 82 10 00 00 00 80 00 00 length 65536 SMID 728 Aborting command 0xfffffe00c50cd240 (da2:mps0:0:2:0): READ(16). CDB: 88 00 00 00 00 01 21 42 80 e0 00 00 00 80 00 00 length 65536 SMID 587 Command timeout on target 2(0x000c) 90000 set, 90.419441905 elapsed mps0: Sending abort to target 2 for SMID 587 (da2:mps0:0:2:0): READ(16). CDB: 88 00 00 00 00 01 21 42 80 e0 00 00 00 80 00 00 length 65536 SMID 587 Aborting command 0xfffffe00c50c14c8 (da4:mps0:0:4:0): READ(16). CDB: 88 00 00 00 00 01 21 42 80 e0 00 00 00 80 00 00 length 65536 SMID 1214 Command timeout on target 4(0x000a) 90000 set, 90.559639039 elapsed (da4:mps0:0:4:0): READ(16). CDB: 88 00 00 00 00 01 26 46 d8 b0 00 00 00 80 00 00 length 65536 SMID 390 Command timeout on target 4(0x000a) 90000 set, 90.116655936 elapsed (da2:mps0:0:2:0): READ(16). CDB: 88 00 00 00 00 01 26 46 d8 b8 00 00 00 80 00 00 length 65536 SMID 485 Command timeout on target 2(0x000c) 90000 set, 90.183680379 elapsed (pass3:mps0:0:3:0): ATA COMMAND PASS THROUGH(16). CDB: 85 06 2c 00 da 00 00 00 00 00 4f 00 c2 00 b0 00 length 0 SMID 763 Command timeout on target 3(0x0009) 60000 set, 60.110738282 elapsed (da2:mps0:0:2:0): WRITE(10). CDB: 2a 00 87 2b 9b 68 00 00 08 00 length 4096 SMID 1161 Command timeout on target 2(0x000c) 90000 set, 90.286155133 elapsed (da3:mps0:0:3:0): WRITE(10). CDB: 2a 00 87 2b 9b 60 00 00 08 00 length 4096 SMID 522 Command timeout on target 3(0x0009) 90000 set, 90.346322166 elapsed (da4:mps0:0:4:0): WRITE(10). CDB: 2a 00 87 2b 9b 60 00 00 08 00 length 4096 SMID 493 Command timeout on target 4(0x000a) 90000 set, 90.406735410 elapsed (xpt0:mps0:0:4:0): SMID 4 task mgmt 0xfffffe00c5090560 timed out mps0: Reinitializing controller mps0: Firmware: 20.00.07.00, Driver: 21.02.00.00-fbsd mps0: IOCCapabilities: 1285c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,HostDisc> mps0: Calling Reinit from mps_wait_command, timeout=60, elapsed=60 mps0: Reinitializing controller mps0: Firmware: 20.00.07.00, Driver: 21.02.00.00-fbsd mps0: IOCCapabilities: 1285c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,HostDisc> mps0: Reinit success mps_config_get_bios_pg3: request for header completed with error 60 mps_user_get_adapter_data: Error while retrieving BIOS Version
Usually, it recovers, but it leaves my applications hanging for IO. I've seen similar issues here https://www.truenas.com/community/threads/timeout-errors-when-mving-files-to-truenas-server.89820/ but nothing that can point to a specific issue that can be investigated.