rniedzial
Cadet
- Joined
- Apr 25, 2023
- Messages
- 7
Long time lurker, first time poster. I've got a problem with my LSI-9300-8e seeings disks attached to a Dell PowerVault MD1400. The LSI is flashed w/ IT firmware and running inside of a Dell R730. I also have a Dell PERC H810 in the R730 and I've confirmed that it does see the drives when in iDRAC (however because this is a RAID card they don't show up in the OS, which I believe is the expected behavior).
I've tried 3 separate pairs of cables now, w/ the latest ones being a pair of 10Gtek 12G External Mini SAS HD SFF-8644 to SFF-8644. So I think I got it narrowed down to either user error or this LSI card having issues. I got it off ebay so ...
I guess my question is, should my setup work? :) Here is output from a few of the troubleshooting commands:
I'm booted into Ubuntu Server 23.04 right now. I have a bunch of SSD disks in there and some NVME drives attached directly to the R730, no other drives show up.
Flashed to latest 16.00.12 firmware from Truenas forums.
I've tried 3 separate pairs of cables now, w/ the latest ones being a pair of 10Gtek 12G External Mini SAS HD SFF-8644 to SFF-8644. So I think I got it narrowed down to either user error or this LSI card having issues. I got it off ebay so ...
I guess my question is, should my setup work? :) Here is output from a few of the troubleshooting commands:
I'm booted into Ubuntu Server 23.04 right now. I have a bunch of SSD disks in there and some NVME drives attached directly to the R730, no other drives show up.
Code:
sudo lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS loop0 7:0 0 73M 1 loop /snap/core22/607 loop1 7:1 0 163M 1 loop /snap/lxd/24643 loop2 7:2 0 164.3M 1 loop /snap/lxd/24758 loop3 7:3 0 49.8M 1 loop /snap/snapd/18596 loop4 7:4 0 53.2M 1 loop /snap/snapd/18933 sda 8:0 0 894.3G 0 disk ├─sda1 8:1 0 1G 0 part /boot/efi ├─sda2 8:2 0 2G 0 part /boot └─sda3 8:3 0 891.2G 0 part └─ubuntu--vg-ubuntu--lv 253:0 0 100G 0 lvm / sdb 8:16 0 931.5G 0 disk sdc 8:32 0 931.5G 0 disk sdd 8:48 0 931.5G 0 disk sde 8:64 0 931.5G 0 disk sdf 8:80 0 931.5G 0 disk sdg 8:96 0 931.5G 0 disk sdh 8:112 0 931.5G 0 disk sdi 8:128 0 931.5G 0 disk sdj 8:144 0 931.5G 0 disk sdk 8:160 0 931.5G 0 disk sdl 8:176 0 931.5G 0 disk sr0 11:0 1 1024M 0 rom nvme0n1 259:0 0 931.5G 0 disk ├─nvme0n1p1 259:1 0 1022M 0 part ├─nvme0n1p2 259:2 0 4G 0 part ├─nvme0n1p3 259:3 0 922.5G 0 part └─nvme0n1p4 259:4 0 4G 0 part nvme1n1 259:5 0 931.5G 0 disk └─nvme1n1p1 259:6 0 931.5G 0 part nvme2n1 259:7 0 931.5G 0 disk └─nvme2n1p1 259:9 0 931.5G 0 part nvme3n1 259:8 0 931.5G 0 disk └─nvme3n1p1 259:10 0 931.5G 0 part
Flashed to latest 16.00.12 firmware from Truenas forums.
Code:
sudo sas3flash -list Avago Technologies SAS3 Flash Utility Version 16.00.00.00 (2017.05.02) Copyright 2008-2017 Avago Technologies. All rights reserved. Adapter Selected is a Avago SAS: SAS3008(C0) Controller Number : 0 Controller : SAS3008(C0) PCI Address : 00:82:00:00 SAS Address : 500605b-0-0388-bdb0 NVDATA Version (Default) : 0e.01.00.08 NVDATA Version (Persistent) : 0e.01.00.08 Firmware Product ID : 0x2221 (IT) Firmware Version : 16.00.12.00 NVDATA Vendor : LSI NVDATA Product ID : SAS9300-8e BIOS Version : 08.29.00.00 UEFI BSD Version : N/A FCODE Version : N/A Board Name : SAS9300-8e Board Assembly : H3-25460-02B Board Tracer Number : SL301P1372 Finished Processing Commands Successfully. Exiting SAS3Flash.
Code:
sudo sas3ircu 0 display Avago Technologies SAS3 IR Configuration Utility. Version 17.00.00.00 (2018.04.02) Copyright (c) 2009-2018 Avago Technologies. All rights reserved. Read configuration has been initiated for controller 0 ------------------------------------------------------------------------ Controller information ------------------------------------------------------------------------ Controller type : SAS3008 BIOS version : 8.29.00.00 Firmware version : 16.00.12.00 Channel description : 1 Serial Attached SCSI Initiator ID : 0 Maximum physical devices : 1023 Concurrent commands supported : 9664 Slot : 5 Segment : 0 Bus : 130 Device : 0 Function : 0 RAID Support : No ------------------------------------------------------------------------ IR Volume information ------------------------------------------------------------------------ ------------------------------------------------------------------------ Physical device information ------------------------------------------------------------------------ ------------------------------------------------------------------------ Enclosure information ------------------------------------------------------------------------ Enclosure# : 1 Logical ID : 500605b0:0388bdb0 Numslots : 8 StartSlot : 0 ------------------------------------------------------------------------ SAS3IRCU: Command DISPLAY Completed Successfully. SAS3IRCU: Utility Completed Successfully.
Code:
sudo dmesg | grep mpt3 [ 9.012033] mpt3sas version 43.100.00.00 loaded [ 9.019273] mpt3sas_cm0: 63 BIT PCI BUS DMA ADDRESSING SUPPORTED, total mem (396117172 kB) [ 9.083772] mpt3sas_cm0: CurrentHostPageSize is 0: Setting default host page size to 4k [ 9.083808] mpt3sas_cm0: MSI-X vectors supported: 96 [ 9.083819] mpt3sas_cm0: 0 56 56 [ 9.085185] mpt3sas_cm0: High IOPs queues : disabled [ 9.085190] mpt3sas0-msix0: PCI-MSI-X enabled: IRQ 229 [ 9.085193] mpt3sas0-msix1: PCI-MSI-X enabled: IRQ 230 [ 9.085195] mpt3sas0-msix2: PCI-MSI-X enabled: IRQ 231 [ 9.085198] mpt3sas0-msix3: PCI-MSI-X enabled: IRQ 232 [ 9.085200] mpt3sas0-msix4: PCI-MSI-X enabled: IRQ 233 [ 9.085202] mpt3sas0-msix5: PCI-MSI-X enabled: IRQ 234 [ 9.085205] mpt3sas0-msix6: PCI-MSI-X enabled: IRQ 235 [ 9.085207] mpt3sas0-msix7: PCI-MSI-X enabled: IRQ 236 [ 9.085210] mpt3sas0-msix8: PCI-MSI-X enabled: IRQ 237 [ 9.085212] mpt3sas0-msix9: PCI-MSI-X enabled: IRQ 238 [ 9.085214] mpt3sas0-msix10: PCI-MSI-X enabled: IRQ 239 [ 9.085217] mpt3sas0-msix11: PCI-MSI-X enabled: IRQ 240 [ 9.085219] mpt3sas0-msix12: PCI-MSI-X enabled: IRQ 241 [ 9.085222] mpt3sas0-msix13: PCI-MSI-X enabled: IRQ 242 [ 9.085224] mpt3sas0-msix14: PCI-MSI-X enabled: IRQ 243 [ 9.085227] mpt3sas0-msix15: PCI-MSI-X enabled: IRQ 244 [ 9.085229] mpt3sas0-msix16: PCI-MSI-X enabled: IRQ 245 [ 9.085231] mpt3sas0-msix17: PCI-MSI-X enabled: IRQ 246 [ 9.085233] mpt3sas0-msix18: PCI-MSI-X enabled: IRQ 247 [ 9.085236] mpt3sas0-msix19: PCI-MSI-X enabled: IRQ 248 [ 9.085238] mpt3sas0-msix20: PCI-MSI-X enabled: IRQ 249 [ 9.085240] mpt3sas0-msix21: PCI-MSI-X enabled: IRQ 250 [ 9.085242] mpt3sas0-msix22: PCI-MSI-X enabled: IRQ 251 [ 9.085244] mpt3sas0-msix23: PCI-MSI-X enabled: IRQ 252 [ 9.085247] mpt3sas0-msix24: PCI-MSI-X enabled: IRQ 253 [ 9.085249] mpt3sas0-msix25: PCI-MSI-X enabled: IRQ 254 [ 9.085251] mpt3sas0-msix26: PCI-MSI-X enabled: IRQ 255 [ 9.085253] mpt3sas0-msix27: PCI-MSI-X enabled: IRQ 256 [ 9.085256] mpt3sas0-msix28: PCI-MSI-X enabled: IRQ 257 [ 9.085258] mpt3sas0-msix29: PCI-MSI-X enabled: IRQ 258 [ 9.085260] mpt3sas0-msix30: PCI-MSI-X enabled: IRQ 259 [ 9.085263] mpt3sas0-msix31: PCI-MSI-X enabled: IRQ 260 [ 9.085265] mpt3sas0-msix32: PCI-MSI-X enabled: IRQ 261 [ 9.085267] mpt3sas0-msix33: PCI-MSI-X enabled: IRQ 262 [ 9.085269] mpt3sas0-msix34: PCI-MSI-X enabled: IRQ 263 [ 9.085272] mpt3sas0-msix35: PCI-MSI-X enabled: IRQ 264 [ 9.085274] mpt3sas0-msix36: PCI-MSI-X enabled: IRQ 265 [ 9.085276] mpt3sas0-msix37: PCI-MSI-X enabled: IRQ 266 [ 9.085278] mpt3sas0-msix38: PCI-MSI-X enabled: IRQ 267 [ 9.085281] mpt3sas0-msix39: PCI-MSI-X enabled: IRQ 268 [ 9.085283] mpt3sas0-msix40: PCI-MSI-X enabled: IRQ 269 [ 9.085285] mpt3sas0-msix41: PCI-MSI-X enabled: IRQ 270 [ 9.085287] mpt3sas0-msix42: PCI-MSI-X enabled: IRQ 271 [ 9.085289] mpt3sas0-msix43: PCI-MSI-X enabled: IRQ 272 [ 9.085292] mpt3sas0-msix44: PCI-MSI-X enabled: IRQ 273 [ 9.085294] mpt3sas0-msix45: PCI-MSI-X enabled: IRQ 274 [ 9.085296] mpt3sas0-msix46: PCI-MSI-X enabled: IRQ 275 [ 9.085298] mpt3sas0-msix47: PCI-MSI-X enabled: IRQ 276 [ 9.085300] mpt3sas0-msix48: PCI-MSI-X enabled: IRQ 277 [ 9.085302] mpt3sas0-msix49: PCI-MSI-X enabled: IRQ 278 [ 9.085305] mpt3sas0-msix50: PCI-MSI-X enabled: IRQ 279 [ 9.085307] mpt3sas0-msix51: PCI-MSI-X enabled: IRQ 280 [ 9.085309] mpt3sas0-msix52: PCI-MSI-X enabled: IRQ 281 [ 9.085311] mpt3sas0-msix53: PCI-MSI-X enabled: IRQ 282 [ 9.085314] mpt3sas0-msix54: PCI-MSI-X enabled: IRQ 283 [ 9.085316] mpt3sas0-msix55: PCI-MSI-X enabled: IRQ 284 [ 9.085318] mpt3sas_cm0: iomem(0x00000000c8140000), mapped(0x00000000f036b9dd), size(65536) [ 9.085324] mpt3sas_cm0: ioport(0x0000000000009000), size(256) [ 9.142647] mpt3sas_cm0: CurrentHostPageSize is 0: Setting default host page size to 4k [ 9.170460] mpt3sas_cm0: scatter gather: sge_in_main_msg(1), sge_per_chain(7), sge_per_io(128), chains_per_io(19) [ 9.176216] mpt3sas_cm0: request pool(0x00000000484c9321) - dma(0x30b2800000): depth(9756), frame_size(128), pool_size(1219 kB) [ 9.371305] mpt3sas_cm0: sense pool(0x00000000852af475) - dma(0x30b4800000): depth(9567), element_size(96), pool_size (896 kB) [ 9.371339] mpt3sas_cm0: sense pool(0x00000000852af475)- dma(0x30b4800000): depth(9567),element_size(96), pool_size(0 kB) [ 9.371943] mpt3sas_cm0: reply pool(0x0000000088b641ca) - dma(0x30b2a00000): depth(9820), frame_size(128), pool_size(1227 kB) [ 9.371981] mpt3sas_cm0: config page(0x0000000000e3b207) - dma(0x30b47c9000): size(512) [ 9.371996] mpt3sas_cm0: Allocated physical memory: size(35896 kB) [ 9.372006] mpt3sas_cm0: Current Controller Queue Depth(9564),Max Controller Queue Depth(9664) [ 9.372019] mpt3sas_cm0: Scatter Gather Elements per IO(128) [ 9.542673] mpt3sas_cm0: _base_display_fwpkg_version: complete [ 9.544018] mpt3sas_cm0: LSISAS3008: FWVersion(16.00.12.00), ChipRevision(0x02), BiosVersion(08.29.00.00) [ 9.544060] mpt3sas_cm0: Protocol=(Initiator,Target), Capabilities=(TLR,EEDP,Snapshot Buffer,Diag Trace Buffer,Task Set Full,NCQ) [ 9.552690] mpt3sas_cm0: sending port enable !! [ 12.097128] mpt3sas_cm0: hba_port entry: 00000000fa42a608, port: 255 is added to hba_port list [ 12.098336] mpt3sas_cm0: host_add: handle(0x0001), sas_addr(0x500605b00388bdb0), phys(8) [ 17.229185] mpt3sas_cm0: port enable: SUCCESS rafal@730:~$ sudo shutdown -h now Broadcast message from root@730 on pts/1 (Mon 2023-04-24 00:25:21 UTC): The system will power off now! Connection to 192.168.0.70 closed by remote host. Connection to 192.168.0.70 closed. rniedzialkowski ~/Downloads/17.05.06.00_MSM_Linux-x64 took 12m42s at 2023-04-23 17:25 ❯❯ ssh rafal@192.168.0.70 rafal@192.168.0.70's password: rafal@730:~$ sudo dmesg | grep mpt3 [sudo] password for rafal: [ 8.864653] mpt3sas version 43.100.00.00 loaded [ 8.871923] mpt3sas_cm0: 63 BIT PCI BUS DMA ADDRESSING SUPPORTED, total mem (396117116 kB) [ 8.941169] mpt3sas_cm0: CurrentHostPageSize is 0: Setting default host page size to 4k [ 8.941195] mpt3sas_cm0: MSI-X vectors supported: 96 [ 8.941200] mpt3sas_cm0: 0 56 56 [ 8.942502] mpt3sas_cm0: High IOPs queues : disabled [ 8.942507] mpt3sas0-msix0: PCI-MSI-X enabled: IRQ 229 [ 8.942509] mpt3sas0-msix1: PCI-MSI-X enabled: IRQ 230 [ 8.942512] mpt3sas0-msix2: PCI-MSI-X enabled: IRQ 231 [ 8.942514] mpt3sas0-msix3: PCI-MSI-X enabled: IRQ 232 [ 8.942517] mpt3sas0-msix4: PCI-MSI-X enabled: IRQ 233 [ 8.942519] mpt3sas0-msix5: PCI-MSI-X enabled: IRQ 234 [ 8.942521] mpt3sas0-msix6: PCI-MSI-X enabled: IRQ 235 [ 8.942524] mpt3sas0-msix7: PCI-MSI-X enabled: IRQ 236 [ 8.942526] mpt3sas0-msix8: PCI-MSI-X enabled: IRQ 237 [ 8.942529] mpt3sas0-msix9: PCI-MSI-X enabled: IRQ 238 [ 8.942531] mpt3sas0-msix10: PCI-MSI-X enabled: IRQ 239 [ 8.942533] mpt3sas0-msix11: PCI-MSI-X enabled: IRQ 240 [ 8.942536] mpt3sas0-msix12: PCI-MSI-X enabled: IRQ 241 [ 8.942538] mpt3sas0-msix13: PCI-MSI-X enabled: IRQ 242 [ 8.942541] mpt3sas0-msix14: PCI-MSI-X enabled: IRQ 243 [ 8.942543] mpt3sas0-msix15: PCI-MSI-X enabled: IRQ 244 [ 8.942545] mpt3sas0-msix16: PCI-MSI-X enabled: IRQ 245 [ 8.942547] mpt3sas0-msix17: PCI-MSI-X enabled: IRQ 246 [ 8.942549] mpt3sas0-msix18: PCI-MSI-X enabled: IRQ 247 [ 8.942552] mpt3sas0-msix19: PCI-MSI-X enabled: IRQ 248 [ 8.942554] mpt3sas0-msix20: PCI-MSI-X enabled: IRQ 249 [ 8.942556] mpt3sas0-msix21: PCI-MSI-X enabled: IRQ 250 [ 8.942558] mpt3sas0-msix22: PCI-MSI-X enabled: IRQ 251 [ 8.942560] mpt3sas0-msix23: PCI-MSI-X enabled: IRQ 252 [ 8.942562] mpt3sas0-msix24: PCI-MSI-X enabled: IRQ 253 [ 8.942565] mpt3sas0-msix25: PCI-MSI-X enabled: IRQ 254 [ 8.942567] mpt3sas0-msix26: PCI-MSI-X enabled: IRQ 255 [ 8.942569] mpt3sas0-msix27: PCI-MSI-X enabled: IRQ 256 [ 8.942572] mpt3sas0-msix28: PCI-MSI-X enabled: IRQ 257 [ 8.942574] mpt3sas0-msix29: PCI-MSI-X enabled: IRQ 258 [ 8.942576] mpt3sas0-msix30: PCI-MSI-X enabled: IRQ 259 [ 8.942578] mpt3sas0-msix31: PCI-MSI-X enabled: IRQ 260 [ 8.942581] mpt3sas0-msix32: PCI-MSI-X enabled: IRQ 261 [ 8.942583] mpt3sas0-msix33: PCI-MSI-X enabled: IRQ 262 [ 8.942585] mpt3sas0-msix34: PCI-MSI-X enabled: IRQ 263 [ 8.942587] mpt3sas0-msix35: PCI-MSI-X enabled: IRQ 264 [ 8.942589] mpt3sas0-msix36: PCI-MSI-X enabled: IRQ 265 [ 8.942592] mpt3sas0-msix37: PCI-MSI-X enabled: IRQ 266 [ 8.942594] mpt3sas0-msix38: PCI-MSI-X enabled: IRQ 267 [ 8.942596] mpt3sas0-msix39: PCI-MSI-X enabled: IRQ 268 [ 8.942598] mpt3sas0-msix40: PCI-MSI-X enabled: IRQ 269 [ 8.942600] mpt3sas0-msix41: PCI-MSI-X enabled: IRQ 270 [ 8.942603] mpt3sas0-msix42: PCI-MSI-X enabled: IRQ 271 [ 8.942605] mpt3sas0-msix43: PCI-MSI-X enabled: IRQ 272 [ 8.942607] mpt3sas0-msix44: PCI-MSI-X enabled: IRQ 273 [ 8.942609] mpt3sas0-msix45: PCI-MSI-X enabled: IRQ 274 [ 8.942611] mpt3sas0-msix46: PCI-MSI-X enabled: IRQ 275 [ 8.942613] mpt3sas0-msix47: PCI-MSI-X enabled: IRQ 276 [ 8.942616] mpt3sas0-msix48: PCI-MSI-X enabled: IRQ 277 [ 8.942618] mpt3sas0-msix49: PCI-MSI-X enabled: IRQ 278 [ 8.942620] mpt3sas0-msix50: PCI-MSI-X enabled: IRQ 279 [ 8.942622] mpt3sas0-msix51: PCI-MSI-X enabled: IRQ 280 [ 8.942624] mpt3sas0-msix52: PCI-MSI-X enabled: IRQ 281 [ 8.942627] mpt3sas0-msix53: PCI-MSI-X enabled: IRQ 282 [ 8.942629] mpt3sas0-msix54: PCI-MSI-X enabled: IRQ 283 [ 8.942631] mpt3sas0-msix55: PCI-MSI-X enabled: IRQ 284 [ 8.942633] mpt3sas_cm0: iomem(0x00000000c8140000), mapped(0x0000000074bf9f23), size(65536) [ 8.942639] mpt3sas_cm0: ioport(0x0000000000009000), size(256) [ 9.000397] mpt3sas_cm0: CurrentHostPageSize is 0: Setting default host page size to 4k [ 9.027780] mpt3sas_cm0: scatter gather: sge_in_main_msg(1), sge_per_chain(7), sge_per_io(128), chains_per_io(19) [ 9.033290] mpt3sas_cm0: request pool(0x00000000956d16c8) - dma(0x3096600000): depth(9756), frame_size(128), pool_size(1219 kB) [ 9.198079] mpt3sas_cm0: sense pool(0x0000000003e3e945) - dma(0x309a500000): depth(9567), element_size(96), pool_size (896 kB) [ 9.198109] mpt3sas_cm0: sense pool(0x0000000003e3e945)- dma(0x309a500000): depth(9567),element_size(96), pool_size(0 kB) [ 9.198671] mpt3sas_cm0: reply pool(0x000000001e26e595) - dma(0x3098800000): depth(9820), frame_size(128), pool_size(1227 kB) [ 9.198708] mpt3sas_cm0: config page(0x000000008d557e8b) - dma(0x309a4d7000): size(512) [ 9.198721] mpt3sas_cm0: Allocated physical memory: size(35896 kB) [ 9.198731] mpt3sas_cm0: Current Controller Queue Depth(9564),Max Controller Queue Depth(9664) [ 9.198743] mpt3sas_cm0: Scatter Gather Elements per IO(128) [ 9.368903] mpt3sas_cm0: _base_display_fwpkg_version: complete [ 9.370609] mpt3sas_cm0: LSISAS3008: FWVersion(16.00.12.00), ChipRevision(0x02), BiosVersion(08.29.00.00) [ 9.370646] mpt3sas_cm0: Protocol=(Initiator,Target), Capabilities=(TLR,EEDP,Snapshot Buffer,Diag Trace Buffer,Task Set Full,NCQ) [ 9.379081] mpt3sas_cm0: sending port enable !! [ 11.940275] mpt3sas_cm0: hba_port entry: 0000000083cf4c8a, port: 255 is added to hba_port list [ 11.941518] mpt3sas_cm0: host_add: handle(0x0001), sas_addr(0x500605b00388bdb0), phys(8) [ 17.078205] mpt3sas_cm0: port enable: SUCCESS