M.2. NVME SSD on X10SDV-4C-TLN2F - controller reset due timeout

[kitagua]

Hi,
I am still testing with a freenas 11.3 build consisting of a Supermicro X10SDV-4C-TLN2F, 16 GB ECC RAM and a bunch of disks including a M.2. NVME SSD (SK hynx PC-300). The whole systems seems to work and so does the M.2. SSD however I get a ton of ugly log messages (see below). The M.2. SSD is used as a single drive in a pool (no redundancy) and I am wondering if the drive might be defective or if it is a software issue.
I found some discussions and bug-tickets on similar log messages in the forums and on the web outlining issues with timeout coding in freebsd 11 or missing trim() functionalities of the SSD. However those discussions are already some months/years old and are marked as "resolved". I assume, the patches mentioned there have been included in 11.3 release so I don't know if my issues are related to that. I would appreciate if someone can give me a hint on what could be wrong.

Regards,
Kitagua

Code:

Feb 13 10:28:34 freenas nvme0: Resetting controller due to a timeout.
Feb 13 10:28:34 freenas nvme0: resetting controller
Feb 13 10:28:34 freenas nvme0: aborting outstanding admin command
Feb 13 10:28:34 freenas nvme0: GET LOG PAGE (02) sqid:0 cid:10 nsid:ffffffff cdw10:0fef0001 cdw11:00000000
Feb 13 10:28:34 freenas nvme0: ABORTED - BY REQUEST (00/07) sqid:0 cid:10 cdw0:0
Feb 13 10:33:34 freenas nvme0: Resetting controller due to a timeout.
Feb 13 10:33:34 freenas nvme0: resetting controller
Feb 13 10:33:34 freenas nvme0: aborting outstanding admin command
Feb 13 10:33:34 freenas nvme0: GET LOG PAGE (02) sqid:0 cid:15 nsid:ffffffff cdw10:0fef0001 cdw11:00000000
Feb 13 10:33:34 freenas nvme0: ABORTED - BY REQUEST (00/07) sqid:0 cid:15 cdw0:0
Feb 13 10:38:34 freenas nvme0: Resetting controller due to a timeout.
Feb 13 10:38:34 freenas nvme0: resetting controller
Feb 13 10:38:34 freenas nvme0: aborting outstanding admin command
Feb 13 10:38:34 freenas nvme0: GET LOG PAGE (02) sqid:0 cid:10 nsid:ffffffff cdw10:0fef0001 cdw11:00000000
Feb 13 10:38:34 freenas nvme0: ABORTED - BY REQUEST (00/07) sqid:0 cid:10 cdw0:0
Feb 13 10:38:34 freenas nvme0: resubmitting queued i/o
Feb 13 10:38:34 freenas nvme0: DATASET MANAGEMENT sqid:1 cid:0 nsid:1
Feb 13 10:38:34 freenas nvme0: resubmitting queued i/o
Feb 13 10:38:34 freenas nvme0: DATASET MANAGEMENT sqid:1 cid:0 nsid:1
Feb 13 10:38:34 freenas nvme0: resubmitting queued i/o
Feb 13 10:38:34 freenas nvme0: DATASET MANAGEMENT sqid:1 cid:0 nsid:1
Feb 13 10:38:34 freenas nvme0: resubmitting queued i/o
Feb 13 10:38:34 freenas nvme0: DATASET MANAGEMENT sqid:1 cid:0 nsid:1
Feb 13 10:38:34 freenas nvme0: resubmitting queued i/o
Feb 13 10:38:34 freenas nvme0: DATASET MANAGEMENT sqid:1 cid:0 nsid:1
Feb 13 10:38:34 freenas nvme0: resubmitting queued i/o
Feb 13 10:38:34 freenas nvme0: DATASET MANAGEMENT sqid:1 cid:0 nsid:1
Feb 13 10:38:34 freenas nvme0: resubmitting queued i/o
Feb 13 10:38:34 freenas nvme0: DATASET MANAGEMENT sqid:1 cid:0 nsid:1
Feb 13 10:38:34 freenas nvme0: resubmitting queued i/o
Feb 13 10:38:34 freenas nvme0: DATASET MANAGEMENT sqid:1 cid:0 nsid:1
Feb 13 10:38:34 freenas nvme0: resubmitting queued i/o
Feb 13 10:38:34 freenas nvme0: DATASET MANAGEMENT sqid:1 cid:0 nsid:1
Feb 13 10:38:34 freenas nvme0: resubmitting queued i/o
Feb 13 10:38:34 freenas nvme0: DATASET MANAGEMENT sqid:1 cid:0 nsid:1
Feb 13 10:38:34 freenas nvme0: resubmitting queued i/o
Feb 13 10:38:34 freenas nvme0: DATASET MANAGEMENT sqid:1 cid:0 nsid:1
Feb 13 10:38:34 freenas nvme0: resubmitting queued i/o
Feb 13 10:38:34 freenas nvme0: DATASET MANAGEMENT sqid:1 cid:0 nsid:1
Feb 13 10:38:34 freenas nvme0: aborting outstanding i/o
Feb 13 10:43:34 freenas nvme0: Resetting controller due to a timeout.
Feb 13 10:43:34 freenas nvme0: resetting controller
Feb 13 10:43:34 freenas nvme0: aborting outstanding admin command
Feb 13 10:43:34 freenas nvme0: GET LOG PAGE (02) sqid:0 cid:15 nsid:ffffffff cdw10:0fef0001 cdw11:00000000
Feb 13 10:43:34 freenas nvme0: ABORTED - BY REQUEST (00/07) sqid:0 cid:15 cdw0:0
Feb 13 10:48:34 freenas nvme0: Resetting controller due to a timeout.
Feb 13 10:48:34 freenas nvme0: resetting controller
Feb 13 10:48:34 freenas nvme0: aborting outstanding admin command
Feb 13 10:48:34 freenas nvme0: GET LOG PAGE (02) sqid:0 cid:10 nsid:ffffffff cdw10:0fef0001 cdw11:00000000
Feb 13 10:48:34 freenas nvme0: ABORTED - BY REQUEST (00/07) sqid:0 cid:10 cdw0:0
...
[\code]

 

[Patrick M. Hausen]

I have not identical but similar problems as soon as I use more than 4 NVME devices in my pool. We are working on it:

https://jira.ixsystems.com/browse/NAS-104094

Kind regards
Patrick

 

[kitagua]

Ok thank you - I think I already stumbled upon that ticket too but was not realising its "in progress" state.
So I am going to follow that ticket and wait a bit.