TrueNAS-SCALE-23.10.0.1 wierdness. A few strange things going on.

[James Gardiner]

Hi,
I would like to support some wired stuff happening on my TrueNAS Scale TrueNAS-SCALE-23.10.0.1
install. Firstly, I have a 12diskz2, store when I list DISKS,

As you can see, 5 of the disks are not showing up as expected.
When I list Disks, disks in the Pool are not being shown as bing in the pool.......
---

root@fs2:/# zpool iostat -v
capacity operations bandwidth
pool alloc free read write read write
---------------------------------------- ----- ----- ----- ----- ----- -----
Pool05 1.17T 64.3T 3 21 449K 2.90M
raidz2-0 1.17T 64.3T 3 21 449K 2.90M
43c0ba98-250e-4443-98f6-b79589ba7a6b - - 0 1 37.3K 248K
ce6ad962-098f-4d84-80a4-2a6cb5baec1b - - 0 1 37.4K 248K
9897b690-13f4-4007-889c-dead00dac6a1 - - 0 1 37.5K 248K
3e2c6137-227b-4531-8e8b-edbf352072cd - - 0 1 37.4K 248K
dd3ffa8c-1468-40ad-81ae-31d5f02427a8 - - 0 1 37.3K 248K
9c798e70-6069-489e-95d8-9e733f4a3796 - - 0 1 37.4K 248K
6b9feec1-468a-4bd5-9dc7-b11d1ff41078 - - 0 4 37.2K 248K
12412ca7-1f69-4ef1-98db-b6002e653ee8 - - 0 1 37.4K 248K
df694c66-a2b0-458e-b54d-420605d7c9b8 - - 0 1 37.5K 248K
605e22ee-991f-40db-a293-c61275615c07 - - 0 1 37.4K 248K
1b8a4183-f23c-4da3-9842-9eccc67fa5c0 - - 0 1 37.4K 248K
09e51d9b-49ff-4921-be67-6436923786b1 - - 0 1 37.4K 248K
cache - - - - - -
ee2d650a-a3de-4b5b-9a4c-88ae138b0ee5 667G 1.21T 0 7 1 1.59M
---------------------------------------- ----- ----- ----- ----- ----- -----
TmpPool2 1.89G 1.86T 0 1 33 11.4K
19e54e57-02ba-4040-9097-b20a5922b38f 1.89G 1.86T 0 1 33 11.4K
---------------------------------------- ----- ----- ----- ----- ----- -----
boot-pool 2.35G 910G 0 2 6.20K 25.8K
sda3 2.35G 910G 0 2 6.20K 25.8K
---------------------------------------- ----- ----- ----- ----- ----- -----
root@fs2:/#

---

Second strangeness. The Reporting, specifically of network data is all over the place. Not showing what is really happening. Totoally broken.
Did kind of work, then it started to show strange data. I disabled a ethernet port, putting it on a different/island subnet to make sure it was not in use.

However, the reporting interface for networking seems to think it still going and traffic is going over it. "ip a" shows its in the isolated subnet.
Scratch my head on this one..

Tried the command line that resets the reporting data, no effect..

midclt call reporting.clear

 

[neik]

Do you still experience this problem?
I had the same issues. After a browser cache reset everything was fine.

 

[James Gardiner]

Went into Chrome debugger and forced a full cache reset on reload. No difference.

I also had some very strange stuff going on with the network devices. I disabled them. And it appeared they were disabled in the OS under "IP a" but it did very much appear like they were still in use. Even the logs showed them in use..

Even now. After setting all but the 10GBe interface with an IP, rebooting and starting some transfers (Backing up out main file server to this new test system..

The Monitor intervase is showing the other no longer setup interfaces. AND one also shows tradffic on it. WHEN ITS NOT EVEN CONFIGURED ANYMORE.

At least I appear to have the transfers utilising the 10GBe interface this time. Its super wierd what i have been seeing, and I don;t really have the time to dig too deep, its seems very deep in terms of whats going on. As general command tools I rely on, where not showing me what I would have expected.

For example,

This does NOT maych up with the monitoring page what appears to think 3 interfaces are active and the network page only shows one configured.

 

[ABain]

there are known issues with the reporting, there are several threads on this. Fix is expected in 23.10.1.
On the storage screen shown, we have been aware of some browser caching issues post upgrade, but to date all have been solved by clearing the cache. Do you have a second browser you could use to verify if the disk issue persists?

 

[James Gardiner]

Just logged in from home on my GamingPC, and it shows a similar issue.
Note the system has been rebooted. Different devices are now being listed. See the image below compared to the previous above.

 

[morganL]

I'd suggest focusing on issue per thread. The storage/disk info issue is unexpected.

Can you provide the detailed hardware info.. CPU, MB, SAS/SATA controllers, Drive types.

Is the pool operating? - I assume so.

 

[James Gardiner]

Yes, the reporting issue seems to be known so let that take its own course to rectification.

The server is a Dell r720 (128gig ram). With a Dell pcie card loaded firmware into IT mode. A Dell 12 disk 2RU unit. with 12x 6TB SAS hard drives.
Yes it is operational. As is seen in the zpool iostat command above.

I can post some dmesg or lspci results if you require it, but Dell r720 I expect is a well known hardware platform.

In testing it against my home PC, I find it interesting that the drive devices have changed (After a reboot is not uncommon) but the error orrurs on the top 4 devices (Was top 5 devices first time) So looks like a parsing error in the truenas code.

 

[morganL]

Yes, the reporting issue seems to be known so let that take its own course to rectification.

The server is a Dell r720 (128gig ram). With a Dell pcie card loaded firmware into IT mode. A Dell 12 disk 2RU unit. with 12x 6TB SAS hard drives.
Yes it is operational. As is seen in the zpool iostat command above.

I can post some dmesg or lspci results if you require it, but Dell r720 I expect is a well known hardware platform.

In testing it against my home PC, I find it interesting that the drive devices have changed (After a reboot is not uncommon) but the error orrurs on the top 4 devices (Was top 5 devices first time) So looks like a parsing error in the truenas code.

I haven't seen anyone else report this... SCALE 23.10.1 is in code freeze for final testing. If it's not fixed there, I'd suggest reporting a bug.
If someone else has the same issue, please let us know.

 

[James Gardiner]

Oh, one other bit of key information.
This pool was created in a previous install, I have a system disk issue and decided to re-install to a different disk and import the pool.
I didn't see this issue when I created it under the previous install.

So the pool was not created under this install. Its imported. Probably a key issue as it may be an issue in importing a pool and it not storing metadata properly.

 

[morganL]

Oh, one other bit of key information.
This pool was created in a previous install, I have a system disk issue and decided to re-install to a different disk and import the pool.
I didn't see this issue when I created it under the previous install.

So the pool was not created under this install. Its imported. Probably a key issue as it may be an issue in importing a pool and it not storing metadata properly.

That shouldn't cause this issue...

 

[ABain]

@James Gardiner as suggested please raise a ticket, a link will be provided to allow you to upload a debug privately on creation of the bug
: https://ixsystems.atlassian.net/jira/software/c/projects/NAS/issues

 

[James Gardiner]

Added to Jira.