TrueNas - input-output error but ZFS is clean

[svoltaire2]

OS: TrueNAS-12.0-U1, command as root directly on the server (ssh)

dmesg doesn't report anything, zfs pool looks ok.

When I try to see the contents of some files, I get an input/output error. (cat, ...)
I can see the content of the folders, other files in the folders work as expected. The issue is spread among many files.

Any idea on what to do? thank you by advance for an hint.

Booting using the DEBUG kernel raises a kernel panic when importing the pool.

Issue is on pool: voluz
root@freenas: # zpool status -v pool: freenas-boot state: ONLINE scan: scrub repaired 0B in 00:00:56 with 0 errors on Tue Jan 5 03:45:56 2021 config: NAME STATE READ WRITE CKSUM freenas-boot ONLINE 0 0 0 ada0p2 ONLINE 0 0 0 errors: No known data errors pool: vollog state: ONLINE scan: scrub repaired 0B in 00:00:43 with 0 errors on Sun Dec 20 00:00:44 2020 config: NAME STATE READ WRITE CKSUM vollog ONLINE 0 0 0 ada1p2 ONLINE 0 0 0 errors: No known data errors pool: voluz state: ONLINE scan: scrub repaired 0B in 14:17:25 with 0 errors on Sun Dec 6 14:17:38 2020 config: NAME STATE READ WRITE CKSUM voluz ONLINE 0 0 0 raidz2-0 ONLINE 0 0 0 da0p2 ONLINE 0 0 0 da1p2 ONLINE 0 0 0 da2p2 ONLINE 0 0 0 da3p2 ONLINE 0 0 0 da4p2 ONLINE 0 0 0 logs nvd0p2 ONLINE 0 0 0 cache nvd0p3 ONLINE 0 0 0 errors: No known data errors

Code:

root@freenas:/mnt/voluz/media/music/Flac/Verve JazzClub Collection (2006-2012) [WwW.LoKoTorrents.CoM]/TRENDS/VA - Electric Bossa (2007) # cat folder.jpg
cat: folder.jpg: Input/output error

 

[Chris Moore]

Hardware details needed. Please see guide:

Forum Rules

Welcome to the TrueNAS Community! Updated 11/28/2023 First and foremost I want to thank you for joining the TrueNAS community! My name is JoshDW19 and I'm the Community Manager here at TrueNAS. The TrueNAS Community is a friendly place to discuss open source software, technology, and products...

www.truenas.com

Here is a link to the guide on BB codes:

BB codes

The list of BB codes you can use to spice up the look of your messages. This page shows a list of all BB codes that are available.

www.truenas.com

You should have used the code tag like this [code] then your listing, then [/code] tag again to close so we could see this properly.

NAME STATE READ WRITE CKSUM
voluz ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
da0p2 ONLINE 0 0 0
da1p2 ONLINE 0 0 0
da2p2 ONLINE 0 0 0
da3p2 ONLINE 0 0 0
da4p2 ONLINE 0 0 0
logs
nvd0p2 ONLINE 0 0 0
cache
nvd0p3 ONLINE 0 0 0

This doesn't make sense because your pool is defined by da# instead of gptid. How did you create your pool?

It should look more like this:

Code:

  pool: Emily
 state: ONLINE
  scan: scrub repaired 0B in 06:21:38 with 0 errors on Tue Dec 15 06:21:40 2020
config:

    NAME                                            STATE     READ WRITE CKSUM
    Emily                                           ONLINE       0     0     0
      raidz2-0                                      ONLINE       0     0     0
        gptid/af7c42c6-bf05-11e8-b5f3-0cc47a9cd5a4  ONLINE       0     0     0
        gptid/b07bc723-bf05-11e8-b5f3-0cc47a9cd5a4  ONLINE       0     0     0
        gptid/b1893397-bf05-11e8-b5f3-0cc47a9cd5a4  ONLINE       0     0     0
        gptid/b2bfc678-bf05-11e8-b5f3-0cc47a9cd5a4  ONLINE       0     0     0
        gptid/b3c1849e-bf05-11e8-b5f3-0cc47a9cd5a4  ONLINE       0     0     0
        gptid/b4d16ad2-bf05-11e8-b5f3-0cc47a9cd5a4  ONLINE       0     0     0
      raidz2-1                                      ONLINE       0     0     0
        gptid/acb0b918-ba5d-11e9-b6dd-00074306773b  ONLINE       0     0     0
        gptid/85d8ab3b-e442-11ea-99b6-00074306773b  ONLINE       0     0     0
        gptid/d1ea0d87-ba96-11e9-b6dd-00074306773b  ONLINE       0     0     0
        gptid/b9de3232-bf05-11e8-b5f3-0cc47a9cd5a4  ONLINE       0     0     0
        gptid/baf4aba8-bf05-11e8-b5f3-0cc47a9cd5a4  ONLINE       0     0     0
        gptid/bbf26621-bf05-11e8-b5f3-0cc47a9cd5a4  ONLINE       0     0     0
    logs   
      gptid/ae487c50-bec3-11e8-b1c8-0cc47a9cd5a4    ONLINE       0     0     0
    cache
      gptid/ae52d59d-bec3-11e8-b1c8-0cc47a9cd5a4    ONLINE       0     0     0

errors: No known data errors

 

[svoltaire2]

It seems I had some issues with the copy/paste as I believed I used the proper /CODE tags; definitely my mistake.

I have some tunables. I am using the mrsas driver (see: https://www.syxin.com/2016/05/freebsd-use-mrsas-driver-to-replace-mfi-driver/); some could be legacy and may need cleanup- I have been using Freenas for many years with no big issue so far.

Hardware details: ESXI (latest 7.x) with LSI PCI passthrough to FreeNas. SLOG /L2ARC is SSD.RAM: 16GB assigned. Load is very low (home lab)

 

[Chris Moore]

It seems I had some issues with the copy/paste as I believed I used the proper /CODE tags; definitely my mistake.

It is the difference between a [code] tag and an [icode] tag.

If you use a [code] tag you get output like this:

Code:

   just a  bit of    junk   for   example

But if you use an [icode] tag

Then you can do somthing like this instead, which is handy for highlighting commands.

I am using the mrsas driver (see: https://www.syxin.com/2016/05/freebsd-use-mrsas-driver-to-replace-mfi-driver/); some could be legacy and may need cleanup- I have been using Freenas for many years with no big issue so far.

Hardware details: ESXI (latest 7.x) with LSI PCI passthrough to FreeNas.

Since FreeNAS is saying your data is fine, I would guess that the problem is in the translation layer between the NAS operating system and the mechanical drive.

 

[Chris Moore]

I am using the mrsas driver

In case you missed that, I am saying you should not have changed the driver, you should have changed the card to one that is natively supported in FreeNAS. It is most likely some problem with your hardware RAID controller that caused your data to be corrupted and now you will have a super difficult time trying to troubleshoot the problem.

 

[Dan Tudora]

you should have changed the card to one that is natively supported in FreeNAS

I vote for that

 

[svoltaire2]

I just ran a zpool scrub, and no issue was found.

Code:

./storcli64 /call show AliLog logfile=raidlog.txt

returns nothing

I am running some checks, but it is as if the issue was only from one dataset (I am scanning all my files now). Could it be an ACL thing (even if I am running everything as root)? I find it a bit strange that there is no issue returned by dmesg. is there a way to do a dtrace or something? I will try to boot an ubuntu with zfs and vanilla freebsd to see if I can see the contents.

Not sure if I should open a bug report but the DEBUG kernel (freenas-advanced menu) crashes for me.

 

[jgreco]

It doesn't matter that RAID controllers *seem* to work. They work until they don't. Then you have a mess.

https://www.truenas.com/community/t...s-and-why-cant-i-use-a-raid-controller.81931/

 

[Chris Moore]

Not sure if I should open a bug report but the DEBUG kernel (freenas-advanced menu) crashes for me.

It might be worth opening a bug report, but the first step would be to test it on bare metal to see if the problem exists outside the virtual machine.

https://jira.ixsystems.com/projects

 

[svoltaire2]

I opened a bug report on openZFS, it seems I am not the only one.

 

[tiberiusQ]

Hopefully this has nothing to do with:

Silent data corruption in TrueNAS 12.0

Hi, can someone from ixsystems please comment on this issue? This would be a catastrophic bug!

www.truenas.com

&

FreeNAS (now TrueNAS) is no longer stable

Hello, I'm writing this with deeply concerns about TrueNAS/FreeNAS and the move that seemed a little bit irresponsible regarding quality and testing. I have three virtualization pools that relied on FreeNAS for years. One specifically is running since 2013, other 2014 and the newer one since...

www.truenas.com