Troubleshooting disk format warnings in TrueNAS SCALE 7.4.5

[Sawtaytoes]

Looks like I'm having issues with the `hugo` installation:

Code:

sudo chmod 0755 /usr/bin/apt*
curl -JLO https://www.truenas.com/community/resources/hugo.203/download
unzip ./hugo-7.4.5.zip
sudo apt-get update
sudo apt-get -y install libncurses5 libncurses5-dev
sudo dpkg -i ./hugo/v7.4.5/HUGO-7.4.5.x86_64.deb
sudo chmod 0644 /usr/bin/apt*

...

0 upgraded, 32 newly installed, 0 to remove and 3 not upgraded.
Need to get 0 B/15.4 MB of archives.
After this operation, 64.6 MB of additional disk space will be used.
Extracting templates from packages: 100%
Preconfiguring packages ...
Could not exec dpkg!
E: Sub-process /usr/bin/dpkg returned an error code (100)
sudo: dpkg: command not found

Do I not have the ability to run dpkg? Do I have to be a different user?

Even running as root without `sudo`, I get:
> -bash: /usr/bin/dpkg: Permission denied

 

[CoolWolf]

Looks like I'm having issues with the `hugo` installation:

Code:

sudo chmod 0755 /usr/bin/apt*
curl -JLO https://www.truenas.com/community/resources/hugo.203/download
unzip ./hugo-7.4.5.zip
sudo apt-get update
sudo apt-get -y install libncurses5 libncurses5-dev
sudo dpkg -i ./hugo/v7.4.5/HUGO-7.4.5.x86_64.deb
sudo chmod 0644 /usr/bin/apt*

...

0 upgraded, 32 newly installed, 0 to remove and 3 not upgraded.
Need to get 0 B/15.4 MB of archives.
After this operation, 64.6 MB of additional disk space will be used.
Extracting templates from packages: 100%
Preconfiguring packages ...
Could not exec dpkg!
E: Sub-process /usr/bin/dpkg returned an error code (100)
sudo: dpkg: command not found

Do I not have the ability to run dpkg? Do I have to be a different user?

Even running as root without `sudo`, I get:
> -bash: /usr/bin/dpkg: Permission denied

You need to give the same 0755 permissions to /usr/bin/dpkg as well (same as you did for /usr/bin/apt*) .. and then revert both back to 0644 once you have installed HUGO.

 

[MR.B]

I have a disk with DIF error. When trying to format the disk, I get this error:

error opening device file: /dev/sdbr: Read-only file system


How to remove read-only from a disk?

 

[Sawtaytoes]

You need to give the same 0755 permissions to /usr/bin/dpkg as well (same as you did for /usr/bin/apt*) .. and then revert both back to 0644 once you have installed HUGO.

Thanks! That worked. Posting here for historical reasons:

Code:

sudo chmod 0755 /usr/bin/apt*
sudo chmod 0755 /usr/bin/dpkg
curl -JLO https://www.truenas.com/community/resources/hugo.203/download
unzip ./hugo-7.4.5.zip
sudo apt-get update
sudo apt-get -y install libncurses5 libncurses5-dev
sudo dpkg -i ./hugo/v7.4.5/HUGO-7.4.5.x86_64.deb
sudo chmod 0644 /usr/bin/apt*
sudo chmod 0644 /usr/bin/dpkg

 

[Sawtaytoes]

I tried it on one drive just to see. Yep. It does work! If you cancel it mid-way, you'll need to power-cycle the drive to get it working again.

Code:

# BEFORE
User Capacity:        9,796,820,402,176 bytes [9.79 TB]
Logical block size:   512 bytes
Physical block size:  4096 bytes

# AFTER
User Capacity:        10,000,831,348,736 bytes [10.0 TB]
Logical block size:   4096 bytes

Not only did it work, but I gained a ton of extra space just like the 4Kn 10TB HGST drives I have :).

Using `--simple-progress` was a bad idea because I couldn't see what was happening.

Here's what I did:

Code:

tmux new-session -ds sdap
tmux send-keys -t sdap 'time hugo format --danger-zone -b 4096 -p 0 -g /dev/sdap' Enter
tmux capture-pane -pt sdap

 

[Sawtaytoes]

I used HUGO to remove the T10 type 1 protection and format to 4Kn in a single command:

sudo hugo format --danger-zone --simple-progress -b 4096 -p 0 -g /dev/sdq

The -q option allows to specify multiple devices so you can format multiple drives at the same time.

I couldn't figure out how to get `-q` working, but I did use `-m` to my advantage:

Code:

hugo format --danger-zone -m "H7210A520SUN010T" --fastformat -b 4096 -p 0

Is there any reason we're using a full format normally? Using ChatGPT, it looked up HGST stuff on STH Forums, and they're all recommending the `--fastformat` option. It was done in seconds rather than 13 hours.

 

[Daisuke]

You need to give the same 0755 permissions to /usr/bin/dpkg as well (same as you did for /usr/bin/apt*) .. and then revert both back to 0644 once you have installed HUGO.

I updated the Hugo resource with the missing information, thank you.

 

[Daisuke]

Is there any reason we're using a full format normally?

Every time I used --fastformat, it corrupted some of the sectors once the disk was back online. I tested that on 3 disks, then I had to redo the procedure again. I explained all that, into OP HGST format procedure section. Yes it is long, but I wanted to have a 100% pristine disk and not worry about bad sectors. If you do a S.M.A.R.T. long scan, I'm pretty sure it will throw errors. Please let me know if that is the case. In my case, I was performing the format on unmounted disks from an existing pool (one format at the time), it might be a different case for you. You can format multiple disks at once, so it will not increase the spent time formatting the disks.

I have a JBOD with 32 disks that I will attach to TrueNAS and test the fast format on few disks that we're never attached to a pool, then create a pool with these disks and see if I get sector errors like I previously did. I know from experience, the recommended approach is a slow format.

@Ericloewe please feel free to chime in, since you have more experience.

 

[Daisuke]

I have a disk with DIF error. When trying to format the disk, I get this error:

error opening device file: /dev/sdbr: Read-only file system


How to remove read-only from a disk?

What's the output for sg_map, is the disk unmounted (not part of pool)? What command you used to format the disk?

 

[Daisuke]

I used HUGO to remove the T10 type 1 protection and format to 4Kn in a single command:

sudo hugo format --danger-zone --simple-progress -b 4096 -p 0 -g /dev/sdq

The -q option allows to specify multiple devices so you can format multiple drives at the same time.

Refer to included PDF file or run hugo format -h for all command flags, for some reason man hugo is missing some of them, -q option do not exist. It is probably a typo, I'm sure you meant -g. :)

 

[Daisuke]

I couldn't figure out how to get `-q` working, but I did use `-m` to my advantage

@CoolWolf made a typo, you can specify -g <device path> multiple times, instead of model. If you have the same model attached to a pool, Hugo will detect them and try to format them also. Use hugo show to see all disks info. Personally, I always issue format commands as root user, when I format disks. Commands example, executed as root user:

Code:

tmux new-session -ds multiformat
tmux send-keys -t multiformat 'hugo format --danger-zone --simple-progress -b 4096 -p 0 -g /dev/sda -g /dev/sdb' Enter
tmux capture-pane -pt multiformat

Once the format finished, kill the tmux session:

Code:

# tmux kill-session -t multiformat

Always use sg_map to make sure you have the correct path and disk name.

 

[Daisuke]

Using `--simple-progress` was a bad idea because I couldn't see what was happening.

If you skip it, it will fill the tmux buffers and you will not be able to see the end result, eventually, when you run tmux capture-pane. That's the reason why I added it. In other words, you will end-up with a stuck tmux screen, not knowing when the format is finished.

If you use fast format, this will not be an issue, obviously.

 

[Sawtaytoes]

If you skip it, it will fill the tmux buffers and you will not be able to see the end result, eventually, when you run tmux capture-pane. That's the reason why I added it. In other words, you will end-up with a stuck tmux screen, not knowing when the format is finished.

If you use fast format, this will not be an issue, obviously.

That didn't happen for me (buffer overflow). It ran fine on 2 machines with 30, 12, and 24 drives I ran based on the model number. It also told me how long it took to complete the operation with `time`.

 

[Sawtaytoes]

Every time I used --fastformat, it corrupted some of the sectors once the disk was back online. I tested that on 3 disks, then I had to redo the procedure again. I explained all that, into OP HGST format procedure section. Yes it is long, but I wanted to have a 100% pristine disk and not worry about bad sectors. If you do a S.M.A.R.T. long scan, I'm pretty sure it will throw errors. Please let me know if that is the case. In my case, I was performing the format on unmounted disks from an existing pool (one format at the time), it might be a different case for you. You can format multiple disks at once, so it will not increase the spent time formatting the disks.

I have a JBOD with 32 disks that I will attach to TrueNAS and test the fast format on few disks that we're never attached to a pool, then create a pool with these disks and see if I get sector errors like I previously did. I know from experience, the recommended approach is a slow format.

@Ericloewe please feel free to chime in, since you have more experience.

Ah, that scares me. I felt all the blood rush out of me when you said there could be sector corruption because I just removed one of the zpools with on another machine that had a bunch of photos and videos after copying it to a new zpool with hugo fast-formatted drives.

Those photos and videos appear to be working fine thankfully!

 

[Daisuke]

That didn't happen for me (buffer overflow).

Did you used the --fastformat option? If you did, then is normal not to have buffer issues.

 

[Sawtaytoes]

Did you used the --fastformat option? If you did, then is normal not to have buffer issues.

That might be why. I did use --fast-format.

 

[Daisuke]

Those photos and videos appear to be working fine thankfully!

Did you scanned the disks after formatting to make sure they are error free? It does not matter your files are okay.

 

[Sawtaytoes]

Did you scanned the disks after formatting to make sure they are error free? It does not matter your files are okay.

How would I scan the disks? You mean with a scrub operation or SMART tests?

 

[Daisuke]

You mean with a scrub operation or SMART tests?

Long SMART test.

 

[Sawtaytoes]

Long SMART test.

I've done that on a few disks, and saw SUCCESS messages in TrueNAS.