gptzfsboot: error 128 lba <some block #> (prints these errors on boot but eventually [after ~30 mins] boots)

P.R.

Cadet
Joined
Jan 13, 2020
Messages
7
Hello,

My freeNAS exhibits the following weirdness:

- on boot, it prints several "gptzfsboot: error 128 lba <some block #>" on boot (several for each data disk), prior to running the BTX loader
- after that it prints "BTX loader 1.00 BTX version is 1.02
Consoles: internal video/keyboard
BIOS drive C: is disk0
BIOS drive D: is disk1
BIOS drive E: is disk2
BIOS drive F: is disk3
BIOS drive G: is disk4
read 1 from 0 to 0xce1d3230, error: 0x80
read 1 from 0 to 0xce1d3230, error: 0x80
read 1 from 0 to 0xce3df270, error: 0x80
read 1 from 0 to 0xce3df270, error: 0x80
read 1 from 0 to 0xce5eb2b0, error: 0x80
read 1 from 0 to 0xce5eb2b0, error: 0x80
read 1 from 0 to 0xce7f72f0, error: 0x80
<... few more lines like above ...>"

After that the freeNAS boots normally, but the above takes ~ 30 minutes before that.

Observations:
- the system worked normally with freeNAS in the past - with the same hardware / same type of HDDs
- the error appeared after I added new drives and updated the freeNAS to last version (but the error persists even when doing a clean install with any of the 4 previous versions of freeNAS, and without those added hard drives)
- same error when I move boot & data drives to a different PC
- errors disappear when data disks are detached
- with different data disk (8TB instead of the original 12TB), the error number changes to "gptzfsboot: error 32 lba <some block #>"
- creating a GPT partition table / partitions on a new data drive does not help, deleting partitions / partition tables
- system works normally (including the data hard drives) e.g. with gparted

My understanding is that gptzfsboot, upon being loaded from my first PATA SSD (boot drive set in BIOS) tries to locate BTX loader on all other present disks - at that stage the "gptzfsboot: error 128 lba <some block #>" are produced and this takes a very long time. Then (no other BTX loaders found) the system boots.

Can anyone help, please / is there a way to tell gptzfsboot *not* to search for BTX loader on other disks?

Thanks for any help!

Hardware: IBM x320043635BY, Xeon 3050 @ 2.13 GHz, 8GB ECC RAM, bios v. 1.42, system disk: Transcend 32GB 2.5 PATA SSD (TS32GPSD330), data disks: WD/HGST 12TB SATA (HUH721212ALN600)

Software/settings: boot from BIOS (boot from UEFI does not work), tried freeNAS 11.2_U4 up to U7 (clean install)
 

Attachments

  • IMG_2003.JPG
    IMG_2003.JPG
    270.6 KB · Views: 618

eshwayri

Dabbler
Joined
May 29, 2016
Messages
18
I am having the same exact problem after upgrading to 11.2U7. I get a large number of gptzfsboot error 128 lba errors for 30 to 45 min before the NAS boots normally. It wasn't doing this before. The boot drives are a mirrored pair of 100GB IntelSSDs. This isn't a media issue. I have read this is occurring because gptzfsboot is scanning all the drives. I assume some behavior has changed. How do I fix this so I don't have to wait 45 minutes for the system to boot.
 

Attachments

  • IMG_5714.JPG
    IMG_5714.JPG
    296 KB · Views: 468
  • IMG_5713.JPG
    IMG_5713.JPG
    301.9 KB · Views: 429

P.R.

Cadet
Joined
Jan 13, 2020
Messages
7
I tried to ask for a solution on the freebsd hackers mailing list. So far the only relevant info I was told was that there is no option to tell freebsd's (freenas') gptzfsboot not to scan all drives - that behavior is hard-coded in gptzfsboot. I was given the advice to make an enhancement request to introduce such option in freebsd (which I have not done yet).
In my case the problem started with 11.2U7, but reverting to older version then did not remove the problem. I fail to see a mechanism by which 11.2U7 could have changed something in a permanent way (even if I erased partitions / partition tables on the disks causing the problem, the problem still remains present not only with 11.2U7 but also with earlier, previously well working, freenas versions).
 

eshwayri

Dabbler
Joined
May 29, 2016
Messages
18
I tried to ask for a solution on the freebsd hackers mailing list. So far the only relevant info I was told was that there is no option to tell freebsd's (freenas') gptzfsboot not to scan all drives - that behavior is hard-coded in gptzfsboot. I was given the advice to make an enhancement request to introduce such option in freebsd (which I have not done yet).
In my case the problem started with 11.2U7, but reverting to older version then did not remove the problem. I fail to see a mechanism by which 11.2U7 could have changed something in a permanent way (even if I erased partitions / partition tables on the disks causing the problem, the problem still remains present not only with 11.2U7 but also with earlier, previously well working, freenas versions).

The mechanism must be 11.2U7 updated the gptzfsboot boot loader, and it exists before we get to zfs, so irrespective of which older boot environment you try to use, the new code gets executed first. In Linux terms they upgraded grub, and it doesn't matter which kernel you tell grub to boot to, still the same grub starting first. That's the only thing that makes sense. It's time to open a bug report here. Somone at iXsystems must know what they updated in the boot loader.
 

P.R.

Cadet
Joined
Jan 13, 2020
Messages
7
The mechanism must be 11.2U7 updated the gptzfsboot boot loader, and it exists before we get to zfs, so irrespective of which older boot environment you try to use, the new code gets executed first. In Linux terms they upgraded grub, and it doesn't matter which kernel you tell grub to boot to, still the same grub starting first. That's the only thing that makes sense. It's time to open a bug report here. Somone at iXsystems must know what they updated in the boot loader.
If I recall correctly, I tried to install previous versions from scratch and the problem still remained (which makes no sense to me). I can try one more time later this week, erasing the system disk completely (removing partitions and partition table as well) to be sure that nothing from 11.2U7 could have remained.
Did you manage to revert to the situation without the gptzfsboot errors on your system?
 

eshwayri

Dabbler
Joined
May 29, 2016
Messages
18
I have not tried booting back into U6 since my reading and understanding of this issue is that it occurs prior to the boot environment being touched. Somewhere someone said they tried it, and it made no difference. Since U7 is working once booted, I am loathed to bring down the system for what would likely be a 2 hour+ booting adventure. Since both my VMWare and oVirt infrastructure rely on the NAS, I would need to pretty much bring every server down. I have opened a bug report, and would like some more clarity about what could be causing this before spending extended downtime testing.
 

P.R.

Cadet
Joined
Jan 13, 2020
Messages
7
Thanks for opening the bug report. Can you, please, keep this thread updated should you have resolved the issue / learn new info on the subject?
 

eshwayri

Dabbler
Joined
May 29, 2016
Messages
18
I worked around the issue on my install, but I don't know if it will work for you. My boot disks and ZIL disks are on the system SATA bus; however, my 7 x 6TB data disk pool are on an LSI HBA. What I did was erase and re-flash the LSI HBA with ONLY the firmware (didn't write the BIOS rom back to the card). Now when the system boots (pre-FreeNAS environment), the data disks cannot be seen so it quickly boots.
 

P.R.

Cadet
Joined
Jan 13, 2020
Messages
7
This is not an option for me - my data disks are connected to SATA ports on the motherboard.
The problem is still present on FreeNAS 11.3 and 11.3-U1.
 

blanchet

Guru
Joined
Apr 17, 2018
Messages
515
The best option is to revert to FreeNAS 11.1u7 (which uses GRUB instead of BTX)

Do not try to desperately tune the BIOS to support BTX or you may brick the motherboard
 

evillone

Explorer
Joined
May 17, 2014
Messages
60
im having the same issue, just updated to 11.2-U8
 

Attachments

  • 20200709_232629.jpg
    20200709_232629.jpg
    197.6 KB · Views: 528

P.R.

Cadet
Joined
Jan 13, 2020
Messages
7
Welcome to the club. No solution so far, developers (iXsystems / freeBSD) don't care to solve this. So I am basically waiting for freeNAS to switch to Linux (AFAIK planned by iXsystems), where this problem is not present :-(.
 

Squigley

Dabbler
Joined
May 13, 2020
Messages
19
Just commenting to say that I am seeing the same behaviour on a fresh TrueNAS 12.0 install. Thankfully it doesn't delay the boot process by more than the second or two while it prints all the errors, but it's still an issue there..
 
Top