SOLVED Disk Replacement Failed: devices have different sector alignment

[Opti]

Hello,

One of my drives was showing SMART sector errors, so I thought it a good idea to replace it ASAP. I am running 3 2TB drives in a RAID Z1 config. I purchased a new 2TB drive and followed 8.1.10 in the manual for replacing the drive. Everything was going fine, until I booted up with the new drive and tried to click replace. After hitting OK and it failed with the following error:

Code:

Nov 16 18:58:27 freenas manage.py: [middleware.exceptions:38] [MiddlewareError: Disk replacement failed: "cannot replace 7769563133842286815 with /dev/gptid/fbb23b01-8c3b-11e5-b9af-08606e67e88a.eli: devices have different sector alignment, "]

I am running FreeNAS 9.3 64bit Stable 201509282017.

I have tried the suggestions here and here to add sysctls:

Code:

sysctl vfs.zfs.vdev.larger_ashift_minimal=0
sysctl vfs.zfs.vdev.larger_ashift_disable=1

with no luck. It would seem from my searching that this was a problem pre 9.2.1, but I am using 9.3. There is an option to upgrade my ZPOOL from when I upgraded to 9.3 is this a requirement? (it wasn't mentioned in the manual). Very much appreciate any help, I am a little concerned as if any other drives go I will lose my data :(.

 

[Opti]

Some more information, I followed the advice here and executed

Code:

zdb -C | grep ashift

and received

Code:

ashift: 9

Which would indicate that I am running 512byte sectors as opposed to 4k. The same page gives the advice to use the sysctl parameters but that is not working.

 

[Opti]

Interesting bit of information is that when I edit the Sysctls I get the following output in the console:

Code:

Nov 17 18:29:12 freenas notifier: sysctl: unknown oid 'vfs.zfs.vdev.larger_ashift_disable'

I tried replacing in the terminal with:

Code:

zpool replace -f Volume1 7769563133842286815 /dev/ada2

There was no output, however it looks like a resilver is in process, I'll post back if it works. This would point to perhaps a bug in the UI.

Update: The resilvering worked, which means the command line worked but the UI did not. Hope this helps someone else at some stage.

 

[cyberjock]

You've made a grave mistake by setting up your zpool to use the entire disk (/dev/ada2). Read around. You're in trouble and you'll likely learn of your mistake after you've lost your data.

 

[Opti]

Thanks for the warning, could you perhaps elaborate on what the mistake is? or link me to somewhere that explains it.
There does not seem to be a mention of the warning in the FreeNAS user guide storage section. Further Oracle Solaris ZFS administration seems to recommend it:

The recommended mode of operation is to use an entire disk

source: http://docs.oracle.com/cd/E19253-01/819-5461/gazdp/index.html

In fact documentation here specifically recommends against not using the entire disk:

Use whole disks to enable disk write cache and provide easier maintenance. Creating pools on slices adds complexity to disk management and recovery.

Although that would require me to trust Oracle :P.

The only mention to not using the full disk I came across was here and that was referring to HDs being slightly different sizes (although still marketed as a single size, say 2TB) making re-silvering sometimes painful. Is this what you are referring to?

Thanks again for your response, if you could link me to somewhere that explains it in more detail that would be great. Cheers.

 

[Ericloewe]

Although that would require me to trust Oracle :p.

Rarely a good idea. Never a good idea when dealing with FreeNAS.

 

[solarisguy]

Just recreate your pool in 9.3 (9.3.1) - trust me :D. You can stay with pool version 28, if you need to or want to.

I am out of free time for today, but in short: you were caught by the progress happening too fast to create standards. FreeNAS attempts to workaround some of the issues by having a swap space partition in addition to the data partition.

P.S.
Neither Oracle (or Sun before) or FreeNAS (or FreeBSD for that matter) are using the entire disk. It is a visualization shortcut.

 

[solarisguy]

I am not sure why the thread is SOLVED...

@Opti, you hit multiple issues:

• standard used to be 512-byte blocks, then AF came into play; replacing one type of a drive with another is going always to be tricky
• FreeNAS had a bug (fixed only in 9.2.1.6, I think) that the partitions created were not what they should be on disks with native 512-byte blocks
• ZFS using the entire disk does not mean that ZFS is in control of everything from sector 0 to the last sector on the disk, and different operating system handle it differently, so having experience with one OS does not translate into knowledge of another OS internals
• Oracle (and Sun before) and other companies that ship branded hard drives use device configuration overlay (DCO) (cf. https://en.wikipedia.org/wiki/Device_configuration_overlay) or a similar technology to have disks always appearing with the same number of sectors for any nominal size; FreeNAS deals with the issue by allocating some sectors to swap space (and since swap is not ordinarily used, it does not interfere with ZFS operations)
• FreeNAS documentation does not handle fringe cases (nor anybody expects it to)

If recreating you pool is not a good option, you can replace your 2TB drive with any larger drive. However, I am recommending redoing your pool using the current FreeNAS 9.3.1, and not just simply replacing all the disks with larger ones if you were thinking of growing your pool that way.