ZFS Feature Flags in FreeNAS

OpenZFS' distributed development led to the introduction of Feature Flags. Instead of incrementing version numbers, support for OpenZFS features is indicated by Feature Flags.

Feature Flag states

Feature flags exist in one of three states:

disabled
The Feature Flag is not used by the pool. The pool can be imported on systems that do not support this feature flag.

enabled
The feature has been enabled for use in this pool, but no changes are in effect. The pool can be imported on systems that do not support this feature flag.

active
The on-disk format of the pool includes the changes needed for this feature. Some features may allow for the pool to be imported read-only, while others make the pool completely incompatible with systems that do not support the Feature Flag in question.

Note that many ZFS features, such as compressed ARC or sequential scrub/resilver, do not require on-disk format changes. They do not introduce feature flags and pools used with these features are compatible with systems lacking them.



Feature Flags support in FreeNAS

The following Feature Flags are supported in FreeNAS 9.10.2:

• Asynchronous Destroy
• Empty Block Pointer Objects
• LZ4 compression
• Multi-vdev crash dumps
• Spacemap histogram
• Enabled TXG (not user-facing)
• Hole Birth
• Extensible dataset (not user-facing)
• Embedded data in Block Pointers
• Bookmarks
• Filesystem Limits
• Large Blocks (>128KB)

For details on these features, consult the manual page ( man zpool-features).


The following Feature Flags are introduced in FreeNAS 11.0:

• skein
• sha512

Both of these features are optional checksums, of interest to those running deduplication or ZFS native encryption, as they are faster than the default cryptographically-secure sha256. In both cases, the feature is active if any datasets use it. It will return to the enabled state after the dataset is destroyed (data can be preserved by using zfs send | zfs recv to replicate to a dataset which does not use these checksums).
As such, pools can be upgraded while preserving backward compatibility.



The following Feature Flags are introduced in FreeNAS 11.2:

• Device removal
• Obsolete counts
• Zpool checkpoint

Device removal allows for vdevs to be removed from a pool, subject to limitations. This feature flag becomes active when a vdev is removed using the zpool remove subcommand and will never return to enabled. Obsolete counts is an optimization of Device removal and behaves identically.

Zpool checkpoint provides the zpool checkpoint subcommand, which allows for a checkpoint - a sort of super-snapshot of the whole pool capable of reverting even dataset creation and deletion - to be made, subject to significant limitations. The Feature Flag is active while a checkpoint exists and returns to the enabled state after the checkpoint is deleted or the pool is rewound to the checkpoint.



The following Feature Flags are introduced in FreeNAS 11.3:

• Spacemap v2

Spacemap v2 allows for more efficient and flexible encoding of spacemaps. However, this is immediately backwards-incompatible - the feature will become active as soon as it is enabled and will never return to the enabled state. Again, upgrading a pool with feature@spacemap_v2 breaks backwards compatibility. Note that the pool can still be imported read-only for disaster recovery, though.



TrueNAS 12.0 Beta currently supports the following new Feature Flags (Bold feature flags are read-only backwards compatible):

• Allocation Classes
• Bookmarks v2
• Bookmark written
• Encryption
• Large dnodes
• Livelist
• Log Spacemap
• Project Quota
• Redacted datasets
• Redaction bookmarks
• Resilver defer
• Spacemap histogram
• Userobj accounting

Allocation classes allow for vdevs dedicated to specific types of data, namely dedup tables or special (generic metadata). This feature becomes active if a dedup or special dataset is created and reverts to enabled once all such datasets have been destroyed.

V2 Bookmarks are necessary to support a number of other features, such as encryption and redacted send. The Bookmark written Feature Flag adds accounting for space written since a bookmark. Both features can revert to enabled if all v2 Bookmarks are deleted.

Encryption adds support for ZFS native encryption, the details of which are outside the scope of this document. Encryption becomes active once an encrypted dataset is created and can be reverted to enabled by destroying all encrypted datasets

Large dnodes are a performance/storage optimization. However, the large_dnode feature flag breaks zfs send to systems that do not support this flag.

Livelist is a performance optimization for clone deletion.
The feature is activated when a clone is created and remains active until all clones have been destroyed.

Log Spacemap and Spacemap Histogram are both performance optimizations for heavily-fragmented pools. They becomes active as soon as they are enabled and will never return to being enabled.

Project Quota adds support for quotas against project identifiers. It becomes active as soon as it is enabled and will never return to being enabled.

Redaction bookmarks and Redacted datasets allow sending and receiving datasets with redacted data.

Resilver defer allows a resilver to be deferred until after a currently-running resilver has completed, avoiding unwanted restarts of the operation. The feature becomes active once a resilver has been deferred and returns to being enabled when the deferred resilver begins.

Userobj accounting adds support for accounting object user information by user/group. It becomes active as soon as it is enabled and will never return to being enabled.



Overview of commands

To see the Feature Flags supported by the version of ZFS you're running, use man zpool-features.
To view the status of Feature Flags on a pool, use zpool get all poolname | grep feature.
To view available Feature Flags, use zpool upgrade. Feature Flags can be enabled using zpool upgrade poolname.
Feature flags can be selectively enabled at import time with zpool import -o feature@feature_name=enabled poolname. To enable multiple features at once, specify -o feature@feature1=enabled -o feature@feature2=enabled ... for each feature.