OpenZFS' distributed development led to the introduction of Feature Flags. Instead of incrementing version numbers, support for OpenZFS features is indicated by Feature Flags.
Feature Flag states
Feature flags exist in one of three states:
disabled
The Feature Flag is not used by the pool. The pool can be imported on systems that do not support this feature flag.
enabled
The feature has been enabled for use in this pool, but no changes are in effect. The pool can be imported on systems that do not support this feature flag.
active
The on-disk format of the pool includes the changes needed for this feature. Some features may allow for the pool to be imported read-only, while others make the pool completely incompatible with systems that do not support the Feature Flag in question.
Note that many ZFS features, such as compressed ARC or sequential scrub/resilver, do not require on-disk format changes. They do not introduce feature flags and pools used with these features are compatible with systems lacking them.
Feature Flags support in FreeNAS
The following Feature Flags are supported in FreeNAS 9.10.2:
The following Feature Flags are introduced in FreeNAS 11.0:
As such, pools can be upgraded while preserving backward compatibility.
The following Feature Flags are introduced in FreeNAS 11.2:
Zpool checkpoint provides the
The following Feature Flags are introduced in FreeNAS 11.3:
TrueNAS 12.0 Beta 2.1 currently supports the following new Feature Flags (Bold feature flags are read-only backwards compatible, italicized feature flags are very easy to return to the enabled state):
V2 Bookmarks are necessary to support a number of other features, such as encryption and redacted send. The Bookmark written Feature Flag adds accounting for space written since a bookmark. Both features can revert to enabled if all v2 Bookmarks are deleted.
Sequential Rebuilds (also known as device_rebuild) allow for disks to be reconstructed sequentially by copying from the source disk to the new disk. This process allows for faster recoveries, if the old disk is on its way out. Since this approach precludes verifying checksums, a traditional scrub will be issued after the rebuild is completed to check the pool's integrity. The whole process is longer than a traditional resilver, but does allow for faster recovery.
Encryption adds support for ZFS native encryption, the details of which are outside the scope of this document. Encryption becomes active once an encrypted dataset is created and can be reverted to enabled by destroying all encrypted datasets
Large dnodes are a performance/storage optimization. However, the large_dnode feature flag breaks zfs send to systems that do not support this flag.
Livelist is a performance optimization for clone deletion.
The feature is activated when a clone is created and remains active until all clones have been destroyed.
Log Spacemap is a performance optimization for heavily-fragmented pools. It becomes active as soon as it is enabled and will never return to being enabled.
Project Quota adds support for quotas against project identifiers. It becomes active as soon as it is enabled and will never return to being enabled.
Redaction bookmarks and Redacted datasets allow sending and receiving datasets with redacted data.
Resilver defer allows a resilver to be deferred until after a currently-running resilver has completed, avoiding unwanted restarts of the operation. The feature becomes active once a resilver has been deferred and returns to being enabled when the deferred resilver begins.
Userobj accounting adds support for accounting object user information by user/group. It becomes active as soon as it is enabled and will never return to being enabled.
zstd compression provides very flexible compression. It is typically slower than lz4, but much faster than gzip, and can provide compression ratios better than gzip while still being faster. The features becomes active as soon as a dataset is set to use zstd compression. It will return to the enabled state if all datasets that have used zstd at any point in time are destroyed.
Overview of commands
To see the Feature Flags supported by the version of ZFS you're running, use
To view the status of Feature Flags on a pool, use
To view available Feature Flags, use
Feature flags can be selectively enabled at import time with
Feature Flag states
Feature flags exist in one of three states:
disabled
The Feature Flag is not used by the pool. The pool can be imported on systems that do not support this feature flag.
enabled
The feature has been enabled for use in this pool, but no changes are in effect. The pool can be imported on systems that do not support this feature flag.
active
The on-disk format of the pool includes the changes needed for this feature. Some features may allow for the pool to be imported read-only, while others make the pool completely incompatible with systems that do not support the Feature Flag in question.
Note that many ZFS features, such as compressed ARC or sequential scrub/resilver, do not require on-disk format changes. They do not introduce feature flags and pools used with these features are compatible with systems lacking them.
Feature Flags support in FreeNAS
The following Feature Flags are supported in FreeNAS 9.10.2:
- Asynchronous Destroy
- Empty Block Pointer Objects
- LZ4 compression
- Multi-vdev crash dumps
- Spacemap histogram
- Enabled TXG (not user-facing)
- Hole Birth
- Extensible dataset (not user-facing)
- Embedded data in Block Pointers
- Bookmarks
- Filesystem Limits
- Large Blocks (>128KB)
man zpool-features).The following Feature Flags are introduced in FreeNAS 11.0:
- skein
- sha512
zfs send | zfs recv to replicate to a dataset which does not use these checksums).As such, pools can be upgraded while preserving backward compatibility.
The following Feature Flags are introduced in FreeNAS 11.2:
- Device removal
- Obsolete counts
- Zpool checkpoint
zpool remove subcommand and will never return to enabled. Obsolete counts is an optimization of Device removal and behaves identically.Zpool checkpoint provides the
zpool checkpoint subcommand, which allows for a checkpoint - a sort of super-snapshot of the whole pool capable of reverting even dataset creation and deletion - to be made, subject to significant limitations. The Feature Flag is active while a checkpoint exists and returns to the enabled state after the checkpoint is deleted or the pool is rewound to the checkpoint.The following Feature Flags are introduced in FreeNAS 11.3:
- Spacemap v2
feature@spacemap_v2 breaks backwards compatibility. Note that the pool can still be imported read-only for disaster recovery, though.TrueNAS 12.0 Beta 2.1 currently supports the following new Feature Flags (Bold feature flags are read-only backwards compatible, italicized feature flags are very easy to return to the enabled state):
- Allocation Classes
- Bookmarks v2
- Bookmark written
- Sequential Rebuilds [device_rebuild]
- Encryption
- Large dnodes
- Livelist
- Log Spacemap
- Project Quota
- Redacted datasets
- Redaction bookmarks
- Resilver defer
- Userobj accounting
- zstd compression
V2 Bookmarks are necessary to support a number of other features, such as encryption and redacted send. The Bookmark written Feature Flag adds accounting for space written since a bookmark. Both features can revert to enabled if all v2 Bookmarks are deleted.
Sequential Rebuilds (also known as device_rebuild) allow for disks to be reconstructed sequentially by copying from the source disk to the new disk. This process allows for faster recoveries, if the old disk is on its way out. Since this approach precludes verifying checksums, a traditional scrub will be issued after the rebuild is completed to check the pool's integrity. The whole process is longer than a traditional resilver, but does allow for faster recovery.
Encryption adds support for ZFS native encryption, the details of which are outside the scope of this document. Encryption becomes active once an encrypted dataset is created and can be reverted to enabled by destroying all encrypted datasets
Large dnodes are a performance/storage optimization. However, the large_dnode feature flag breaks zfs send to systems that do not support this flag.
Livelist is a performance optimization for clone deletion.
The feature is activated when a clone is created and remains active until all clones have been destroyed.
Log Spacemap is a performance optimization for heavily-fragmented pools. It becomes active as soon as it is enabled and will never return to being enabled.
Project Quota adds support for quotas against project identifiers. It becomes active as soon as it is enabled and will never return to being enabled.
Redaction bookmarks and Redacted datasets allow sending and receiving datasets with redacted data.
Resilver defer allows a resilver to be deferred until after a currently-running resilver has completed, avoiding unwanted restarts of the operation. The feature becomes active once a resilver has been deferred and returns to being enabled when the deferred resilver begins.
Userobj accounting adds support for accounting object user information by user/group. It becomes active as soon as it is enabled and will never return to being enabled.
zstd compression provides very flexible compression. It is typically slower than lz4, but much faster than gzip, and can provide compression ratios better than gzip while still being faster. The features becomes active as soon as a dataset is set to use zstd compression. It will return to the enabled state if all datasets that have used zstd at any point in time are destroyed.
Overview of commands
To see the Feature Flags supported by the version of ZFS you're running, use
man zpool-features.To view the status of Feature Flags on a pool, use
zpool get all poolname | grep feature.To view available Feature Flags, use
zpool upgrade. Feature Flags can be enabled using zpool upgrade poolname.Feature flags can be selectively enabled at import time with
zpool import -o feature@feature_name=enabled poolname. To enable multiple features at once, specify -o feature@feature1=enabled -o feature@feature2=enabled ... for each feature.
