Skip to navigation
Reference

Feature flags

View as Markdown

Feature flags

Feature flags are represented as a 64-bit bitmask field in aistore cluster configuration denoting assorted (named) capabilities. Most take effect at runtime; flags marked restart required take effect only after the affected nodes restart.

The features themselves are enumerated below. Not all feature flags - henceforth, “features” - are cluster-global.

Assorted features, denoted by (*) below, can also be changed on a per-bucket basis.

By default, all features are disabled, and the corresponding 64-bit field is set to zero.

Table of Contents

Tagging system

Feature flags are now organized with descriptive tags to help users understand their purpose and impact. The CLI displays features in a 3-column format: FEATURE | TAGS | DESCRIPTION.

Tag categories

  • Domain-specific: s3, lz4, etl, mpu, telemetry
  • Performance impact: perf, overhead
  • Network/operations: net, ops, security
  • Compatibility: compat, promote
  • Deployment: deploy

Integrity impact indicators

Feature flags that affect data integrity are marked with directional indicators:

  • integrity+ - Enhances data integrity (safer, more conservative)
  • integrity- - Potentially compromises data integrity (trading safety for performance)
  • integrity? - Complex integrity implications (depends on deployment and runtime context)

This helps operators quickly identify flags that involve safety trade-offs.

Validation and conflicts

Feature flags now include validation logic to prevent conflicting configurations:

  • Disable-Cold-GET and Streaming-Cold-GET are mutually exclusive
  • Additional validation rules may be added for other conflicting combinations

The validation occurs both at the cluster level and when setting bucket properties.

Names and comments

nametagscomment
Skip-Loading-VersionChecksum-MD(*)perf,integrity-skip loading existing object’s metadata, Version and Checksum (VC) in particular
Do-not-Auto-Detect-FileSharepromote,opsdo not auto-detect file share (NFS, SMB) when promoting shared files to AIS
S3-API-via-Roots3,compat,opshandle S3 requests via aistore-hostname/ (whereby the default: aistore-hostname/s3)
Fsync-PUT(*)integrity+,overheadPUT and cold-GET: commit (or sync) the object payload to stable storage
LZ4-Block-1MBlz4.tar.lz4 format, lz4 compression: maximum uncompressed block size=1MB (default: 256K)
LZ4-Frame-Checksumlz4checksum lz4 frames
Do-not-Allow-Passing-FQN-to-ETLetl,securitydo not allow passing fully-qualified name of a locally stored object to (local) ETL containers
Ignore-LimitedCoexistence-Conflictsops,integrity-run in presence of “limited coexistence” type conflicts
S3-Presigned-Request(*)s3,security,compatpass-through client-signed (presigned) S3 requests for subsequent authentication by S3
Do-not-Optimize-Listing-Virtual-Dirsoverheadwhen prefix doesn’t end with ’/’ and is a subdirectory: don’t assume there are no prefixed object names
Disable-Cold-GETperf,integrity-do not perform cold GET request when using remote bucket
Streaming-Cold-GETperf,integrity-write and transmit cold-GET content back to user in parallel, without finalizing in-cluster object
S3-Reverse-Proxys3,net,opsuse reverse proxy calls instead of HTTP-redirect for S3 API
S3-Use-Path-Styles3,compatuse older path-style addressing (as opposed to virtual-hosted style), e.g., https://s3.amazonaws.com/BUCKET/KEY
Do-not-Delete-When-Rebalancingintegrity?,opsdisable lazy deletion during global rebalance: do not delete misplaced sources of the migrated objects
Do-not-Set-Control-Plane-ToSnet,opsintra-cluster control plane: use default network priority (do not set IPv4 ToS to low-latency)
Trust-Crypto-Safe-Checksumsintegrity+,overheadwhen checking whether objects are identical trust only cryptographically secure checksums
S3-ListObjectVersionss3,overheadwhen versioning info is requested, use ListObjectVersions API (beware: extremely slow, versioned S3 buckets only)
Enable-Detailed-Prom-Metricstelemetry,overheadinclude (bucket, xaction) Prometheus variable labels with every GET and PUT transaction
Force-Container-CPU-Memdeployforce container-based CPU and memory metrics when automated environment detection fails (restart required)
Resume-Interrupted-MPUmpu,opsresume interrupted multipart uploads from persisted partial manifests
Keep-Unknown-FQNintegrity?,opsdo not delete unrecognized/invalid FQNs during space cleanup (‘ais space-cleanup’)
Load-Balance-GETperfwhen bucket is n-way mirrored read object replica from the least-utilized mountpath
Count-Object-NotFound-Statstelemetry,opscount GET(object) 404 as errors (default: don’t)
Enable-Go-Runtime-Metricstelemetry,ops,overheadpublish a low-cardinality subset of Go runtime metrics (goroutines, GC, heap) via Prometheus
Dload-Allow-Private-Egresssecurity-allow downloader egress to private RFC1918/ULA addresses; loopback and link-local remain blocked
S3-Redirect-Rebuilds3,compat,security-allow S3 clients that rebuild redirected requests instead of following the Location URI (forbidden when AuthN or intra-cluster signing is configured)
System-Reserved-KTLSperf,net,ops,compatoffload TLS transmit path to the kernel and enable Linux sendfile (reserved for internal use; may be redefined or removed at any time) (restart required)
S3-TensorFlow-Querys3,compat,integrity-legacy TensorFlow workaround: interpret ? in decoded S3 object paths as query parameters; breaks object keys containing ?
Disable-GET-Deadlineops,security-GET(object): disable client write and backend read deadlines - i.e., the 64 KiB/s minimum transfer rate per window (timeout.send_file_time); other timeouts remain in force. See Minimum transfer rate. Stalled transfers can then hold object locks, open files, and goroutines indefinitely

Global features

$ ais config cluster features <TAB-TAB>
Do-not-Optimize-Listing-Virtual-Dirs   Force-Container-CPU-Mem
Skip-Loading-VersionChecksum-MD        Disable-Cold-GET                       Resume-Interrupted-MPU
Do-not-Auto-Detect-FileShare           Streaming-Cold-GET                     Keep-Unknown-FQN
S3-API-via-Root                        S3-Reverse-Proxy                       Load-Balance-GET
Fsync-PUT                              S3-Use-Path-Style                      Count-Object-NotFound-Stats
LZ4-Block-1MB                          Do-not-Delete-When-Rebalancing         Enable-Go-Runtime-Metrics
LZ4-Frame-Checksum                     Do-not-Set-Control-Plane-ToS           Dload-Allow-Private-Egress
Do-not-Allow-Passing-FQN-to-ETL        Trust-Crypto-Safe-Checksums            S3-Redirect-Rebuild
Ignore-LimitedCoexistence-Conflicts    S3-ListObjectVersions                  none
S3-Presigned-Request                   Enable-Detailed-Prom-Metrics           System-Reserved-KTLS
S3-TensorFlow-Query                    Disable-GET-Deadline

For example:

$ ais config cluster features S3-API-via-Root  Skip-Loading-VersionChecksum-MD  Load-Balance-GET
PROPERTY         VALUE
features         Skip-Loading-VersionChecksum-MD
                 S3-API-via-Root
                 Load-Balance-GET
FEATURE                              TAGS                   DESCRIPTION
Skip-Loading-VersionChecksum-MD      perf,integrity-        (*) skip loading existing object's metadata, Version and Checksum (VC) in particular              <<< colored
Do-not-Auto-Detect-FileShare         promote,ops            do not auto-detect file share (NFS, SMB) when _promoting_ shared files to AIS
S3-API-via-Root                      s3,compat,ops          handle s3 requests via `aistore-hostname/` (default: `aistore-hostname/s3`)                       <<< colored
Fsync-PUT                            integrity+,overhead    (*) when finalizing PUT(object): fflush prior to (close, rename) sequence
LZ4-Block-1MB                        lz4                    .tar.lz4 format, lz4 compression: max uncompressed block size=1MB (default: 256K)
LZ4-Frame-Checksum                   lz4                    checksum lz4 frames (default: don't)
Do-not-Allow-Passing-FQN-to-ETL      etl,security           do not allow passing fully-qualified name of a locally stored object to (local) ETL containers
Ignore-LimitedCoexistence-Conflicts  ops,integrity-         run in presence of _limited coexistence_ type conflicts (same as e.g. CopyBckMsg.Force but globally)
S3-Presigned-Request                 s3,security,compat     (*) pass-through client-signed (presigned) S3 requests for subsequent authentication by S3
Do-not-Optimize-Listing-Virtual-Dirs overhead               when prefix doesn't end with '/' and is a subdirectory: don't assume there are no _prefixed_ obj names
Disable-Cold-GET                     perf,integrity-        disable cold-GET (from remote bucket)
Streaming-Cold-GET                   perf,integrity-        write and transmit cold-GET content back to user in parallel, without _finalizing_ in-cluster object
S3-Reverse-Proxy                     s3,net,ops             intra-cluster communications: instead of regular HTTP redirects reverse-proxy S3 API calls to designated targets
S3-Use-Path-Style                    s3,compat              use older path-style addressing (as opposed to virtual-hosted style), e.g., `https://s3.amazonaws.com/BUCKET/KEY`
Do-not-Delete-When-Rebalancing       integrity?,ops         disable lazy deletion during global rebalance: do not delete misplaced sources of the migrated objects
Do-not-Set-Control-Plane-ToS         net,ops                intra-cluster control plane: use default network priority (do not set IPv4 ToS to low-latency)
Trust-Crypto-Safe-Checksums          integrity+,overhead    when checking whether objects are identical trust only cryptographically secure checksums
S3-ListObjectVersions                s3,overhead            when versioning info is requested, use ListObjectVersions API (beware: extremely slow, versioned S3 buckets only)
Enable-Detailed-Prom-Metrics         telemetry,overhead     include (bucket, xaction) Prometheus variable labels with every GET and PUT transaction
Force-Container-CPU-Mem              deploy                 force container-based CPU and memory metrics when automated environment detection fails (startup only)
Resume-Interrupted-MPU               mpu,ops                resume interrupted multipart uploads from persisted partial manifests
Keep-Unknown-FQN                     integrity?,ops         do not delete unrecognized/invalid FQNs during space cleanup ('ais space-cleanup')
Load-Balance-GET                     perf                   when bucket is n-way mirrored read object replica from the least-utilized mountpath               <<< colored
Count-Object-NotFound-Stats          telemetry,ops          count GET(object) 404 as errors
Enable-Go-Runtime-Metrics            telemetry,ops,overhead publish selected Go runtime metrics via Prometheus
Dload-Allow-Private-Egress           security-              allow downloader egress to private RFC1918/ULA addresses; loopback and link-local remain blocked
S3-Redirect-Rebuild                  s3,compat,security-    allow S3 clients that rebuild redirected requests instead of following the Location URI (forbidden when AuthN or intra-cluster signing is configured)
System-Reserved-KTLS                 perf,net,ops,compat    offload TLS transmit path to the kernel and enable Linux sendfile (reserved for internal use; may be redefined or removed at any time)
S3-TensorFlow-Query                  s3,compat,integrity-   legacy TensorFlow: reinterpret '?' inside a decoded S3 object path as the start of query parameters
Disable-GET-Deadline                 ops,security-          GET(object): disable client write and backend read deadlines (minimum transfer rate, see timeout.send_file_time)
Cluster config updated

Notice the [FEATURE, TAGS, DESCRIPTION] table above that shows all supported feature flags along with their respective tags and descriptions. Currently selected features are shown in color. Tags help identify the purpose and impact of each feature at a glance.

Tip: Use the tags to quickly identify features by category. For example, look for integrity- tags to identify features that may trade data integrity for performance, or overhead tags for features that may impact performance.

To view the current (configured) setting, type the same command and hit Enter:

$ ais config cluster features
PROPERTY         VALUE
features         Skip-Loading-VersionChecksum-MD
                 S3-API-via-Root
                 Load-Balance-GET
FEATURE                              TAGS                   DESCRIPTION
Skip-Loading-VersionChecksum-MD      perf,integrity-        (*) skip loading existing object's metadata, Version and Checksum (VC) in particular               <<< colored
Do-not-Auto-Detect-FileShare         promote,ops            do not auto-detect file share (NFS, SMB) when _promoting_ shared files to AIS
S3-API-via-Root                      s3,compat,ops          handle s3 requests via `aistore-hostname/` (default: `aistore-hostname/s3`)                        <<< colored
Fsync-PUT                            integrity+,overhead    (*) when finalizing PUT(object): fflush prior to (close, rename) sequence
LZ4-Block-1MB                        lz4                    .tar.lz4 format, lz4 compression: max uncompressed block size=1MB (default: 256K)
LZ4-Frame-Checksum                   lz4                    checksum lz4 frames (default: don't)
Do-not-Allow-Passing-FQN-to-ETL      etl,security           do not allow passing fully-qualified name of a locally stored object to (local) ETL containers
Ignore-LimitedCoexistence-Conflicts  ops,integrity-         run in presence of _limited coexistence_ type conflicts (same as e.g. CopyBckMsg.Force but globally)
S3-Presigned-Request                 s3,security,compat     (*) pass-through client-signed (presigned) S3 requests for subsequent authentication by S3
Do-not-Optimize-Listing-Virtual-Dirs overhead               when prefix doesn't end with '/' and is a subdirectory: don't assume there are no _prefixed_ obj names
Disable-Cold-GET                     perf,integrity-        disable cold-GET (from remote bucket)
Streaming-Cold-GET                   perf,integrity-        write and transmit cold-GET content back to user in parallel, without _finalizing_ in-cluster object
S3-Reverse-Proxy                     s3,net,ops             intra-cluster communications: instead of regular HTTP redirects reverse-proxy S3 API calls to designated targets
S3-Use-Path-Style                    s3,compat              use older path-style addressing (as opposed to virtual-hosted style), e.g., `https://s3.amazonaws.com/BUCKET/KEY`
Do-not-Delete-When-Rebalancing       integrity?,ops         disable lazy deletion during global rebalance: do not delete misplaced sources of the migrated objects
Do-not-Set-Control-Plane-ToS         net,ops                intra-cluster control plane: use default network priority (do not set IPv4 ToS to low-latency)
Trust-Crypto-Safe-Checksums          integrity+,overhead    when checking whether objects are identical trust only cryptographically secure checksums
S3-ListObjectVersions                s3,overhead            when versioning info is requested, use ListObjectVersions API (beware: extremely slow, versioned S3 buckets only)
Enable-Detailed-Prom-Metrics         telemetry,overhead     include (bucket, xaction) Prometheus variable labels with every GET and PUT transaction
Force-Container-CPU-Mem              deploy                 force container-based CPU and memory metrics when automated environment detection fails (startup only)
Resume-Interrupted-MPU               mpu,ops                resume interrupted multipart uploads from persisted partial manifests
Keep-Unknown-FQN                     integrity?,ops         do not delete unrecognized/invalid FQNs during space cleanup ('ais space-cleanup')
Load-Balance-GET                     perf                   when bucket is n-way mirrored read object replica from the least-utilized mountpath               <<< colored
Count-Object-NotFound-Stats          telemetry,ops          count GET(object) 404 as errors
Enable-Go-Runtime-Metrics            telemetry,ops,overhead publish selected Go runtime metrics via Prometheus
Dload-Allow-Private-Egress           security-              allow downloader egress to private RFC1918/ULA addresses; loopback and link-local remain blocked
S3-Redirect-Rebuild                  s3,compat,security-    allow S3 clients that rebuild redirected requests instead of following the Location URI (forbidden when AuthN or intra-cluster signing is configured)
System-Reserved-KTLS                 perf,net,ops,compat    offload TLS transmit path to the kernel and enable Linux sendfile (reserved for internal use; may be redefined or removed at any time)
S3-TensorFlow-Query                  s3,compat,integrity-   legacy TensorFlow: reinterpret '?' inside a decoded S3 object path as the start of query parameters
Disable-GET-Deadline                 ops,security-          GET(object): disable client write and backend read deadlines (minimum transfer rate, see timeout.send_file_time)

The same in JSON:

$ ais config cluster features --json
{
    "features": "1552"
}

Finally, to reset feature flags back to the system defaults, <TAB-TAB> to select none from the completion list, or simply run:

$ ais config cluster features none
PROPERTY         VALUE
features         none
Cluster config updated

Bucket features

By default, created and added (ie., discovered upon remote lookup) buckets inherit their properties from the cluster.

For background and details, see Bucket Properties.

It is possible, however, to change the defaults both at bucket creation time and/or at any later time.

Here’s a brief 1-2-3 demonstration in re specifically: feature flags.

1. show existing bucket-scope features

$ ais bucket props set ais://nnn features <TAB-TAB>
Skip-Loading-VersionChecksum-MD   Streaming-Cold-GET                Count-Object-NotFound-Stats
Fsync-PUT                         S3-Use-Path-Style                 none
S3-Presigned-Request              S3-ListObjectVersions
Disable-Cold-GET                  Resume-Interrupted-MPU

2. select and set

$ ais bucket props set ais://nnn features S3-Presigned-Request
"features" set to: "S3-Presigned-Request" (was: "none")
Bucket props successfully updated.
FEATURE                          TAGS                    DESCRIPTION
Skip-Loading-VersionChecksum-MD  perf,integrity-        (*) skip loading existing object's metadata, Version and Checksum (VC) in particular
Fsync-PUT                        integrity+,overhead    (*) when finalizing PUT(object): fflush prior to (close, rename) sequence
S3-Presigned-Request             s3,security,compat     (*) pass-through client-signed (presigned) S3 requests for subsequent authentication by S3   <<<<<< colored
Disable-Cold-GET                 perf,integrity-        disable cold-GET (from remote bucket)
Streaming-Cold-GET               perf,integrity-        write and transmit cold-GET content back to user in parallel, without _finalizing_ in-cluster object
S3-Use-Path-Style                s3,compat              use older path-style addressing (as opposed to virtual-hosted style), e.g., `https://s3.amazonaws.com/BUCKET/KEY`
Resume-Interrupted-MPU           mpu,ops                resume interrupted multipart uploads from persisted partial manifests
S3-ListObjectVersions            s3,overhead            when versioning info is requested, use ListObjectVersions API (beware: extremely slow, versioned S3 buckets only)
Count-Object-NotFound-Stats      telemetry,ops          count GET(object) 404 as errors

3. reset feature flags back to zero (or ‘none’)

$ ais bucket props set ais://nnn features none
"features" set to: "0" (was: "S3-Presigned-Request")
Bucket props successfully updated.
FEATURE                          TAGS                    DESCRIPTION
Skip-Loading-VersionChecksum-MD  perf,integrity-        (*) skip loading existing object's metadata, Version and Checksum (VC) in particular
Fsync-PUT                        integrity+,overhead    (*) when finalizing PUT(object): fflush prior to (close, rename) sequence
S3-Presigned-Request             s3,security,compat     (*) pass-through client-signed (presigned) S3 requests for subsequent authentication by S3
Disable-Cold-GET                 perf,integrity-        disable cold-GET (from remote bucket)
Streaming-Cold-GET               perf,integrity-        write and transmit cold-GET content back to user in parallel, without _finalizing_ in-cluster object
S3-Use-Path-Style                s3,compat              use older path-style addressing (as opposed to virtual-hosted style), e.g., `https://s3.amazonaws.com/BUCKET/KEY`
Resume-Interrupted-MPU           mpu,ops                resume interrupted multipart uploads from persisted partial manifests
S3-ListObjectVersions            s3,overhead            when versioning info is requested, use ListObjectVersions API (beware: extremely slow, versioned S3 buckets only)
Count-Object-NotFound-Stats      telemetry,ops          count GET(object) 404 as errors

Validation errors

When setting conflicting feature flags, the system will reject the configuration - e.g.:

$ ais config cluster features Disable-Cold-GET Streaming-Cold-GET
Error: feature flags "Disable-Cold-GET" and "Streaming-Cold-GET" are mutually exclusive

Example: Count-Object-NotFound-Stats

By default, a missing object (GET(object) returning 404) does not increment ERR-GET.

$ ais get ais://nnn/missing
Error: object "ais://nnn/missing" does not exist
$ ais performance counters --regex err-get
TARGET           ERR-GET(n)      ERR-GETBATCH(n)         ERR-GETBLOB(n)
t[...]           -               -                       -

To restore the legacy behavior, enable Count-Object-NotFound-Stats either globally or on a per-bucket basis.

For example, set it on a bucket:

$ ais bucket props set ais://nnn features Count-Object-NotFound-Stats
"features" set to: "Count-Object-NotFound-Stats" (was: "none")
Bucket props successfully updated.

Now the same missing-object GET increments ERR-GET:

$ ais get ais://nnn/missing
Error: object "ais://nnn/missing" does not exist
$ ais performance counters --regex err-get
TARGET           ERR-GET(n)      ERR-GETBATCH(n)         ERR-GETBLOB(n)
t[...]           1               -                       -

In other words, the feature does not change GET semantics: the request still returns 404. It only changes whether object-not-found responses are counted as GET errors in the corresponding telemetry.