Is this page helpful?

Configuration#

template<typename T>
using rapidsmpf::config::OptionFactory = std::function<T(std::string const&)>#

Type alias for a factory function that constructs options from strings.

The factory receives the string representation of an option value and returns an instance of the option type. If the option is unset, the function receives an empty string and should either return a meaningful default value or throw std::invalid_argument.

Note

The factory must not access other options, as a lock is held during option initialization and doing so may cause a deadlock.

const std::unordered_map<std::string, std::string> rapidsmpf::config::DEFAULTS{{"statistics", "false"}, {"pinned_memory", "false"}, {"pinned_initial_pool_size", "0%"}, {"pinned_max_pool_size", "80%"}, {"spill_device_limit", "80%"}, {"periodic_spill_check", "1ms"}, {"num_streams", "16"}, {"num_streaming_threads", "1"}, {"memory_reserve_timeout", "100ms"}, {"allow_overbooking_by_default", "true"}, {"log", "WARN"}, {"ucxx_progress_mode", "thread-blocking"},}#

String-form default values for config options.

Defaults are stored as strings and parsed through the same factories used for user-supplied values. Options::get<T>(key, factory) consults this map automatically: when the user has not supplied a value, the factory receives the registered default string for key.

Options are not required to have an entry in this map. If no default is registered for a key, the factory receives the empty string.

To add a new option, add an entry here and reference it at the call site via Options::get<T>("<key>", factory).

void rapidsmpf::config::get_environment_variables(
std::unordered_map<std::string, std::string> &output,
std::string const &key_regex = "RAPIDSMPF_(.*)"
)#

Populates a map with environment variables matching a given regular expression.

This function scans the current process’s environment variables and inserts those whose keys match the provided regular expression into the output map. Only variables with keys not already present in output are inserted; existing keys are left unchanged.

The key_regex should contain a single capture group that extracts the portion of the environment variable key you want to use as the map key. For example, to strip the RAPIDSMPF_ prefix, use RAPIDSMPF_(.*) as the regex. The captured group will be used as the key in the output map.

Example:

  • Environment variable: RAPIDSMPF_FOO=bar

  • key_regex: “RAPIDSMPF_(.*)”

  • Resulting map entry: { “FOO”, “bar” }

Warning

This function uses std::regex and relies on the global environ symbol, which is POSIX-specific and is not thread-safe.

Parameters:

  • output – [out] The map to populate with matching environment variables. Only keys that do not already exist in the map will be added.

  • key_regex – [in] A regular expression with a single capture group to match and extract the environment variable keys. Only environment variables with keys matching this pattern will be considered.

Throws:

std::invalid_argument – If key_regex doesn’t contain exactly one capture group.

std::unordered_map<std::string, std::string> rapidsmpf::config::get_environment_variables(
std::string const &key_regex = "RAPIDSMPF_(.*)"
)#

Returns a map of environment variables matching a given regular expression.

This is a convenience overload. See the documentation for the first variant of get_environment_variables() for details on matching and behavior.

See also

get_environment_variables(std::unordered_map<std::string, std::string>&, std::string const&)

Parameters:

key_regex – A regular expression with a single capture group to match and extract the environment variable keys.

Throws:

std::invalid_argument – If key_regex doesn’t contain exactly one capture group.

Returns:

A map containing all matching environment variables, with keys as extracted by the capture group.

class OptionValue#
#include <config.hpp>

Configuration option value.

The OptionValue class encapsulates a value (of any type using std::any) and a string representation of the value.

Public Functions

OptionValue() = default#

Default constructor.

Constructs an empty OptionValue.

inline OptionValue(std::string value_as_string)#

Constructs OptionValue from a string representation.

Parameters:

value_as_string – A string representation of the value.

template<typename T>
inline explicit OptionValue(T value)#

Constructs OptionValue from a typed value.

The value is stored directly and no string representation is provided. Options constructed this way are considered initialized and make the Options instance unserializable.

Template Parameters:

T – Type of the value to store.

Parameters:

value – The value to store.

inline std::any const &get_value() const#

Retrieves the stored value.

Returns:

A const reference to the std::any value.

inline std::string const &get_value_as_string() const#

Retrieves the string representation of the value.

Is the empty string, if not string representation exist.

Returns:

A const reference to the string representation.

inline void set_value(std::any value)#

Sets the value if it has not been set already.

Parameters:

value – The new value to store.

Throws:

std::invalid_argument – if the value is already set.

class Options#
#include <config.hpp>

Manages configuration options for RapidsMPF operations.

The Options class provides a high-level interface for storing and retrieving configuration options.

All keys are trimmed and converted to lower case using rapidsmpf::trim() and rapidsmpf::to_lower().

Note

Copying rapidsmpf::config::Options is efficient as it uses a shared pointer to the shared options (OptionsShared).

Public Functions

Options(std::unordered_map<std::string, OptionValue> options = {})#

Constructs an Options instance from option values.

Parameters:

options – A map of option keys to their corresponding option value.

Throws:

std::invalid_argument – If keys are not case-insensitive.

Options(
std::unordered_map<std::string, std::string> options_as_strings
)#

Constructs an Options instance from option values as strings.

Parameters:

options_as_strings – A map of option keys to their string representations.

Throws:

std::invalid_argument – If keys are not case-insensitive.

bool insert_if_absent(
std::string const &key,
std::string_view option_as_string
)#

Inserts an option only if it is not already present.

This method checks whether the given option key exists in the current set of options. If it does not, the option is inserted in its string representation.

Parameters:

  • key – The option key to insert. The key is trimmed and converted to lower case before insertion.

  • option_as_string – The string representation of the option value.

Returns:

true if the option was inserted; false if it was already present.

std::size_t insert_if_absent(
std::unordered_map<std::string, std::string> options_as_strings
)#

Inserts multiple options if they are not already present.

This method attempts to insert each option key-value pair from the provided map into the current set of options. Each insertion is performed only if the key does not already exist in the options.

Parameters:

options_as_strings – A map of option keys to their string representations.

Returns:

Number of newly inserted options (0 if none were added).

template<typename T>
inline bool insert_if_absent(
std::string const &key,
T value
)#

Inserts an option only if it is not already present.

This method stores a typed value directly, bypassing the string-based representation used for lazy parsing. Once inserted, the option is initialized, and subsequent calls to get<T>() for the same key must use the same T.

This method is only enabled for non string-like types. Values convertible to std::string_view (for example std::string, std::string_view, or string literals) are handled by the string-based overloads instead.

Because no string representation is stored, inserting an option using this method makes the Options instance unserializable. This is consistent with the behavior of get(), as serialization relies exclusively on the original string representations of options.

Template Parameters:

T – Type of the value to store.

Parameters:

  • key – The option key to insert. The key is trimmed and converted to lower case before insertion.

  • value – The value to store.

Returns:

true if the option was inserted; false if it was already present.

template<typename T>
inline T const &get(
std::string const &key,
OptionFactory<T> factory
)#

Retrieves a configuration option by key.

If the option is not present, it will be constructed using the provided factory, which receives the string representation of the option (or an empty string if unset).

Note

Once a key has been accessed with a particular T, subsequent calls to get on the same key must use the same T. Using a different T for the same key will result in a std::bad_any_cast.

Template Parameters:

T – The type of the option to retrieve.

Parameters:

  • key – The option key (should be lower case).

  • factory – Function to construct the option from a string.

Throws:

  • std::invalid_argument – If the stored option type does not match T.

  • std::bad_any_cast – If T doesn’t match the type of the option.

Returns:

Reference to the option value.

std::unordered_map<std::string, std::string> get_strings() const#

Retrieves all option values as strings.

This method returns a map of all currently stored options where both the keys and values are represented as strings.

Options that do not have a string representation, such as those inserted using insert_if_absent<T>(), are included with an empty string value.

Returns:

A map where each key is the option name and each value is the string representation of the corresponding option’s value.

std::vector<std::uint8_t> serialize() const#

Serializes the options into a binary buffer.

An Options instance can only be serialized if all options are still represented exclusively by their original string values. Serialization is based on these string representations and cannot reflect options that have been accessed, parsed, or initialized with typed values.

As a result, serialization is disallowed if any option has been accessed via get() or inserted using the typed insert_if_absent<T>() method, since their string values may no longer accurately reflect their state.

The format (v1) is:

  • [4 bytes MAGIC “RMPF”][1 byte version][1 byte flags][2 bytes reserved]

  • [std::uint64_t count] — number of key-value pairs.

  • [count * 2 * std::uint64_t] — offset pairs (key_offset, value_offset) for each entry.

  • [raw bytes] — all key and value strings, contiguous and null-free.

Offsets are absolute byte positions into the buffer.

Serialization limits:

  • Maximum options: 65,536 entries

  • Maximum key size: 4 KiB

  • Maximum value size: 1 MiB

  • Maximum total buffer size: 64 MiB

Note

To ease Python/Cython compatibility, a std::vector<std::uint8_t> is returned instead of std::vector<std::byte>.

Throws:

std::invalid_argument – If any option has already been accessed or inserted as a typed value.

Returns:

A byte vector representing the serialized options.

Public Static Functions

static Options deserialize(std::vector<std::uint8_t> const &buffer)#

Deserializes a binary buffer into an Options object.

See Options::serialize() for the binary format.

Parameters:

buffer – The binary buffer produced by Options::serialize().

Throws:

  • std::invalid_argument – If the buffer is malformed or incomplete.

  • std::out_of_range – If offsets exceed buffer boundaries.

Returns:

An Options object reconstructed from the buffer.