nemo_gym.data.tasks.harbor.compose_variables

Index of the } that closes the ${ opening at start, nested ${...} included.

environment/.env read as Compose’s dotenv reader reads it (compose-go/dotenv): an unquoted or double-quoted value is interpolated against above and then the lines before it, a single-quoted value is literal. A dollar that stays literal is kept doubled, as parse_env_file keeps it, so the document’s literal check leaves it alone.

A service’s env_file entries (a path, a list of paths or long-syntax mappings) as (path, required) pairs, in order. An entry with no path raises ComposeSubstitutionError; so does one with a format, as only the default dotenv format is read.

path resolved under base_dir (Harbor’s ${CONTEXT_DIR}/x.env is absolute and lands there too). A path that resolves outside, or a symlink, is refused: task folders are third-party content and the file must be part of what prepare digested. So is a file the task digest skips (*~, *.swp, __pycache__ and the like, see nemo_gym.data.tasks.harbor.digest.is_ignored): it would change the document without changing the identity the server checks at seed.

value as the document’s interpolation and literal checks read a literal: every $ doubled.

The quoted env_file value opening at text[0]: (value, quote, characters consumed). The value runs to its closing quote, across newlines if need be. Double quotes decode \n, \r, \t, \\, \" and \$ (the last becoming escaped_dollar) and keep any other \X as written; single quotes are literal except for \', which is an escaped quote in either style.

The resolver of entries resolve_env_files wrote: absolute paths already checked by their own rule.

Compose interpolation: $$ (a literal dollar, kept doubled for _compose_literals), ${NAME}, ${NAME:-default}, ${NAME-default}, ${NAME:?error}, ${NAME?error}, ${NAME:+alternative}, ${NAME+alternative} and bare $NAME. Defaults, alternatives and error texts are interpolated too, so ${A:-${B:-x}} nests the way Compose documents.

[environment].env as Harbor’s resolve_env_vars hands it to docker compose (see compose_variables), each value literal to the document (_literal).

The variables a task’s Compose file may interpolate, resolved in the order Harbor’s docker compose sees.

Harbor runs docker compose with the process environment, the task’s [environment].env over it and its own infrastructure variables over both (merge_compose_env in harbor/environments/docker/compose_env.py and _compose_env_vars in docker.py at v0.22.0, called with include_os_env=True), and Compose reads the project directory’s .env below everything the command’s environment sets. Highest first:

1. Gym’s infrastructure set: MAIN_IMAGE_NAME (the resolved task image), CONTEXT_DIR (the task’s environment/ folder), TEST_DIR (TEST_DIR), COMPOSE_PROJECT_NAME (hb__<hash>, Harbor’s environment id of the context and the main image name, a stable placeholder: compose-go injects the real value from --project-name, the trial session id under Harbor and compose-<uuid> per episode in Gym’s adapter, which has no Compose-generated names for it to match), PREBUILT_IMAGE_NAME when docker_image is declared, CPUS and MEMORY (as <n>M, Harbor’s spelling) from the task’s resources, and the HOST_*_PATH / ENV_*_PATH log mounts as the /logs tree;
2. the task’s [environment].env, each value read as Harbor’s resolve_env_vars reads it (harbor/utils/env.py): a whole value ${VAR} or ${VAR:-default} comes from the process environment, the default standing in when the variable is unset, and one with neither raises ImageConfigError naming the task and the variable, as Harbor fails the trial; any other value is literal;
3. the process environment;
4. environment/.env, its unquoted and double-quoted values interpolated against the levels above and the lines before them, as Compose’s dotenv reader does; single-quoted values are literal.

A reference none of them sets takes its default (${VAR:-x}, ${VAR-x}), else it is an empty string, as docker compose makes it, with one warning naming the task and the variable (added to unset when given, so the caller warns once for the whole document, else logged here). ${VAR:?msg} and ${VAR?msg} raise ImageConfigError with the message. The process environment is not part of the task’s digest: prepare and the environment server must run with the same variables, as Harbor’s one process does.

Every value is literal to the document, as it is to Compose (which splices a variable’s value in verbatim and never re-scans it): a $ in a host, task or infrastructure value is doubled here, the convention parse_env_file and _dotenv_variables keep for a literal dollar, so a password such as $2b$12$... neither trips the document’s residual check nor loses a $$ when the sandbox starts.

The KEY=value lines of a Compose env file in file order, as (key, value, quote). quote is ', " or "" for an unquoted value, which is how Compose’s dotenv reader decides what to do with it: a single-quoted value is literal, the other two it interpolates. Blank lines and # comments are skipped and an optional export prefix dropped. A quoted value ends at its closing quote, which may be on a later line, and the rest of that line is a comment; its escapes are decoded as _quoted_env_value says, a double-quoted \$ becoming escaped_dollar. An unquoted value ends at the first # that follows whitespace and is returned as written, $ included. A line that is not KEY=value or a quote that never closes raises ComposeSubstitutionError naming the line number only, as env files hold secrets.

A service’s env_file entries (a path, a list of paths or long-syntax mappings) as the absolute paths of the files to read, in order, each resolved by resolve and existing. A missing required file raises ComposeSubstitutionError naming the entry as written and, when it differs, where it resolved to; an entry with required: false is dropped when absent, as Compose skips it; an entry with a format is refused, as only the default dotenv format is read.

Gym’s infrastructure set for task (source 1 of compose_variables, the one Harbor’s merge_compose_env puts above every other), as raw values: the caller escapes them for the document.

Read each service’s env_file entries into its environment mapping: relative to base_dir under the task’s rule (under_task_folder), or, with base_dir=None, the absolute paths resolve_env_files wrote, each file’s own rule applied already.

Every service’s environment becomes a mapping of strings. A service’s own entries win over its files, as in Compose. A bare NAME entry in a list-form environment (or NAME: with no value in the mapping form) takes the task variable of that name; when there is none the variable is left unset, as Compose leaves it, so it also removes the env_file value of that name. A missing required file raises ComposeSubstitutionError naming it; an entry with required: false is skipped when absent; an entry with a format is refused, as only the default dotenv format is read. A file that does not parse raises with the service, the file and the line number, never the line.

A deployment’s compose_variables values as the document reads literals (_literal on each), the way every source of compose_variables is escaped; None (no deployment values, as at prepare) stays None.

KEY=value lines of a Compose env_file, taken literally.

Lines are read as env_file_entries reads them. Values are not interpolated: a $ in a value is escaped as $$ so the document’s later interpolation and literal checks leave it as one $.

The key path (services.api.environment.DB_PASS, services.a.command[1]) of the first string still carrying a $NAME or ${...} after substitution, or None. The path, not the value: the value next to a residual may be a secret the host supplied, and the caller reports this in an error message.

A service’s env_file entries with each path resolved by resolve to the absolute path of the file, the file not checked to exist: a required entry becomes its absolute path, an optional one (required: false) {"path": ..., "required": False}, so the reading (env_file_paths) can skip it when it is absent. Compose resolves each -f file’s paths before the merge and reads the env files of the merged document after it, so a file an earlier document names and a later file drops (env_file: !reset []) is never required to exist; what resolve refuses (a task’s entry outside its folder) is refused here, at the file’s own rule.

Each service’s env_file entries replaced by what resolve_env_file_entries resolves them to (absolute paths, an optional entry keeping its required: false), the list’s type kept, the files not read or checked to exist yet. Compose resolves a file’s relative paths against that file before it merges the -f files and reads the env files once after the merge, so a missing file is reported then, if the merged document still names it; inline_env_files with base_dir=None is that reading. Mutates and returns document.

Give the typed healthcheck fields their types back after interpolation, which turns every scalar it touches into a string: disable becomes a boolean (${DISABLE:-false} must not disable the check by being a non-empty string) and retries an integer. Any other string value raises ComposeSubstitutionError naming the field. Mutates and returns document.

Interpolate ${VAR} references in every string of a Compose document from variables alone.

The process environment is never consulted, so a dataset resolves the same way on every deployment. A variable that is neither defined nor given a default raises ComposeSubstitutionError naming it.

substitute_compose with Compose’s rule for a bare ${VAR} or $VAR nobody sets: an empty string, the name added to unset for the caller to warn about once. Only the bare form raises the resolver’s default message; ${VAR:-x} keeps its default, ${VAR:?msg} its refusal, and every other failure passes through unchanged. The loader and the Dockerfile build (nemo_gym.data.tasks.harbor.image_build) interpolate a task’s Compose file through this one rule, so the two never disagree on the same file.

The task file’s resolver: entries relative to base_dir (the task’s environment folder) and kept under it, see _env_file_path.