Skip to navigation
API ReferenceFull Library ReferenceNemo GymNemo GymDataTasksHarbor

nemo_gym.data.tasks.harbor.dockerfile

View as Markdown

Recognize Dockerfiles Gym can run without a build, and resolve them as Docker would.

Two shapes are recognized:

  • Pull mode: FROM <image> plus WORKDIR, ENV, USER or LABEL lines. Nothing changes the filesystem, so Gym runs the base image directly and applies the recorded settings at sandbox creation (base_image_only).
  • Overlay mode: the same, plus RUN lines. Gym pulls the base image and runs the RUN lines inside the sandbox at seed, each with the WORKDIR, ENV and USER in effect at that point of the Dockerfile (overlay_image). This is the shape of SWE-bench style tasks: a published environment image plus setup commands.

ENV, WORKDIR and USER are resolved the way docker build resolves them: the base image’s own configuration (BaseImageConfig, recorded from the registry at prepare) is the starting state, $NAME and ${NAME} expand against the accumulated environment (not inside single quotes; \$ is a literal dollar), a relative WORKDIR joins onto the previous one. The resolved values are literal: nothing reaches the sandbox with a live $ in it. A Dockerfile whose resolution depends on the base image (a RUN line, a reference to a variable the Dockerfile did not set, a relative first WORKDIR) raises ImageConfigRequired when no configuration is given.

An empty CMD [] is accepted and ignored. An empty ENTRYPOINT [] is accepted and recorded (entrypoint_cleared): the sandbox provider starts the image with its own keepalive either way, and the Harbor environment server runs the base image’s recorded Entrypoint beside it unless the Dockerfile cleared it. Anything else (COPY, ADD, multi-stage FROM, ARG-templated images, heredoc RUN, RUN --mount, a real ENTRYPOINT or CMD, HEALTHCHECK, EXPOSE, VOLUME, SHELL, STOPSIGNAL, a FROM flag such as --platform, an # escape= parser directive) needs a real build, which is not part of this loader yet; DockerfileNeedsBuild names the first instruction that requires it.

Module Contents

Classes

NameDescription
BaseImageA Dockerfile that reduces to a base image plus creation-time settings.
BaseImageConfigWhat a base image’s OCI configuration sets before its Dockerfile’s first instruction.
DockerfileThe parsed instructions of a pull-mode or overlay-mode Dockerfile, before resolution.
DockerfileNeedsBuildThe Dockerfile holds an instruction this loader cannot apply without a build.
ImageConfigRequiredResolving the Dockerfile needs the base image’s recorded configuration, which was not given.
OverlayImageA base image plus the RUN lines to apply at seed, in Dockerfile order.
OverlayRunOne RUN line with the Dockerfile state in effect when it runs, resolved to literals.
OverlayWorkdirOne WORKDIR instruction: the directory to create, owned by the user in effect at it.
_ExpanderDocker’s shell-word processing: quote removal, escapes and $ expansion against env.

Functions

NameDescription
_closing_braceIndex of the &#125; closing the &#123; that opens text, skipping nested $&#123;...&#125;; -1 when unterminated.
_env_pairs(key, raw value) pairs of an ENV line; values keep their quotes for the expander.
_json_arrayThe JSON array an exec-form instruction holds, or None when the arguments are shell form.
_logical_linesJoin continuation lines and drop comments and blanks, as Docker’s parser does.
_reject_escape_directiveRefuse a Dockerfile whose leading comment block sets # escape=.
_run_commandThe validated arguments of a RUN line: a shell-form line, or the exec-form JSON array as written.
base_image_onlyReturn the base image and settings, or None when the file has RUN lines or needs a build.
dockerfile_has_heredocWhether any RUN, COPY or ADD instruction of a Dockerfile opens a heredoc.
has_heredocWhether command opens a Dockerfile heredoc, judged word by word as BuildKit does.
overlay_imageThe base image, settings and RUN lines, or None when the file needs a build.
parse_dockerfileRead a pull-mode or overlay-mode Dockerfile; raise DockerfileNeedsBuild for anything else.
resolve_dockerfileApply the instructions in order from the base image’s state, as docker build would.
split_wordsSplit on unquoted blanks, keeping quotes and escapes in the words (Docker’s parseWords).

Data

DEFAULT_PATH

DEFAULT_SHELL

RECORD_VERSION

_BLANKS

_CONTINUATION

_HEREDOC_WORD

_NAME

API

class nemo_gym.data.tasks.harbor.dockerfile.BaseImage(
    image: str,
    workdir: str | None = None,
    env: dict[str, str] = dict(),
    user: str | None = None,
    entrypoint_cleared: bool = False
)
Dataclass

A Dockerfile that reduces to a base image plus creation-time settings.

env holds the keys the Dockerfile’s ENV lines set, with their resolved final values; workdir and user are the resolved final values (None when neither the Dockerfile nor a given base image configuration sets them). entrypoint_cleared is whether the Dockerfile wrote ENTRYPOINT [], so the base image’s entrypoint must not run.

entrypoint_cleared
bool = False
env
dict[str, str] = field(default_factory=dict)
image
str
user
str | None = None
workdir
str | None = None
class nemo_gym.data.tasks.harbor.dockerfile.BaseImageConfig(
    env: dict[str, str] = dict(),
    workdir: str | None = None,
    user: str | None = None,
    entrypoint: tuple[str, ...] | None = None,
    healthcheck: dict[str, typing.Any] | None = None,
    shell: tuple[str, ...] = DEFAULT_SHELL
)
Dataclass

What a base image’s OCI configuration sets before its Dockerfile’s first instruction.

env is the image’s Env as a mapping; workdir and user are WorkingDir and User (None when the image sets none, meaning / and root); shell is the image’s Shell, which runs its shell-form RUN lines (Docker’s /bin/sh -c when unset).

entrypoint
tuple[str, ...] | None = None
env
dict[str, str] = field(default_factory=dict)
healthcheck
dict[str, Any] | None = None
shell
tuple[str, ...] = DEFAULT_SHELL
user
str | None = None
workdir
str | None = None
classmethod

Read a compose-images.json entry (see nemo_gym.data.tasks.harbor.image_configs).

Raises ValueError for an entry written before RECORD_VERSION: it lacks fields this resolver reads (an absent Env would read as a known, empty environment), so the dataset must be prepared again to re-record it.

class nemo_gym.data.tasks.harbor.dockerfile.Dockerfile(
    image: str,
    instructions: tuple[tuple[str, str], ...] = (),
    entrypoint_cleared: bool = False
)
Dataclass

The parsed instructions of a pull-mode or overlay-mode Dockerfile, before resolution.

entrypoint_cleared
bool = False
has_runs
bool
image
str
instructions
tuple[tuple[str, str], ...] = ()
class nemo_gym.data.tasks.harbor.dockerfile.DockerfileNeedsBuild()

Bases: ValueError

The Dockerfile holds an instruction this loader cannot apply without a build.

class nemo_gym.data.tasks.harbor.dockerfile.ImageConfigRequired()

Bases: ValueError

Resolving the Dockerfile needs the base image’s recorded configuration, which was not given.

class nemo_gym.data.tasks.harbor.dockerfile.OverlayImage(
    image: str,
    workdir: str | None = None,
    env: dict[str, str] = dict(),
    user: str | None = None,
    entrypoint_cleared: bool = False,
    runs: tuple[nemo_gym.data.tasks.harbor.dockerfile.OverlayRun, ...] = (),
    workdirs: tuple[nemo_gym.data.tasks.harbor.dockerfile.OverlayWorkdir, ...] = ()
)
Dataclass

Bases: BaseImage

A base image plus the RUN lines to apply at seed, in Dockerfile order.

workdirs are the WORKDIR instructions, in order, each with the user in effect at it and its place among the runs (see OverlayWorkdir).

runs
tuple[OverlayRun, ...] = ()
workdirs
tuple[OverlayWorkdir, ...] = ()
class nemo_gym.data.tasks.harbor.dockerfile.OverlayRun(
    command: str,
    workdir: str = '/',
    env: dict[str, str] = dict(),
    user: str | None = None,
    shell: tuple[str, ...] | None = DEFAULT_SHELL
)
Dataclass

One RUN line with the Dockerfile state in effect when it runs, resolved to literals.

command is the command text: a shell-form line as written, or an exec-form argument vector joined with shell quoting. shell runs it: the image’s Shell in effect for a shell-form line (Docker’s /bin/sh -c unless the image sets one), None for an exec-form line, which runs its arguments directly, without a shell. env is the whole environment the line runs with (the base image’s Env plus the ENV lines so far), workdir the directory it runs in and user who runs it (None is root).

command
str
env
dict[str, str] = field(default_factory=dict)
shell
tuple[str, ...] | None = DEFAULT_SHELL
user
str | None = None
workdir
str = '/'
class nemo_gym.data.tasks.harbor.dockerfile.OverlayWorkdir(
    path: str,
    user: str | None = None,
    before: int = 0
)
Dataclass

One WORKDIR instruction: the directory to create, owned by the user in effect at it.

before is how many RUN lines precede the instruction, which places it among OverlayImage.runs: the directory is made before run before (counting from 0), or after the last run when no RUN follows it. Docker creates every WORKDIR whether or not a RUN uses it, so the finished image has them all.

before
int = 0
path
str
user
str | None = None
class nemo_gym.data.tasks.harbor.dockerfile._Expander(
    env: collections.abc.Mapping[str, str],
    known: bool
)

Docker’s shell-word processing: quote removal, escapes and $ expansion against env.

With known=False the base image’s environment is unknown, so a reference to a variable the Dockerfile did not set itself raises ImageConfigRequired instead of silently expanding to nothing.

nemo_gym.data.tasks.harbor.dockerfile._Expander._dollar(
    word: str,
    index: int
) -> tuple[str, int]

Expand the $ at index; return the text and the index after the expression.

nemo_gym.data.tasks.harbor.dockerfile._Expander._lookup(
    name: str
) -> str | None
nemo_gym.data.tasks.harbor.dockerfile._Expander.expand(
    word: str
) -> str
nemo_gym.data.tasks.harbor.dockerfile._closing_brace(
    text: str
) -> int

Index of the } closing the { that opens text, skipping nested ${...}; -1 when unterminated.

nemo_gym.data.tasks.harbor.dockerfile._env_pairs(
    arguments: str
) -> list[tuple[str, str]]

(key, raw value) pairs of an ENV line; values keep their quotes for the expander.

nemo_gym.data.tasks.harbor.dockerfile._json_array(
    arguments: str
) -> list | None

The JSON array an exec-form instruction holds, or None when the arguments are shell form.

nemo_gym.data.tasks.harbor.dockerfile._logical_lines(
    text: str
) -> list[str]

Join continuation lines and drop comments and blanks, as Docker’s parser does.

A trailing backslash is removed together with the newline and the next line is appended verbatim (no separator is inserted). Comment and blank lines are dropped even inside a continuation. Only the first line of an instruction has its leading blanks removed.

nemo_gym.data.tasks.harbor.dockerfile._reject_escape_directive(
    text: str
) -> None

Refuse a Dockerfile whose leading comment block sets # escape=.

Docker reads parser directives only in the comments before the first instruction, and this loader joins continuation lines with a backslash alone (_logical_lines), so a file that switches the escape character would be read with the wrong line structure (RUN echo foo \ would swallow the next instruction). A build honours the directive; a # escape= after an instruction is a plain comment, as it is for Docker. # syntax= and # check= do not change how lines are read.

nemo_gym.data.tasks.harbor.dockerfile._run_command(
    arguments: str
) -> str

The validated arguments of a RUN line: a shell-form line, or the exec-form JSON array as written.

nemo_gym.data.tasks.harbor.dockerfile.base_image_only(
    text: str,
    base: nemo_gym.data.tasks.harbor.dockerfile.BaseImageConfig | None = None
) -> nemo_gym.data.tasks.harbor.dockerfile.BaseImage | None

Return the base image and settings, or None when the file has RUN lines or needs a build.

nemo_gym.data.tasks.harbor.dockerfile.dockerfile_has_heredoc(
    text: str
) -> bool

Whether any RUN, COPY or ADD instruction of a Dockerfile opens a heredoc.

Each instruction’s arguments are judged as has_heredoc judges one line, word by word as BuildKit does, so RUN cat > /x << 'EOF' (a space before the delimiter, which BuildKit does not read as a heredoc either) is not one. The body lines of a heredoc are not instructions, but they are only ever reached once the opening line has already answered.

nemo_gym.data.tasks.harbor.dockerfile.has_heredoc(
    command: str
) -> bool

Whether command opens a Dockerfile heredoc, judged word by word as BuildKit does.

nemo_gym.data.tasks.harbor.dockerfile.overlay_image(
    text: str,
    base: nemo_gym.data.tasks.harbor.dockerfile.BaseImageConfig | None = None
) -> nemo_gym.data.tasks.harbor.dockerfile.OverlayImage | None

The base image, settings and RUN lines, or None when the file needs a build.

Raises ImageConfigRequired when the file depends on a base image configuration and none is given.

nemo_gym.data.tasks.harbor.dockerfile.parse_dockerfile(
    text: str
) -> nemo_gym.data.tasks.harbor.dockerfile.Dockerfile

Read a pull-mode or overlay-mode Dockerfile; raise DockerfileNeedsBuild for anything else.

Apply the instructions in order from the base image’s state, as docker build would.

With base given, the starting environment is its Env (plus Docker’s default PATH when it has none), the working directory its WorkingDir or / and the user its User. Without it, only a Dockerfile whose values do not depend on the base image resolves; anything else raises ImageConfigRequired.

nemo_gym.data.tasks.harbor.dockerfile.split_words(
    text: str
) -> list[str]

Split on unquoted blanks, keeping quotes and escapes in the words (Docker’s parseWords).

nemo_gym.data.tasks.harbor.dockerfile.DEFAULT_PATH = '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'
 
nemo_gym.data.tasks.harbor.dockerfile.DEFAULT_SHELL = ('/bin/sh', '-c')
 
nemo_gym.data.tasks.harbor.dockerfile.RECORD_VERSION = 2
 
nemo_gym.data.tasks.harbor.dockerfile._BLANKS = re.compile('[ \\t\\v\\f\\r]+')
 
nemo_gym.data.tasks.harbor.dockerfile._CONTINUATION = re.compile('\\\\[ \\t]*$')
 
nemo_gym.data.tasks.harbor.dockerfile._HEREDOC_WORD = re.compile('^<<-?(?:\\w+|\\"[^\\"]+\\"|\'[^\']+\')$')
 
nemo_gym.data.tasks.harbor.dockerfile._NAME = re.compile('[A-Za-z_][A-Za-z0-9_]*')