AutoML#

Invoke AutoML through the applications/tao-run-automl skill bank entry. It automatically selects parameters for a chosen model, action, and set of action inputs.

Each AutoML run evaluates multiple recommendations. At the end of a run, you can access the best configuration and its model checkpoint or action artifact. Multi-objective runs also return the non-dominated recommendations on the Pareto front.

The selected model and action determine AutoML support. The model skill must contain automl_enabled: true, and the selected action must provide a valid schemas/<action>.schema.json search schema plus action defaults from either references/spec_template_<action>.yaml or the schema’s default object. Ask the agent to enumerate the supported model and action pairs before planning a run.

AutoML Capabilities#

AutoML provides the following capabilities:

  • AutoML optimizes parameters for train, distill, prune, quantize, and other schema-backed actions.

  • Multi-objective optimization combines weighted scalarization, per-objective directions and scales, and Pareto-front results.

  • AutoML runs external Python-script actions in an existing virtual environment.

  • The runner automatically deletes trial artifacts and retains checkpoints according to promotion, resume, Hybrid, and Pareto-front safety rules.

  • AutoMLRunner.run() returns a stable result contract, and the package provides a standard result formatter and strict validation of automl_settings keys.

  • Packaged model schemas enable AutoML for DINOv3 continual self-supervised learning and Fast Foundation Stereo training. They also expose DINOv3 backbone choices for SegFormer and Visual ChangeNet searches.

AutoML Algorithm Explanation#

This section gives additional details of the supported AutoML algorithms.

Choosing an Algorithm#

If you are unsure which algorithm to pick, use this guidance:

  • When you are new to AutoML or want a quick start, use bayesian. It is the most well-understood algorithm and produces consistently good results with minimal tuning.

  • When you have a limited GPU budget and need speed, use hyperband. It discards poor trials early and converges faster than a full Bayesian search.

  • When the search space is large (many hyperparameters), use bohb or dehb. Both scale better than pure Bayesian approaches in high-dimensional spaces.

  • When multiple GPUs are available, use bfbo (generates a whole batch of recommendations in parallel) or asha (asynchronous, no sync barrier between workers).

  • When you want hyperparameters to evolve during training, use pbt. Unlike other algorithms, PBT can change learning rate or other parameters mid-training, not just between runs.

  • When you need quick pruning of bad trials, use hyperband_es. It adds explicit early-stopping thresholds on top of Hyperband to cut off clearly failing trials sooner.

Bayesian Optimization#

Bayesian optimization aims to identify optimal configurations more quickly than standard baselines (such as standard random search) by adaptively selecting hyperparameters based on past experimental information.

  • Use a surrogate model to fit the Gaussian Process (GP) to existing data (X,y), where X is a vector of recommendations and y is the observed validation map.

  • Bayesian Optimization adaptively proposes new recommendations based on the fitted GP that produces the best improvement in expectation until reaching the number of max_recommendations.

Hyperband#

Hyperband addresses the issue of expensive hyperparameter optimization by speeding up random search through adaptive resource allocation. Hyperband follows the SuccessiveHalving algorithm to uniformly allocate a budget to a set of hyperparameter recommendations. During AutoML runs, Hyperband evaluates the performance of all recommendations, throws out the worst half, and repeats this process until one recommendation remains (see this research paper for more details). Three hyperband-related parameters (R, nu, epoch_multiplier) pre-define a hyperband sequence of trials based on how many trials to perform in each round and how many resources to give to each trial in that round.

  • The R parameter determines the maximum resources (i.e. the number of epochs and recommendations).

  • The Nu parameter controls the proportion of recommendations discarded in each use of the SuccessiveHalving algorithm.

  • The epoch_multiplier and the resources (r) are multiplied in each SuccessiveHalving iterations(i) to determine the epoch number of each trial. Each run of SuccessiveHalving is referred to as a stage, and each stage contains SuccessiveHalving iterations(i).

The below tables describe the SuccessiveHalving iterations(i), number of hyperparameter recommendation(n), and resources(r) given the R and nu.

Table2. The pre-defined Hyperband experiment schedule when set with R = 81, nu=3.

s=0

s=1

s=2

s=3

i

n

r

n

r

n

r

n

r

0

81

1

27

3

9

9

6

27

1

27

3

9

9

3

27

2

81

2

9

9

3

27

1

81

3

3

27

1

81

4

1

81

Table3. The pre-defined Hyperband experiment schedule when set with R = 27, nu=3.

s=0

s=1

s=2

i

n

r

n

r

n

r

0

27

1

9

3

6

9

1

9

3

3

9

2

27

2

3

9

1

27

3

1

27

BOHB (Bayesian Optimization and Hyperband)#

BOHB combines Bayesian optimization with Hyperband’s early stopping. It builds a statistical model of which hyperparameter regions tend to do well, then uses Hyperband’s resource scheduling to allocate more epochs to the most promising trials. The result is faster convergence than vanilla Bayesian search when you have a large number of hyperparameters.

Key parameters:

  • automl_kde_samples: Number of completed trials used to fit the performance model. Raise this if the search feels too random early on; lower it to react faster to new results.

  • automl_top_n_percent (default: 15): What fraction of past trials counts as “good” when building the model. Lower = more selective = more focused search.

  • automl_min_points_in_model: How many trials to run randomly before the model kicks in. Useful for very cold starts where you have no data yet.

BFBO (Batch First Bayesian Optimization)#

A variant of Bayesian optimization that generates a full batch of recommendations at once before waiting for results. This means you can fill all your available GPUs immediately rather than waiting for each experiment to finish before picking the next configuration.

Key parameters:

  • automl_max_recommendations: Total experiments to run (same meaning as the bayesian algorithm’s max_recommendations).

ASHA (Asynchronous Successive Halving Algorithm)#

ASHA is like Hyperband but without requiring all workers to finish a stage before the next begins. Workers that finish early move straight to the next round, keeping all GPUs busy even when some experiments run faster than others. Best for heterogeneous clusters or when your experiments have very different runtimes.

Key parameters:

  • automl_max_concurrent: How many trials to run at the same time.

  • automl_max_trials: Total trials before stopping the search.

PBT (Population-Based Training)#

PBT is fundamentally different from the other algorithms: it does not pick hyperparameters once at the start and then train. Instead it runs a group of models in parallel and periodically replaces the worst-performing ones with copies of the best-performing ones, then randomly mutates their hyperparameters. This lets the learning rate, batch size, and other parameters evolve during training — which is especially useful for schedules that are hard to set in advance.

Key parameters:

  • automl_population_size: Number of parallel models. Larger populations explore more of the hyperparameter space but require more GPUs.

  • automl_eval_interval: How often (in epochs) the population is ranked and the weakest members are replaced.

DEHB (Differential Evolution with Hyperband)#

DEHB pairs Hyperband’s multi-fidelity scheduling (cheap early evaluation, expensive late evaluation) with differential evolution — a mutation-based global optimizer that explores the hyperparameter space by combining existing solutions. It tends to outperform vanilla Hyperband on very large or continuous search spaces.

Key parameters:

  • automl_mutation_factor: How far a mutant configuration can move from its parent (0–2). Higher values = more exploration; lower values = more exploitation.

  • automl_crossover_prob: Fraction of hyperparameters inherited from the mutant vs. the original (0–1). Higher = mutant-leaning; lower = parent-leaning.

Hyperband with Early Stopping (hyperband_es)#

A drop-in extension of Hyperband that can terminate a trial mid-stage if its validation metric falls below a threshold — even before the stage’s epoch budget is exhausted. This is useful when you know that any run below a certain accuracy at epoch 5 is never going to recover by epoch 20.

Key parameters:

  • automl_early_stop_threshold: The metric value below which a trial is stopped. Set this based on the minimum acceptable validation accuracy/mIoU/etc. for your task.

  • automl_min_early_stop_epochs: A trial must run at least this many epochs before it can be early-stopped. Prevents killing trials before they have had a chance to warm up.

Hyperband Parameter Auto-Adjustment Mechanism#

By default, the Hyperband pamameter R is set to 27 and nu is set to 3. Based on these values and the training max_epoch of each experiment, the hyperband parameter auto-adjustment mechanism calculates the epoch_multiplier by max_epoch^(log(nu)/log(R)). Here, if the calculated epoch_multiplier is less than 3, then R is reduced by 1 and the epoch_multiplier is recalculated until this value is equal to or greater than 3.

Prerequisites#

The agent reads applications/tao-run-automl/SKILL.md to plan an AutoML run. The only env-var prerequisites are WANDB_API_KEY (for trial tracking) and NVIDIA_API_KEY (only when you pick the llm, hybrid, or autoresearch algorithm). Export them in the same shell before starting your agent — see Getting Started with NVIDIA TAO Toolkit for the full install and environment setup.

Running AutoML from Your Agent#

Drive the applications/tao-run-automl skill from a natural-language prompt. For example:

Run TAO AutoML on visual-changenet using the ``bayesian`` algorithm for 20
trials. Dataset is at ``s3://my-bucket/pcb-data/``. Track in WandB under
project ``pcb-aoi-hpo``. Run on the local Docker daemon.

The agent checks model-level AutoML enablement and the selected action schema before it proposes a search space. A model can support one action without supporting every action.

Note

Run one AutoML job at a time per backend. Each trial within an AutoML run is a separate scheduled job, and queueing trials from multiple simultaneous AutoML runs extends the time to complete any single trial.

Getting Started#

Required Fields#

The agent fills the following fields into the AutoML specification. Surface them by name in your prompt or accept the defaults:

  • model_name: The TAO model to optimize. Ask the agent to enumerate the supported model and action pairs if you are unsure.

  • action: The model action to optimize. The default is train. AutoML also supports schema-backed actions such as distill, prune, and quantize.

  • action_inputs: The dataset, parent checkpoint, teacher checkpoint, calibration data, or other artifact that the selected action requires. The agent uses the exact specification paths and directory layouts from the corresponding model skill.

  • automl_algorithm: The AutoML algorithm to use. Refer to the AutoML Algorithm Explanation section for a brief explanation of each algorithm.

    • bayesian: Adaptively proposes new hyperparameters based on past experiments. This algorithm generally provides better accuracy results, but with a longer execution time for all experiments, when compared to Hyperband.

    • hyperband: Hyperband accelerates the expensive hyperparameter optimization using random search with adaptive resource allocation.

    • bohb: Combines Bayesian optimization with Hyperband for sample-efficient search.

    • bfbo: Batch First Bayesian Optimization — generates batches of recommendations in parallel.

    • asha: Asynchronous Successive Halving — well-suited for distributed and heterogeneous compute.

    • pbt: Population-Based Training — adapts hyperparameters online during training.

    • dehb: Differential Evolution with Hyperband — effective for large, high-dimensional search spaces.

    • hyperband_es: Hyperband with early stopping — adds per-trial threshold-based early termination.

    • llm: LLM-guided search. Proposes hyperparameters using an LLM brain (NVIDIA NIM, OpenAI, or any OpenAI-compatible endpoint). Requires NVIDIA_API_KEY or AUTOML_LLM_API_KEY.

    • hybrid: Combines LLM proposals with classical Bayesian/Hyperband search. Requires the same LLM credentials as llm.

    • autoresearch: Agentic search that explores the space autonomously using an LLM brain. Requires the same LLM credentials as llm.

Action-Specific AutoML#

Set the action when constructing the runner. The runner loads the matching action contract, default specification, and search schema from the selected model skill:

from tao_automl.runner import AutoMLRunner

runner = AutoMLRunner(
    sdk=sdk,
    skill_dir="/path/to/tao-skills/skills/models/<model-skill>",
    action="quantize",
)

Use the following algorithm guidance for compression actions:

  • Treat distill as a training-like action when it runs for multiple epochs and writes checkpoints. Multi-fidelity algorithms are appropriate only when the action exposes a meaningful epoch or rung budget.

  • Use bayesian or bfbo by default for single-shot prune and quantize actions. Use eval_fn when a follow-up evaluation or inference action must measure the optimization metric.

  • Use Hyperband, ASHA, BOHB, or DEHB for pruning and quantization only when the action schema exposes a representative partial-fidelity budget, such as retraining epochs or calibration samples.

The action schema supplies the searchable fields, ranges, choices, and default parameters. For a fixed quantization backend, AutoML removes known-incompatible mode and algorithm combinations from the generated search space.

Note

Non-training actions often require a checkpoint or artifact that an earlier action produced. Supplying an action name does not create that prerequisite; pass the required input through the exact field that the model skill declares.

Packaged Model Search Spaces#

AutoML provides the following packaged search spaces:

  • The DINOv3 continual self-supervised training search tunes dataset.batch_size and dataset.workers by default. Treat train_loss as a search signal and use downstream evaluation to select a representation for deployment.

  • The Fast Foundation Stereo training search tunes train.optim.lr and train.optim.lr_decay by default. The model skill packages an AutoML action schema only for train.

  • The SegFormer and Visual ChangeNet training schemas expose the vit_small_dinov3, vit_small_plus_dinov3, vit_base_dinov3, vit_large_dinov3, and vit_huge_plus_dinov3 backbone choices.

Execution Runtime#

Packaged TAO model actions remain container-backed by default. The runner uses the model skill’s resolved container_image for every recommendation and related evaluation. Installing the AutoML controller or TAO software development kit (SDK) in a Python environment does not replace that model-action container.

AutoML also supports external actions that explicitly declare execution.type: python_script. For this mode, install nvidia-tao-automl[virtualenv] and use VirtualEnvSDK:

from tao_automl.runner import AutoMLRunner
from tao_sdk.platforms.virtualenv import VirtualEnvSDK

sdk = VirtualEnvSDK(
    venv_path="/work/venvs/model",
    work_dir="/work/automl-jobs",
)
runner = AutoMLRunner(
    sdk=sdk,
    skill_dir="/work/external-model-skill",
    action="train",
)
result = runner.run(
    automl_settings={
        "algorithm": "bayesian",
        "metric": "accuracy",
        "direction": "maximize",
        "automl_max_recommendations": 4,
        "run_baseline": False,
        "run_final_evaluation": False,
    },
    gpu_count=0,
)

The model skill must include schemas/<action>.schema.json and a Python-script execution mapping in skill_info.yaml. The SDK serializes each recommendation as JSON, YAML, or TOML and starts the script with the selected environment’s Python interpreter without activating the environment or invoking a shell. Each recommendation receives isolated configuration, log, result, and exit records. The runner can recover these records after the controller restarts.

Do not pass image in Python-script mode. This mode accepts only declared local inputs, so stage remote data before submission. Pass explicit gpu_ids when the action requires exclusive device selection; gpu_count alone records requested capacity but does not reserve local GPUs.

Optional Fields#

For each model, TAO has set a default set of parameters to run the AutoML search. You can add new valid parameters via additional_automl_parameters or remove some parameters from the default list via remove_default_automl_parameters. The eligible parameter list for each model is documented in that model’s skill.

  • additional_automl_parameters: Add additional parameters to the AutoML search algorithm using this list of strings (e.g. additional_automl_parameters = ['parameter1','parameter2']).

    Any parameter in the hyperlink table that does not have the automl_enabled column set to True/False can be added to the AutoML search space.

    For example, for DetectNet_v2:

    • dataset_config.target_class_mapping can’t be added to the additional_automl_parameters list as it is not eligible to be included in the search space.

    • training_config.regularizer.weight doesn’t make sense to add to this list, as it is already enabled.

    • augmentation_config.preprocessing.output_image_width can be added, as the automl_enabled column is set to neither True nor False.

  • remove_default_automl_parameters: Remove parameters that are enabled by default for AutoML search (e.g. remove_default_automl_parameters = ['parameter1','parameter2']).

    Any parameter in the hyperlink table that has the automl_enabled column set to True can be removed from the AutoML search space.

    For example, for DetectNet_v2, training_config.regularizer.weight can be removed from the AutoML search space.

Treating a List Parameter as a Continuous Range

Some parameters have a predefined list of valid values (valid_options). By default, the AutoML optimizer samples only from those values. If you want the optimizer to treat the parameter as a continuous float instead — exploring values between and beyond the list items — set disable_list: True for that parameter:

custom_automl_ranges = {
    "train.learning_rate": {
        "disable_list": True  # Explore as a continuous float, not just the listed values
    }
}

This gives the optimizer more freedom and can find better configurations when the predefined list is sparse.

Weighted List Options

For parameters with discrete list options (valid_options), you can specify weights to prioritize certain options over others during AutoML search. This is particularly useful when you know certain hyperparameter values are more likely to perform well.

  • option_weights: Assign weights to list options to control their sampling probability. The weights must be positive numbers and their length must match the number of valid options.

    For example, if a parameter has valid_options: [0.1, 0.5, 1.0, 2.0] and you want to favor the middle values, you can set:

    custom_automl_ranges = {
    "parameter_name": {
        "option_weights": [0.1, 0.4, 0.4, 0.1]  # Higher weights = higher probability
    }
}

    The weights are automatically normalized, so [1, 2, 2, 1] is equivalent to [0.1, 0.4, 0.4, 0.1].

Multi-Objective Optimization#

Use automl_settings["objectives"] to optimize more than one metric. Each objective accepts metric, direction, weight, and scale:

automl_settings = {
    "algorithm": "bayesian",
    "objectives": [
        {
            "metric": "accuracy",
            "direction": "maximize",
            "weight": 1.0,
        },
        {
            "metric": "latency",
            "direction": "minimize",
            "weight": 1.0,
            "scale": 100.0,
        },
    ],
}

The first entry is the primary metric. The eval_fn callback or metric artifacts for each successful recommendation must provide all configured objective values, for example {"accuracy": 0.88, "latency": 10.0}. AutoML rejects missing or non-finite values.

AutoML orients each value so that larger is better, divides by scale, multiplies by weight, and sums the contributions. The search algorithms use this scalar score to select one best recommendation. The result also contains the raw objective_values, the objective_score, and a pareto_front containing every non-dominated successful recommendation.

For the common accuracy-versus-latency case, you can use multi_objective: true (or include_latency: true) with latency_metric, latency_weight, and latency_scale. Use the explicit objectives list when you need more than two objectives or want a fully reproducible configuration. The metric and direction settings define a single-objective run.

Specification Value Handling#

AutoML normalizes recommendation values before persistence and submission. Booleans, integers, floating-point numbers, strings, lists, and mappings retain their JSON-compatible types. AutoML converts NumPy scalars and arrays to equivalent Python values and converts paths to strings. It rejects non-finite numbers, circular structures, non-string mapping keys, and unsupported objects rather than silently stringifying them. For explicit Python-script actions, AutoML also validates the complete merged specification against the selected action schema before launch.

Algorithm-Specific Parameters#

There are some algorithm-specific parameters set to default values that determine the AutoML experiment schedules. You can optionally modify them as follows:

Bayesian

  • automl_max_recommendations — The maximum number of full-scale training experiments to run. The default value is 20. Setting this value to 10 runs 10 training experiments in sequential order, as the training configuration file of nth experiment is computed from the (n-1)th experiment. At the end of 10 experiments, the algorithm returns the training configuration file and binary weights to the experiment that achieved the best accuracy.

Hyperband

  • automl_R — The maximum resources (i.e. the number of recommendations and maximum epochs). The default value is 27 and is adjusted as explained in the Hyperband Parameter Auto-Adjustment Mechanism section.

  • automl_nu — The proportion of recommendations discarded in each use of the SuccessiveHalving algorithm. The default value is 3.

  • epoch_multiplier — The number of epochs, determined by multiplying this value with the per-trial resource (r). The default value is 10 and can be adjusted as explained in the Hyperband Parameter Auto-Adjustment Mechanism section.

    The values of R and nu are computed to determine the number of experiments to run within each stage of the Hyperband run and the corresponding number of epochs for each experiment. For example, setting "R=27, Nu=3, epoch_multiplier=10" runs three stages of experiments, as described in Table 3 of the AutoML Algorithm Explanation section:

    • The first stage proposes 27 new recommendations to run for 10 epochs. Then, Hyperband keeps the 9 (27/3) best performing recommendations to run another 30 (10*3) epochs, and repeats until one recommendation remains.

    • The second and third stages respectively propose 9 and 3 new recommendations (based on Table 3) and follow the same procedure.

    • After all the experiments conclude, Hyperband returns the training configuration file and binary weights to the experiment that achieved the best accuracy.

BOHB ("automl_algorithm": "bohb")

  • automl_kde_samples — trials used to fit the internal performance model (raise if search feels too random).

  • automl_top_n_percent — fraction of top trials that define “good” configs (default 15; lower = more selective).

  • automl_min_points_in_model — random trials to run before model-guided search begins.

BFBO ("automl_algorithm": "bfbo")

  • automl_max_recommendations — total experiments (same meaning as Bayesian max_recommendations).

ASHA ("automl_algorithm": "asha")

  • automl_max_concurrent — maximum trials running at the same time.

  • automl_max_trials — total trial budget before the search stops.

PBT ("automl_algorithm": "pbt")

  • automl_population_size — number of parallel models (needs at least this many GPU slots).

  • automl_eval_interval — epochs between population ranking and replacement rounds.

DEHB ("automl_algorithm": "dehb")

  • automl_mutation_factor — exploration vs. exploitation trade-off (0–2; default ~0.5).

  • automl_crossover_prob — fraction of parameters inherited from the mutant (0–1; default ~0.5).

Hyperband-ES ("automl_algorithm": "hyperband_es")

  • automl_early_stop_threshold — validation metric below this value triggers trial termination.

  • automl_min_early_stop_epochs — minimum epochs a trial must run before early stopping applies.

SLURM Auto-Resume

When running AutoML on SLURM clusters, interrupted trials are automatically re-queued and resumed from the last available checkpoint. No additional configuration is required; the SLURM backend detects preemptions and restarts the affected trial automatically.

Artifact and Checkpoint Retention#

automl_delete_intermediate_ckpt defaults to True. The runner uses the job-scoped cleanup application programming interface from the platform SDK to delete failed and non-best terminal artifacts after they are no longer needed. The runner protects the active recommendation, the current best result, promotion and resume parents, and the latest decision window for multi-fidelity algorithms. Multi-objective runs retain every Pareto-front artifact. Hybrid runs retain successful artifacts when the runner cannot verify a full-fidelity winner.

For training actions, automl_checkpoint_retention_strategy controls which checkpoint files each retained job writes:

  • auto (default) selects best when the merged specification exposes train.checkpointer and otherwise selects terminal.

  • best retains one top-ranked monitored checkpoint. Use this only with trainers whose checkpointer supports replacing periodic checkpoints.

  • terminal saves at the recommendation’s effective final epoch and preserves the rung budget used by Hyperband-family algorithms.

The runner cleans up job-scoped results from Docker, Kubernetes, Brev, SLURM, and virtual-environment execution when the selected SDK can prove ownership of the output route. Cleanup-aware backends reject output routes they cannot reclaim safely before the first recommendation launches. Set automl_delete_intermediate_ckpt: false only when you intentionally need all trial artifacts and have planned their storage and cleanup.

Runner Recovery and Cancellation#

The runner persists active platform job identities in the AutoML workspace. With resume=True and the full timestamped workspace path, it reconciles in-flight jobs, recovers terminal metrics and artifacts, and avoids duplicate submissions before requesting new recommendations. When the runner handles SIGINT or SIGTERM, it requests cancellation and waits for the backend to confirm that each writer is terminal. If the backend cannot confirm cancellation, the runner retains the active-job record and artifacts for a later resume.

You can change action-related specification parameters such as batch size, learning rate, pruning amount, quantization algorithm, and checkpoint frequency for each recommendation. These parameters often depend on the computing hardware specifications, such as GPU memory size.

AutoML Outcomes#

Stable Runner Result#

AutoMLRunner.run() returns a stable result dictionary with the following top-level fields:

  • best contains the recommendation ID, the specs mapping, the primary metric value, the objective values and score, and any automatic adjustments.

  • progress reports the completed and total recommendations, best metric, recommendation ID, and algorithm.

  • baseline and final_evaluation report the evaluation status, metric, source or failure reason, record path when available, and comparisons.

  • history contains each recommendation’s metrics, objective values, status, failure reason, and adjustments.

  • pareto_front contains the non-dominated recommendations for multi-objective runs. Single-objective results omit this field.

Use the standard formatter when presenting this result:

from tao_automl import format_result

result = runner.run(...)
print(format_result(result))

Programmatic callbacks receive Recommendation objects. A recommendation also implements the mapping interface for its specification overrides. You can use rec["train.optim.lr"], rec.get("train.optim.lr", 1e-4), and dict(rec) while attributes such as rec.id and rec.result remain available.

The runner rejects unknown automl_settings keys before launching a job and suggests close matches for misspellings. The runner accepts num_recommendations as an alias for automl_max_recommendations; passing conflicting values for both names is an error.

Failed recommendations remain visible in history with their failure reason. Bayesian search does not fit their synthesized fallback score into the Gaussian process, because infrastructure failures do not provide a trustworthy metric; it proposes the next recommendation from successful observations instead.

While an AutoML run is in flight, the agent surfaces the following per-trial status payload on demand:

{
    "c9efeab0-0756-47d5-b6ae-3b51e2c02b34": {
        "detailed_status": {
            "message": "AutoML run is successful with best checkpoints under /results/c9efeab0-0756-47d5-b6ae-3b51e2c02b34",
            "status": "SUCCESS"
        }
    },
    "ea7bba9f-c88f-41ba-8b44-75b9c6fc1ec5": {
        "detailed_status": {
            "date": "5/6/2025",
            "message": "train action completed successfully for segformer",
            "status": "SUCCESS",
            "time": "15:36:29"
        },
        "epoch": 9,
        "eta": "0:00:00",
        "kpi": [
            {
                "metric": "train_loss",
                "values": {
                    "0": 0.7115289568901062,
                    "9": 0.3484381139278412
                }
            },
            {
                "metric": "val_loss",
                "values": {
                    "8": 0.3261469602584839,
                    "9": 0.3261469602584839
                }
            },
            {
                "metric": "val_miou",
                "values": {
                    "8": 0.5810916124069727,
                    "9": 0.5810916124069727
                }
            },
        ],
        "max_epoch": 9,
        "time_per_epoch": "0:00:01.477164",
    }
}

Understanding the AutoML Status#

The status response provides detailed information about the AutoML run and its experiments:

  • AutoML Brain ID (e.g., c9efeab0-0756-47d5-b6ae-3b51e2c02b34): This is a unique identifier for the AutoML orchestrator that manages the entire AutoML run. The status shows the overall run information, including where the best checkpoints are stored.

  • Experiment ID (e.g., ea7bba9f-c88f-41ba-8b44-75b9c6fc1ec5): Each experiment within the AutoML run has its own unique identifier. The AutoML brain runs multiple experiments with different hyperparameter configurations, each with its own UUID.

  • Detailed Status: Contains information about the experiment’s current state:

    • message: Descriptive information about the experiment status

    • status: Current state (e.g., “SUCCESS”)

    • date and time: When the status was last updated

  • Training Progress:

    • epoch: Current training epoch

    • max_epoch: Total number of epochs for this experiment

    • eta: Estimated time remaining for the current experiment

    • time_per_epoch: Average time taken per epoch

  • Performance Metrics (kpi): Lists various metrics tracked during training:

    • train_loss: Training loss values at different epochs

    • val_loss: Validation loss values

    • val_miou: Validation mean intersection over union (for segmentation tasks)

    • Other metrics specific to the model type

This status information helps you monitor the progress of your AutoML run, track the performance of individual experiments, and understand how the hyperparameter optimization is proceeding.

Note

The eta for AutoML experiment completion, that is just the training time, in addition to the following times, which are dependent on the config values you set:

  • The time for evaluation at every n epochs

  • The time for miscellaneous actions like saving a checkpoint at every n epochs

  • The time for loading the model from disk for resuming training in Hyperband

Results of AutoML Experiments#

The below table provides several experimental results of network models on different AI applications.

  • These numbers are profiled with the default AutoML specific parameters: max_recommendation=20 for Bayesian and R=27, nu=3 for Hyperband. For some network models (efficientdet, mask_rcnn, multitask_classification), the default automl parameter is adjusted by the Hyperband Parameter Auto-adjustment Mechanism to be R=9, nu=3 for Hyperband.

  • All experiments are executed on a single GPU (Tesla V100) with Intel Xeon CPU E5-2698 v4.

  • When using multi-GPU mode, the time taken is expected to scale accordingly; similarly, using more powerful GPUs will reduce the execution time.

Table1. The estimated time for an AutoML (Bayesian/Hyperband) experiment for each network model

model

epoch

dataset

single experiment

Bayesian

Hyperband

detectnet_v2

80

FLIR20

45.7min

913.2min

534.2min

efficientdet

6

FLIR20

82min

1640min

410min

faster_rcnn

80

FLIR20

252min

5040 min

2948.4min

retinanet

100

FLIR20

75min

1500min

702min

ssd

80

FLIR20

174min

3480min

2035min

yolo3

80

FLIR20

60min

1200min

702min

yolo4

80

FLIR20

110min

2200min

1287min

yolo4_tiny

80

FLIR20

87min

1740min

1017.9min

mask_rcnn

5

FLIR20

66min

1320min

237.6min

lprnet

24

OpenALPR

1.1min

22min

39.2min

multiclass_classification

80

Pascal VOC

80min

1600min

1279.7min

multitask_classification

10

Fashion Product

12.5min

250min

799.8min

unet

50

ISBI

4.5min

90min

98.5min