Best Checkpoint Saving#

By default, TAO writes a periodic model_epoch_*.pth checkpoint at every train.checkpoint_interval and maintains a *_latest symlink that points at the most recent one. This periodic history is unbounded. It keeps every interval checkpoint and does not consider model performance on the validation metric. The best epoch is not necessarily the last one, so the *_latest checkpoint is not always the one you want to deploy.

The train.checkpointer configuration adds metric-based checkpointing: TAO monitors a validation metric and keeps the checkpoint(s) that score best on it. This lets you retain the best model without having to inspect the training logs and copy checkpoints by hand.

Parameters#

All fields live under train.checkpointer:

Parameter	Type	Default	Description
`enable_topk`	bool	`False`	Save checkpoint(s) ranked by a monitored metric. This is additive to periodic checkpointing unless `replace_periodic` is enabled. When `False` (the default), metric-based checkpointing is off and only periodic checkpoints are written.
`replace_periodic`	bool	`False`	When best-checkpoint saving is enabled, replace the unbounded periodic checkpoint callback with a single metric-best checkpoint and make it own the `*_latest` symlink. This mode always keeps exactly one best checkpoint, regardless of `save_top_k`.
`monitor`	str	`null`	Metric to rank checkpoints by; for example, `val_loss`, `val_acc`, `val_miou`, or `mAP`. Leave unset to use the network’s default monitored metric.
`mode`	str	`null`	Direction that counts as “better” for `monitor`: `min` for losses, `max` for accuracy, mAP, or mIoU. Leave unset to use the network’s default.
`save_top_k`	int	`1`	How many best checkpoints to keep (`1` = only the single best). Ignored when `replace_periodic` is enabled, which always keeps exactly one.

Additive vs. Replace Behavior#

Best-checkpoint saving has two modes, both gated on enable_topk=True:

Additive (default, replace_periodic=False ). TAO appends a second, metric-monitored checkpoint callback alongside the existing periodic one. The periodic model_epoch_*.pth history, the *_latest symlink, and the exception checkpoint are all left untouched; you simply also get up to save_top_k best-scoring checkpoints (model_best_*.pth). Use this when you want the full periodic history and the best checkpoints.
Replace ( replace_periodic=True ). TAO removes the unbounded periodic checkpoint callback and keeps a single metric-best checkpoint instead. That best checkpoint then owns the *_latest symlink, so downstream steps that consume *_latest (evaluate, export, inference) automatically pick up the best epoch. This mode always keeps exactly one checkpoint, so save_top_k has no effect. Use it to bound disk usage and make *_latest mean “best” rather than “most recent.” Exception checkpointing remains independent in either mode.

Choosing the Monitor and Mode#

monitor names the validation metric to rank by and mode gives its direction. When both are left unset, TAO falls back to the network’s default monitored metric and direction, which is the recommended starting point. Each network already declares a sensible default (for example, a detection network monitors mAP with mode: max, while a network trained to a loss uses val_loss with mode: min). Override them only when you want to rank on a different metric than the network default. If you set monitor to a metric that is not logged during validation, no best checkpoint can be selected, so make sure the metric name matches one the network reports.

When to Prefer Metric-Best Checkpoints#

Deploying the best model, not the last one. With periodic-only checkpointing you must read the logs to find the best epoch and copy it out by hand. Metric-best checkpointing keeps it for you, and replace_periodic=True additionally makes *_latest point at it.
Bounding disk usage. The periodic callback keeps every interval checkpoint. Setting replace_periodic=True keeps exactly one, and save_top_k bounds the additive mode to a fixed number.
Guarding against late-training regressions. If a run overfits or diverges near the end, the last checkpoint can be worse than an earlier one. Metric-best checkpointing preserves the peak-performing epoch regardless of what happens afterward.

Keep periodic checkpointing (leave enable_topk=False) when you need the full training history. For example, to resume from an arbitrary epoch or to analyze training dynamics.

Example#

Keep the single best checkpoint by validation loss, in addition to the periodic checkpoints:

train:
  checkpoint_interval: 1
  checkpointer:
    enable_topk: true
    monitor: val_loss
    mode: min
    save_top_k: 1

Keep the three best checkpoints by mAP, still alongside the periodic history:

train:
  checkpointer:
    enable_topk: true
    monitor: mAP
    mode: max
    save_top_k: 3

Replace the periodic history with a single best-by-mIoU checkpoint that owns *_latest:

train:
  checkpointer:
    enable_topk: true
    replace_periodic: true
    monitor: val_miou
    mode: max

The same overrides work from the command line, for example:

<task> train \
  -e /path/to/experiment.yaml \
  results_dir=/path/to/results \
  train.checkpointer.enable_topk=True \
  train.checkpointer.replace_periodic=True \
  train.checkpointer.monitor=val_miou \
  train.checkpointer.mode=max

Note

The tao-deploy TrainConfig carries the same train.checkpointer schema, so a specification file written for training remains valid across the deploy tasks.