Observability

AITune exposes observability through logs, build logs, tuning telemetry, NVTX ranges, and optional hardware metrics. Use these signals to understand which graph was tuned, which backend won, why candidates failed, and how much hardware each phase used.

Logging level

AITune emits progress messages at INFO level (strategy steps, backend selection, build status, validation, etc.). If the root or aitune logger is left at its default level (often WARNING), those messages are not shown.

Enable INFO for better verbosity:

1
import logging
2
3
logging.basicConfig(level=logging.INFO, force=True)

With INFO (or DEBUG for more detail), you will see messages such as:

• Which strategy and backend are running
• Build and validation progress
• Selected backend and high-level timing

Backend build output

During tuning, backend builds (e.g. TensorRT) run inside an output control context. By default, their stdout, stderr, and logging are not printed to the console; they are redirected to a build log file (e.g. build.log in the backend cache directory). That keeps the terminal quiet and avoids flooding it with compiler/build logs.

To see that backend output live on the console as well, set:

$
export AITUNE_CONSOLE_OUTPUT=1

Accepted values (case-insensitive): 1, true, yes. If unset or any other value, console output from the build phase remains disabled (output only in the log file).

Behavior summary:

So:

• Default (unset): Cleaner terminal; inspect build.log under the backend cache dir if you need build details.
• Set to 1: More verbose terminal; same content is also written to the log file.

Tuning telemetry

AITune always collects tuning telemetry for each run, graph, and backend attempt. The report includes backend build status, selected backends, and failure information. When performance validation is enabled, it also includes throughput measurements and baseline throughput.

By default, telemetry is written to:

~/.cache/aitune/tuning_data/report.json

Override the path with AITUNE_TUNING_DATA_PATH:

$
export AITUNE_TUNING_DATA_PATH=/tmp/aitune-report.json

or in Python:

1
import aitune.torch as ait
2
3
ait.config.tuning_data_output_path = "/tmp/aitune-report.json"

You can also snapshot telemetry during a long-running process:

1
from aitune.torch import snapshot_tuning_data
2
3
snapshot_tuning_data()

NVTX and hardware metrics

Set AITUNE_NVTX_EVENTS=1 to emit NVTX annotations for Nsight Systems. Set AITUNE_HARDWARE_METRICS=1 to collect GPU/CPU utilization, memory, and power metrics during tuning and annotated inference windows.

Hardware metrics are written to a timestamped CSV file by default. Set AITUNE_HARDWARE_METRICS_PATH to choose a fixed output path.

See Profiling and Hardware Metrics for the full workflow.

Runtime attribution

After tuning, aitune.torch.profile(...) produces a structured per-run report that measures wall time per run and attributes CPU and device time across both AITune-managed regions and untuned module regions (including method-style entry points like vae.decode), with an explicit residual covering time spent outside any region. The profile combines PyTorch Profiler data with AITune wrapper annotations and returns in-memory data plus a Markdown renderer. A raw Chrome trace is written only when you pass an explicit trace_file.

1
import json
2
from pathlib import Path
3
4
import aitune.torch as ait
5
6
profile = ait.profile(
7
    obj=model,
8
    input_data=inputs,
9
    trace_file="trace.json",
10
)
11
12
Path("profile.json").write_text(json.dumps(profile.data, indent=2) + "\n")
13
Path("profile.md").write_text(profile.markdown())

See Performance Profile for the full workflow.

Environment variable reference

Maximum verbosity

For the most visible tuning process:

1. Set the logging level to INFO (or DEBUG) in your script so AITune’s own messages are shown.
2. Set AITUNE_CONSOLE_OUTPUT=1 so backend build output is shown on the console and still captured in the build log.
3. Set AITUNE_TUNING_DATA_PATH when you want a predictable telemetry report path.
4. Set AITUNE_NVTX_EVENTS=1 and AITUNE_HARDWARE_METRICS=1 for profiling runs.

$
export AITUNE_CONSOLE_OUTPUT=1
$
export AITUNE_TUNING_DATA_PATH=/tmp/aitune-report.json
$
python your_tuning_script.py

1
import logging
2
3
logging.basicConfig(level=logging.INFO, force=True)
4
5
# ... your tuning code ...

See also aitune.utils.setup_logging, control_output, and the tuning-data reporting API.