nemo_evaluator.core.resources

Module Contents

Functions

aggregate_runtime_metrics	Aggregate all run data from run_times directory.
get_token_usage_from_cache	Extract token usage metrics from the cache database.
get_token_usage_from_cache_db	Extract token usage metrics from the cache database.
monitor_memory_usage	Run func(*args, **kwargs) while polling RSS via psutil. Returns (func_return_value, peak_rss_bytes, peak_tree_rss_bytes) where:

API

nemo_evaluator.core.resources.aggregate_runtime_metrics(output_dir: str) → dict[str, Any]

Aggregate all run data from run_times directory.

nemo_evaluator.core.resources.get_token_usage_from_cache(cache_dir: str) → dict

Extract token usage metrics from the cache database.

nemo_evaluator.core.resources.get_token_usage_from_cache_db(

cache_db_path: str | pathlib.Path,

) → dict

Extract token usage metrics from the cache database.

Args: cache_db_path: Path to the SQLite cache database file

Returns: Dictionary containing token usage metrics: { “total_tokens”: int, “prompt_tokens”: int, “completion_tokens”: int, “total_cached_requests”: int } Returns empty dict if no data found or error occurs.

nemo_evaluator.core.resources.monitor_memory_usage(

func,

*args,

interval_ms,

cache_dir: str | None = None,

output_dir: str | None = None,

**kwargs,

) → tuple[nemo_evaluator.api.api_dataclasses.EvaluationResult, dict[str, Any]]

Run func(*args, **kwargs) while polling RSS via psutil. Returns (func_return_value, peak_rss_bytes, peak_tree_rss_bytes) where:

• peak_rss_bytes: peak memory usage of the main process

• peak_tree_rss_bytes: peak memory usage of the entire process tree (main + children)