nemo_evaluator.core.entrypoint#

Module Contents#

Functions#

get_args

run

run_eval

CLI entry point for running evaluations.

show_available_tasks

API#

nemo_evaluator.core.entrypoint.get_args() → argparse.Namespace[source]#
nemo_evaluator.core.entrypoint.run(args) → None[source]#
nemo_evaluator.core.entrypoint.run_eval() → None[source]#

CLI entry point for running evaluations.

This function parses command line arguments and executes evaluations. It does not take parameters directly - all configuration is passed via CLI arguments.

CLI Arguments: –eval_type: Type of evaluation to run (e.g., “mmlu_pro”, “gsm8k”) –model_id: Model identifier (e.g “meta/llama-3.1-8b-instruct”) –model_url: API endpoint URL (e.g “https://integrate.api.nvidia.com/v1/chat/completions” for chat endpoint type) –model_type: Endpoint type (“chat”, “completions”, “vlm”, “embedding”) –api_key_name: Environment variable name for API key integration with endpoints (optional) –output_dir: Output directory for results –run_config: Path to YAML Run Configuration file (optional) –overrides: Comma-separated dot-style parameter overrides (optional) –dry_run: Show rendered config without running (optional) –debug: Enable debug logging (optional, deprecated, use NV_LOG_LEVEL=DEBUG env var)

Usage: run_eval() # Parses sys.argv automatically

nemo_evaluator.core.entrypoint.show_available_tasks() → None[source]#