Benchmarking LoRA Models

Parameter-Efficient Fine-Tuning (PEFT) methods allow efficient fine-tuning of large pretrained models. NIM supports Low-Rank Adaptation (LoRA), a lightweight way to tailor LLMs for specific domains. With NIM, you can load and deploy multiple LoRA adapters. Follow the Parameter-Efficient Fine-Tuning guide to load Hugging Face or NeMo adapters and pass the adapter directory to NIM through an environment variable.

After adapters are loaded, query the LoRA model like the base model by replacing the base model ID with the LoRA model name:

curl -X 'POST' \
  'http://0.0.0.0:8000/v1/completions' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
"model": "llama3-8b-instruct-lora_vhf-math-v1",
"prompt": "John buys 10 packs of magic cards. Each pack has 20 cards and 1/4 of those cards are uncommon. How many uncommon cards did he get?",
"max_tokens": 128
}'

Use AIPerf to benchmark LoRA deployments by passing adapter IDs with -m. The following example benchmarks two LoRA models after you deploy llama3-8b-instruct-lora_vnemo-math-v1 and llama3-8b-instruct-lora_vhf-math-v1 per the PEFT guide. The --model-selection-strategy {round_robin,random} flag controls whether adapters are called round-robin or at random.

export INPUT_SEQUENCE_LENGTH=200
export OUTPUT_SEQUENCE_LENGTH=200
export CONCURRENCY=10
export REQUEST_COUNT=$(($CONCURRENCY * 3))
export MODEL=meta/llama-3.1-8b-instruct
aiperf profile \
   -m $MODEL \
   --endpoint-type chat \
   --streaming \
   -u localhost:8000 \
   --synthetic-input-tokens-mean $INPUT_SEQUENCE_LENGTH \
   --synthetic-input-tokens-stddev 0 \
   --concurrency $CONCURRENCY \
   --request-count $REQUEST_COUNT \
   --warmup-request-count 10 \
   --output-tokens-mean $OUTPUT_SEQUENCE_LENGTH \
   --extra-inputs max_tokens:$OUTPUT_SEQUENCE_LENGTH \
   --extra-inputs min_tokens:$OUTPUT_SEQUENCE_LENGTH \
   --extra-inputs ignore_eos:true \
   --tokenizer meta-llama/llama-3.1-8b-instruct \
   --artifact-dir artifact/ISL${INPUT_SEQUENCE_LENGTH}_OSL${OUTPUT_SEQUENCE_LENGTH}/CON${CONCURRENCY}

Best Practices for Multi-LoRA Benchmarking

Multi-LoRA benchmarking depends on model size, adapter configuration, and load. Consider these factors:

• Base model: Both small and large models, such as Llama 3 8B and Llama 3 70B, can serve as LoRA base models. Smaller models often excel at traditional NLP tasks such as classification; larger models handle complex reasoning. LoRA lets you fine-tune large models on modest hardware—for example, a 70B model on a single NVIDIA DGX H100 or NVIDIA H100 GPU with 4-bit quantization.

• Adapters: You can experiment with adapter rank for accuracy. Operators can standardize rank for better batching. Common ranks are 8, 16, 32, and 64.

• Test parameters: Refer to Parameters and Best Practices for general guidance. LoRA-specific settings include the following:

  • Output length control: Set ignore_eos so generation continues until max_tokens. This keeps OSL consistent without task-specific training data.

  • System load: Concurrency should reflect real usage and stay within effective batching limits. For an 8B model on one GPU, up to 250 concurrent clients is a realistic upper bound.

  • Task type: Cover both generative and non-generative workloads. ISL in the 200–2000 token range and OSL in the 1–2000 token range spans classification, summarization, translation, and code generation.