Running NIM on WSL2

This section explains how to set up and run Riva ASR NIM on WSL2, whether the client machine is the same as or different from the WSL2 host.

Prerequisites

WSL2 is required for hosting any NIM. Refer to the official NVIDIA NIM on WSL2 documentation for setup instructions.

Make sure you complete the NGC Authentication steps.

Running NIM When the Client and WSL2 Host Are the Same

If the client and WSL2 host machine are the same, follow these steps:

1. Launch NIM on WSL2

Open WSL2 terminal and execute following commands to deploy the Parakeet 0.6b en-US streaming mode profile with batch size of 1. This profile supports one session at at time and is optimized for low memory usage.

For more information refer Launching the NIM.

export CONTAINER_ID=parakeet-0-6b-ctc-en-us
export NIM_TAGS_SELECTOR="name=parakeet-0-6b-ctc-riva-en-us,mode=str,bs=1"

docker run -it --rm --name=$CONTAINER_ID \
   --runtime=nvidia \
   --gpus '"device=0"' \
   --shm-size=8GB \
   -e NGC_API_KEY \
   -e NIM_HTTP_API_PORT=9000 \
   -e NIM_GRPC_API_PORT=50051 \
   -p 9000:9000 \
   -p 50051:50051 \
   -e NIM_TAGS_SELECTOR \
   nvcr.io/nim/nvidia/$CONTAINER_ID:latest

2. Run Client Inference

When the NIM is running, open a new WSL2 terminal and follow the steps in Running Inference.

Note

As we are deploying streaming mode profile for Parakeet 0.6b en-US model, only the client commands under section Streaming mode example will work.

To run client commands under Offline mode example, deploy the profile for offline mode.

Refer Supported Models for more information.

Running NIM When the Client and WSL2 Host Are Different

If the client is on a different machine than the WSL2 host, you need to set up port forwarding, because WSL2 uses NAT.

The process involves forwarding traffic from your Windows host’s external IP and port to the WSL instance’s internal IP and port. All these steps require a Windows terminal with admin privilege.

1. Find the WSL2 Internal IP

In the WSL2 terminal, execute this command:

hostname -I

2. Forward Traffic from Windows Host to WSL2

Open Windows terminal with admin privilege and run below commands.

Forward HTTP port

netsh interface portproxy add v4tov4 listenport=9000 listenaddress=0.0.0.0 connectport=9000 connectaddress=<wsl2-ip>

Forward gRPC port

netsh interface portproxy add v4tov4 listenport=50051 listenaddress=0.0.0.0 connectport=50051 connectaddress=<wsl2-ip>

Note

WSL2 IP addresses change after every restart, so you need to update the forwarding rule with the correct IP every restart.

3. Verify Port Forwarding

netsh interface portproxy show v4tov4

You should see an output similar to this:

Listen on IPv4:            Connect to IPv4:
Address         Port       Address         Port
--------------- ---------- --------------- ----------
0.0.0.0         9000       <wsl2-ip>       9000
0.0.0.0         50051      <wsl2-ip>       50051

4. Allow Traffic Through Windows Firewall

To allow the client from the other machine proper access, add a firewall rule for Port 8001 in Windows Powershell with admin privilege:

New-NetFirewallRule -DisplayName "Allow gRPC on Port 9000" -Direction Inbound -Action Allow -Protocol TCP -LocalPort 9000
New-NetFirewallRule -DisplayName "Allow gRPC on Port 50051" -Direction Inbound -Action Allow -Protocol TCP -LocalPort 50051

5. Run the Client Inference

Follow the steps in Run Client Inference. Make sure to change the IP address in the --server arguments to the Windows host machine IP, which is running the WSL2 NIM.