Serve NuRec with Multiple Worker Processes#

As of 26.08, serve-grpc can run as a multi-process pool instead of a single process, which raises render throughput on one GPU. This page covers when to use the pool, how to size and launch it, and the operational behavior it requires. For the gRPC API itself, which is identical in both modes, see Use the NuRec gRPC API Server.

How the Pool Works#

Add --multiproc-workers N, with N greater than 0, to launch the pool. One owner process holds a single read-only copy of the model weights and shares it to N worker processes over CUDA IPC, behind a front proxy that presents one gRPC endpoint. At the default N = 0, serve-grpc runs the single-process server and behavior is unchanged.

Sharing weights this way is what makes the pool cheaper than running N independent servers: you pay for one copy of the weights rather than N.

Note

--multiproc-workers=1 is the real owner-plus-one-worker-plus-proxy topology, not the single-process server. Use 0 for single-process.

A pool launch is the standard serve-grpc invocation plus the pool flags, with MPS already running:

export CUDA_MPS_PIPE_DIRECTORY="/tmp/nvidia-mps-$(id -u)"
install -d -m 700 "$CUDA_MPS_PIPE_DIRECTORY"
nvidia-cuda-mps-control -d      # must be up before the server starts; see Start MPS

NUREC_DOCKER_ARGS+=(
    --volume "${CUDA_MPS_PIPE_DIRECTORY:-/tmp/nvidia-mps}:${CUDA_MPS_PIPE_DIRECTORY:-/tmp/nvidia-mps}"
    --env "CUDA_MPS_PIPE_DIRECTORY=${CUDA_MPS_PIPE_DIRECTORY:-/tmp/nvidia-mps}"
)
nurec_dev bazelisk run //:run -- serve-grpc \
    --artifact-glob "$NUREC_SRC/outputs/<RUN-ID>/artifacts/last.usdz" \
    --multiproc-workers 4 \
    --require-mps \
    --multiproc-max-in-flight 8 \
    --health-port 8081

The container requirements are unchanged from the single-process server, and every non-pool flag means what it does there — see Launching the gRPC Server. All pool processes run inside the one container so that they share an IPC namespace.

Before You Adopt the Pool#

The pool is a drop-in replacement for the single-process server — same gRPC API, same client, same flags. Confirm these five constraints apply to your deployment before you adopt it.

  • The gsplat renderer is the only validated renderer, and the pool always runs it. An unset or explicit --renderer default is coerced to gsplat at launch, and the coercion is reported on stdout.

  • NVIDIA MPS is required for throughput to scale. Without it, the workers time-slice the GPU and more than one worker buys you nothing. See Start MPS.

  • Asset editing is not supported. See Asset Editing Under the Pool.

  • Temporal Harmonizer sessions are not supported. --enable-harmonizer-temporal together with --multiproc-workers greater than 0 is a launch usage error, and the session RPCs abort with FAILED_PRECONDITION. The stateless --enable-harmonizer does work, applied per worker.

  • CUDA IPC (same-node) delivery is not supported. A session’s buffer pool lives on the one worker process that allocated it, and the pool cannot guarantee a use_cuda_ipc=true render reaches that same worker. open_ipc_session, close_ipc_session, render_rgb(use_cuda_ipc=true), render_camera(use_cuda_ipc=true), and any use_cuda_ipc=true item in batch_render_rgb/batch_render_camera all abort with UNIMPLEMENTED. Use single-process serve-grpc for CUDA IPC.

Command Line Parameters#

These parameters are additions to the serve-grpc flags documented in serve-grpc Command Line Parameters.

  • --multiproc-workers: Number of worker processes. 0 runs the single-process server (default: 0)

  • --multiproc-bringup-concurrency: How many workers to bring up at once at startup. 1 serializes startup, a value greater than 1 overlaps it, and a value of 0 or less brings up all workers in parallel. Each worker transiently holds a full copy of the weights on the GPU while loading, so this caps the startup VRAM peak (default: 1)

  • --require-mps/--no-require-mps: With more than one worker, fail fast if NVIDIA MPS is not active instead of warning (default: False)

  • --multiproc-max-in-flight: Cap concurrent in-flight renders fleet-wide. 0 is unlimited (default: 0)

  • --multiproc-admission-timeout-s: How long an over-limit render waits for an admission slot before being shed with RESOURCE_EXHAUSTED. 0 sheds immediately. A nonzero timeout requires --multiproc-max-in-flight greater than 0 (default: 0.0)

  • --multiproc-require-scene-cache/--no-multiproc-require-scene-cache: Fail fast if --download-cache-size is smaller than --cache-size. By default the server raises the download cache size instead and reports the adjustment at INFO level (default: False)

Passing any multi-process-only option explicitly at an enabled value together with --multiproc-workers=0 is a usage error, as is a negative --multiproc-workers. Passing one at its default value alongside --multiproc-workers=0 is accepted, so wrapper scripts that always emit every flag keep working.

Size the Pool#

The pool is a single-GPU topology. All workers share one device through CUDA IPC, and there is no per-worker GPU assignment. To use several GPUs, run one pool per GPU — each pinned with CUDA_VISIBLE_DEVICES and given its own --port — and load-balance across them in front.

Two different VRAM figures matter when you choose N.

Phase

Cost

Controlled by

Steady state

One copy of the weights, held by the owner, plus each worker’s own render working set.

--multiproc-workers

Startup peak

Each worker transiently holds a full copy of the weights while loading, before it aliases onto the owner’s IPC weights.

--multiproc-bringup-concurrency

The startup peak, not the steady state, is what usually runs a host out of memory. The default --multiproc-bringup-concurrency 1 serializes bring-up so the peak is one extra weight copy at a time, which is the safe choice. Raise it to shorten startup on a GPU with headroom.

Both figures depend on your artifact and your render settings, so measure them once against a representative scene rather than estimating. Start the single-process server on that scene, issue one render, and read the GPU’s memory use:

nvidia-smi --query-gpu=memory.used --format=csv

That figure is approximately one copy of the weights plus one worker’s working set. Steady-state pool use is then roughly one weight copy plus N working sets, and the startup peak adds one further weight copy per concurrent bring-up.

Throughput depends on the same variables, so size N by measurement too. Start at --multiproc-workers 2 with MPS active and raise N one step at a time while watching render latency. Throughput stops improving once the workers saturate the GPU’s compute, which shows up as latency climbing faster than the added workers return.

Validate the pool on a real GPU host, with multiple workers and MPS started externally, before you deploy it. The render-grpc client supports concurrent rendering. For concurrency limits and retry behavior, see Admission Control and Client Fan-Out.

Start MPS#

NuRec only detects MPS. It never starts or stops it, because CUDA initializes at import time, so MPS has to be up before the server launches. Start the control daemon yourself — from the container entrypoint, a Kubernetes init container, systemd, or by hand:

export CUDA_MPS_PIPE_DIRECTORY="/tmp/nvidia-mps-$(id -u)"
install -d -m 700 "$CUDA_MPS_PIPE_DIRECTORY"
nvidia-cuda-mps-control -d

Verify it is up before launching. NuRec treats MPS as active when the control pipe exists in $CUDA_MPS_PIPE_DIRECTORY, or in either /tmp/nvidia-mps or /tmp/nvidia-mps-$(id -u) when that variable is unset:

ls "${CUDA_MPS_PIPE_DIRECTORY:-/tmp/nvidia-mps}/control"

With more than one worker and no MPS, the server warns at startup and continues. Pass --require-mps to make that a startup failure instead, which is the safer choice for an automated deployment, since the warning is easy to miss in container logs.

Note

MPS is unnecessary for a single worker, and NuRec skips the check when --multiproc-workers is 1 or less.

Admission Control and Client Fan-Out#

--multiproc-max-in-flight N caps concurrent renders fleet-wide. At the default --multiproc-admission-timeout-s 0, every render beyond the cap is shed immediately as RESOURCE_EXHAUSTED rather than queued — for both singular render_camera calls and each item of a batch request.

This is the one behavior difference most likely to surprise you when you move a working client to the pool. Any client that fans renders out concurrently, including the bundled render-grpc client, hits sheds as soon as it exceeds N, even though a single-process server has no admission control and accepts the same workload. Choose one of the following.

  • Let over-limit renders wait. Set --multiproc-admission-timeout-s to a nonzero value so they block for a slot, bounded by the client deadline, instead of being shed.

  • Make the client shed-tolerant. The bundled render-grpc client retries RESOURCE_EXHAUSTED with capped exponential backoff, but its first shed installs a self-imposed concurrency limit of one, so an unbounded client serializes for the rest of the run. Set its --max-concurrent-renders N to the server’s --multiproc-max-in-flight to avoid that. See render-grpc Command Line Parameters.

Concurrency is not the only admission dimension. --allowed-image-formats restricts which camera response formats the server accepts and behaves identically under the pool, rejecting a disallowed format with INVALID_ARGUMENT before any scene load or render.

Warning

Under concurrent batch load with admission enabled, admission-blocked handler threads share the main gRPC server’s bounded thread pool. Host health on a dedicated --health-port so a Kubernetes liveness probe cannot be starved by those threads and restart the pod mid-serve.

Scene Cache and Readiness#

The pool reinterprets two of the scene-management flags described in serve-grpc Command Line Parameters.

  • --cache-size counts resident scenes fleet-wide, with coordinated LRU eviction across the owner and the workers. This differs from the single-process server, where the same flag counts loaded backends and one scene can hold several. Size it in scenes when you run the pool.

  • --max-backends-per-scene has no effect under the pool. The per-scene backend cap is a single-process concept.

A scene registered on demand through x-nre-scene-url stays reloadable by scene_id after GPU eviction. It is dropped, and a later ID-only request returns NOT_FOUND, only when its file is evicted from the on-disk download cache — at which point repeating the x-nre-scene-url header re-downloads it. See Register Scenes on Demand.

get_available_scenes reports every scene currently registered with the owner. Registration does not imply that a scene is attached to a worker or that it validated successfully.

With --test-scenes-are-valid, every startup --artifact-glob scene is loaded and attached across the pool before the proxy reports SERVING; later on-demand registrations are still validated on their first scene-using RPC. With validation disabled, a scene whose metadata is readable but which cannot load is still listed, and its first scene-using RPC returns UNAVAILABLE. A scene ID that was never registered, or whose on-disk registration was invalidated by cache eviction, returns NOT_FOUND.

Warning

In both single-process and multi-process mode, a get_available_scenes entry reflects registration rather than readiness: a listed scene may be unloaded, unattached, or unvalidated. An enumerate-then-render client can therefore get NOT_FOUND or UNAVAILABLE on a listed scene’s first render. Use get_loaded_scenes as the render-readiness check.

Inspect the Effective Configuration#

The owner answers get_server_config with the effective launch configuration: serving.mode=multiprocess plus the authoritative multiproc.* keys, including workers, renderer, max_scenes, download_cache_size, max_in_flight, admission_timeout_s, startup_scenes, and validated_startup_scenes.

Unprefixed common keys retain the values you requested. Where launch transformed a value — the --renderer default coercion, for example — the corresponding multiproc.* key holds the effective one, so read the multiproc.* value when the two disagree. The single-process-only max_workers key is not reported for the pool.

Version compatibility. The additions that came with the pool are additive only: the new serving.mode and multiproc.* discovery keys, with a byte-identical proto wire format. Integrators that negotiate capability on major.minor are unaffected. A client asserting an exact patch version will mismatch after upgrading, even on a single-process upgrade, so relax such checks to major.minor.

Deployment and Shutdown#

Run the pool under a process supervisor or cgroup, such as systemd or Kubernetes, that restarts a nonzero process exit and reaps descendant processes. The pool exits nonzero on worker death or on a scene control-plane degradation, so that the supervisor replaces the whole fleet rather than leaving a partially failed server in rotation. A client-requested shut_down exits 0 after the normal health transition and proxy drain.

  • Stop gracefully. Invoke the gRPC shut_down RPC and wait for the process to exit. The owner holds the single CUDA IPC copy of the weights, so this path transitions health away from SERVING, drains the proxy, and lets workers finish before the shared tensors are freed. In Kubernetes, use a preStop hook that invokes that RPC and waits, and give the pod enough termination grace for the hook and any in-flight renders to finish before SIGTERM is sent.

  • Do not treat SIGTERM or SIGINT as a graceful shutdown. Neither signals the pool’s drain path.

Warning

Do not hard-kill the owner under load. A render already in flight when the owner is SIGKILLed can read now-freed IPC memory, because there is no cross-process render barrier between the owner and its workers.

Asset Editing Under the Pool#

Asset editing is available only on the single-process server. Under the pool:

  • edit_assets returns in band at transport status OK with EditAssetsResponse(success=False) and a message explaining that editing is unsupported. A client that branches on response.success sees a clean failure; one that checks only the gRPC status code reads it as a success.

  • restore_model_parameters aborts with gRPC status UNIMPLEMENTED.

Use a single-process serve-grpc for in-place edits. See Use NuRec with Asset Harvester Output.

Two related features are separate mechanisms and work in both modes: per-request dynamic-actor re-posing through --enable-editing-actors, and per-frame aspect_state driving of asset edits baked offline with nurec compile-asset-edits. See Dynamic Object Types for the latter.

On this page