Provider-backed Inference

OpenShell grants model-provider access through provider profiles and sandbox attachments. The workload calls the provider’s native API. OpenShell evaluates the request against the profile-derived network policy and substitutes the real credential only at an endpoint authorized by that profile.

This keeps the complete access contract in one place:

Define and Attach a Hosted Provider

Start from an existing profile, review its access, and import it under a new ID. For example, export the NVIDIA profile:

openshell provider profile export nvidia -o yaml > nvidia-native.yaml

In nvidia-native.yaml, change id to nvidia-native, give the profile a distinct display_name, and set binaries to the paths that may call the API. Keep the credential and endpoint definitions you intend to grant. For a Python workload, the edited fields can look like:

id: nvidia-native
display_name: NVIDIA Native API
binaries:
  - /usr/bin/python3
  - /usr/bin/python3.13
  - /usr/local/bin/python
  - /sandbox/.venv/**

Lint and import the complete edited profile, then create a provider from its new ID:

openshell provider profile lint -f nvidia-native.yaml
openshell provider profile import -f nvidia-native.yaml
openshell provider create \
  --name nvidia-prod \
  --type nvidia-native \
  --from-existing
openshell sandbox create \
  --name inference-demo \
  --provider nvidia-prod \
  -- python app.py

The imported provider profile supplies NVIDIA_API_KEY as an opaque placeholder and allows the profile’s native endpoint. Configure the client with the real model identifier:

import os
from openai import OpenAI
client = OpenAI(
    base_url="https://integrate.api.nvidia.com/v1",
    api_key=os.environ["NVIDIA_API_KEY"],
    timeout=300,
)
response = client.responses.create(
    model="nvidia/nemotron-3-nano-30b-a3b",
    input="Hello!",
)
print(response.output_text)

Use the same export, edit, and import flow for OpenAI, Anthropic, or another hosted service. Set a new profile ID, retain only the endpoints and credentials the workload needs, and name the actual client binaries. Provider attachment does not select or rewrite a model.

This release still loads built-in profiles for compatibility, so existing providers continue to resolve their profiles. Treat those built-ins as starting templates for new provider definitions rather than the primary setup workflow.

Attach a Provider to a Running Sandbox

openshell sandbox provider attach inference-demo nvidia-prod
openshell sandbox provider list inference-demo

Running sandboxes poll for provider and effective-policy changes. Launch a new process after attachment so it receives the new credential placeholder:

openshell sandbox exec inference-demo -- python app.py

An already-running process does not gain new environment variables. Credential rotation and detach still take effect immediately at placeholder resolution.

Detach the provider to revoke its policy and credential access:

openshell sandbox provider detach inference-demo nvidia-prod

Use a Custom or Self-hosted Endpoint

Do not reuse the built-in openai or anthropic profile for an alternate host. Those profiles bind credentials to the public vendor endpoints. Import a profile that names the intended endpoint and the binaries that may call it.

For a credentialless Ollama server on the gateway host, save this profile as ollama-openai.yaml:

id: ollama-openai
display_name: Ollama
description: Host-local Ollama OpenAI-compatible API
category: inference
inference_capable: true
credentials: []
endpoints:
  - host: host.openshell.internal
    port: 11434
    protocol: rest
    access: read-write
    enforcement: enforce
binaries:
  - /usr/bin/curl
  - /usr/local/bin/curl
  - /usr/bin/python3
  - /usr/local/bin/python
  - /sandbox/.uv/python/**
  - /sandbox/.venv/**

Import the profile, create an instance, and attach it:

openshell provider profile lint -f ollama-openai.yaml
openshell provider profile import -f ollama-openai.yaml
openshell provider create --name ollama --type ollama-openai
openshell sandbox create \
  --name ollama-client \
  --provider ollama \
  --env OPENAI_BASE_URL=http://host.openshell.internal:11434/v1 \
  -- python app.py

Use any non-empty placeholder value if the client library requires an API key for a server that does not authenticate requests:

import os
from openai import OpenAI
client = OpenAI(
    base_url=os.environ["OPENAI_BASE_URL"],
    api_key="unused",
)
response = client.chat.completions.create(
    model="qwen3.5:0.8b",
    messages=[{"role": "user", "content": "Hello"}],
)

For an authenticated alternate endpoint, declare a credential in the custom profile, bind it to that endpoint, and create the provider from the original credential source. OpenShell never exports stored credential values.

Verify Access

Inspect the effective policy, including provider-derived entries:

openshell policy get inference-demo --full

Then run a native request from a new sandbox process. A successful request confirms endpoint policy, binary attribution, credential substitution, DNS, and upstream service behavior.

If the request is denied:

• Confirm the provider is attached with openshell sandbox provider list.
• Confirm the caller binary and native endpoint appear in openshell policy get <sandbox> --full.
• If the provider was attached after the process started, launch a new process.
• If the sandbox uses a gateway global policy override, add the native endpoint there because a global override suppresses provider-derived policy layers.
• Inspect logs for credential_endpoint_mismatch. That error means policy admitted the request but the provider profile did not authorize its credential at the requested endpoint.

Migrate from Managed Inference Routes

OpenShell removed the workspace-global managed inference route and the openshell inference commands. Upgrades remove stored route records. Provider records, provider refresh configuration, and existing sandbox attachments are preserved. Built-in profiles remain available during the transition, but new setups should import an explicit profile under a new ID.

The old route cannot be converted automatically. It applied one provider and model to every sandbox in a workspace, while provider attachments intentionally grant access to selected sandboxes. OpenShell cannot infer which sandboxes should receive that authority.

Now

Earlier releases configured one shared route:

openshell provider create --name nvidia-prod --type nvidia --from-existing
openshell inference set \
  --provider nvidia-prod \
  --model nvidia/nemotron-3-nano-30b-a3b \
  --timeout 300

Clients called https://inference.local, supplied a placeholder key and model, and relied on OpenShell to rewrite the request.

After

Export the old provider type’s profile, edit its ID and access contract, import it, and create a replacement provider from the original credential source:

openshell provider profile export nvidia -o yaml > nvidia-native.yaml
# Edit id, display_name, endpoints, and binaries in nvidia-native.yaml.
openshell provider profile lint -f nvidia-native.yaml
openshell provider profile import -f nvidia-native.yaml
openshell provider create \
  --name nvidia-native-prod \
  --type nvidia-native \
  --from-existing
openshell sandbox provider attach inference-demo nvidia-native-prod

OpenShell does not export stored credential values or change a provider’s profile type in place. If the original credential source is unavailable, the compatibility built-in lets you attach the preserved provider while you arrange credential rotation into the replacement provider.

Update each workload to:

1. Call the provider’s native endpoint.
2. Read the credential variable declared by its profile.
3. Send the real provider model identifier.
4. Configure request timeouts in the client.
5. Use the provider’s native request format.

Launch a new process after attachment and verify a native request before upgrading production workloads. Code that still calls inference.local fails DNS resolution because OpenShell no longer resolves or trusts that virtual host.

Migration Checklist

Before upgrading, record the old provider, model, and timeout with the previous release’s openshell inference get. Identify actual consumers, export and edit the source profile, import it under a new ID, and create the replacement provider from the original credential source. Attach it only to those sandboxes, migrate their clients, and test the native path while the old route is still available. Delete the old route on the previous release to expose missed consumers. After upgrading the gateway, delete and recreate every pre-upgrade sandbox so no old supervisor, DNS entry, trust material, or cached route survives. Reapply only the provider attachments each replacement sandbox needs.

Special cases require additional work:

• A provider with an alternate OPENAI_BASE_URL or ANTHROPIC_BASE_URL needs an endpoint-bearing custom profile for that host.
• Host-local services need host.openshell.internal or a reachable LAN/service hostname, not 127.0.0.1 or localhost.
• Google Vertex AI clients must use the native Vertex endpoint and authentication behavior. See Google Vertex AI.
• A bridge-fronted AWS Bedrock deployment needs a custom profile that declares the bridge endpoint and allowed client binaries.

Security Differences

Provider attachment preserves credential non-disclosure: workloads receive opaque placeholders, and the proxy substitutes a real credential only after network policy and endpoint binding both pass. Native requests are no longer filtered or rewritten by a model-specific router. The provider profile’s L7 rules therefore define the allowed API surface, and the workload controls headers, model selection, request shape, streaming, and timeout behavior.

Next Steps

• Profiles
• Providers
• Customize Sandbox Policies
• Google Vertex AI