Run Local Inference with Ollama

This tutorial covers two Ollama deployments:

The self-contained Ollama community sandbox. This is the recommended path.
A host-level Ollama service shared by explicitly authorized sandboxes.

Prerequisites

Complete the Quickstart before proceeding.

Option A: Ollama Community Sandbox

The community image bundles Ollama and supported coding agents:

openshell sandbox create --from ollama

Inside the sandbox, run a local or cloud model:

ollama run qwen3.5
ollama run kimi-k2.5:cloud

Or launch a coding agent against Ollama:

ollama launch claude
ollama launch codex
ollama launch opencode

For an automated workflow:

ollama launch claude --yes --model qwen3.5

Useful starter models include qwen3.5:0.8b for smoke tests and qwen3.5 for coding and tool use. Check the Ollama model library for current model details.

Update the bundled Ollama installation with update-ollama, or request an update at sandbox start:

openshell sandbox create --from ollama -e OLLAMA_UPDATE=1

Option B: Host-level Ollama

Use this path when Ollama runs on the same machine as the OpenShell gateway and selected sandboxes should share it.

Start Ollama on a Reachable Address

curl -fsSL https://ollama.com/install.sh | sh
OLLAMA_HOST=0.0.0.0:11434 ollama serve

In another terminal, pull a model:

ollama pull qwen3.5:0.8b

Import an Endpoint-bearing Profile

Save this as ollama-openai.yaml:

id: ollama-openai
display_name: Ollama
description: Host-level Ollama OpenAI-compatible API
category: inference
inference_capable: true
credentials: []
endpoints:
  - host: host.openshell.internal
    port: 11434
    protocol: rest
    access: read-write
    enforcement: enforce
binaries:
  - /usr/bin/curl
  - /usr/local/bin/curl
  - /usr/bin/python3
  - /usr/local/bin/python
  - /sandbox/.uv/python/**
  - /sandbox/.venv/**

openshell provider profile lint -f ollama-openai.yaml
openshell provider profile import -f ollama-openai.yaml
openshell provider create --name ollama --type ollama-openai

The provider has no secret because this Ollama server does not authenticate. Its attachment still carries the endpoint and binary policy.

Attach and Verify

openshell sandbox create \
  --name ollama-client \
  --provider ollama \
  -- \
  curl http://host.openshell.internal:11434/v1/chat/completions \
    --json '{
      "model":"qwen3.5:0.8b",
      "messages":[{"role":"user","content":"hello"}],
      "max_tokens":10
    }'

For an OpenAI SDK client, set its base URL to http://host.openshell.internal:11434/v1, use any non-empty API key value the SDK accepts, and select the real Ollama model in the request.

Troubleshooting

Bind host-level Ollama to 0.0.0.0, not 127.0.0.1.
Use host.openshell.internal, not localhost, from a sandbox.
Confirm attachment with openshell sandbox provider list ollama-client.
Inspect the effective policy with openshell policy get ollama-client --full.
Run ollama ps and ollama pull <model> when the model is unavailable.
If the gateway is remote, the hostname refers to the remote gateway host, not your laptop. Use a shared service address or tunnel instead.