Install Substrate
Install Agent Substrate’s control plane and node runtime on Kubernetes, with an in-cluster RustFS object store for snapshots by default. When you finish, your cluster is ready for you to configure and run agent workloads.
This integration is experimental and uses Substrate v1.0.0-rc.1, commit
f86fde9dfb5424c8d16f613de80469bdbef06541. The S3 instructions below work
around limitations in that release’s installer.
Prerequisites
- Kubernetes 1.35 or newer. See Check cluster capabilities for the required certificate APIs and volume projections.
- Linux x86-64 (
amd64) or ARM64 (arm64) nodes. - For the default RustFS deployment, a StorageClass that can provision a
50 GiB
ReadWriteOncevolume. - For the bundled PostgreSQL installation, storage capacity for its default
500 GiB
ReadWriteOncevolume. - Git, Go 1.27 or newer,
kubectl,curl, andshasumon the installation workstation.
What this installs
The installation prepares the Substrate platform and node runtime, backed by PostgreSQL and object storage. WorkerPools, ActorTemplates, and actors are workload resources you create afterward. The diagram shows two workers as an example; each worker can host multiple actors when capacity allows.
For lifecycle and networking details, see Substrate’s architecture documentation and resource glossary.
Check cluster capabilities
Substrate 1.0.0, pinned here to v1.0.0-rc.1, requires the native
PodCertificateRequest and ClusterTrustBundle APIs and their Pod volume
projections. Both APIs must be available as certificates.k8s.io/v1 or
certificates.k8s.io/v1beta1.
For configuration instructions, see the Kubernetes
feature-gate reference
and the versioned documentation for
podCertificate projections
and clusterTrustBundle projections.
Check the API-server version:
The server version must be 1.35 or newer.
Check the node versions:
Nodes that run Substrate or its clients must use Kubernetes 1.35 or newer.
Discover the certificate APIs:
Confirm that clustertrustbundles and podcertificaterequests appear with API
version certificates.k8s.io/v1 or certificates.k8s.io/v1beta1. Other resources
in this group require no additional checks here.
These checks confirm API availability. After deploying Substrate, verify certificate issuance and volume projection in Verify the installation.
Identify node architecture
This guide assumes the Substrate control plane, atelets, workers, and actors run on nodes with the same CPU architecture. Identify that architecture before continuing:
Read the ARCH column: amd64 means x86-64, and arm64 means ARM64.
Use this architecture for workload builds and configuration. If your deployment uses both architectures, match each component’s binaries, images, and runtime assets to the nodes where it runs.
The supplied
gvisor-default SandboxConfig
selects the runtime assets for each node.
Choose nodes for Substrate (optional)
For mixed-architecture clusters or dedicated node pools, configure nodeSelector
and any required tolerations before deployment. Infrastructure and sandbox
workloads can use different nodes. Run atelet on every node eligible for workers.
Upstream manifest paths below are relative to
deps/substrate/manifests/ate-install/.
Prepare the installation tools
Fetch the pinned release for its installer and manifests:
Download kubectl-ate and its checksum from the
rc1 release.
Set ATE_CLI to the asset for your workstation: linux or darwin for the
operating system, and amd64 or arm64 for the architecture. The example below
uses Linux x86-64:
The checksum check must report OK, and the CLI must print its help text.
Keep this bin directory on PATH when managing Substrate. The deployment
step runs ate-setup directly from the checkout with go run.
Deploy in-cluster object storage
Deploy RustFS from the pinned Substrate checkout. Its manifest includes a persistent volume claim, a single RustFS server, a ClusterIP Service, a bucket initialization Job, and a NetworkPolicy allowing Substrate’s snapshot plugins and that Job to reach the S3 API.
If you already have remote object storage, skip this RustFS deployment. Follow
Substrate’s snapshot storage setup
and object-store plugin configuration.
Update the bucket, endpoint, region, and credentials below for an existing S3
service. For GCS, use the upstream service-account setup and retain the plugins’
gcs backend instead of applying the S3 settings.
For static S3 credentials, create ate-objectstore in ate-system with the
AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY keys expected below. If you use
workload identity instead, omit those two Secret references from the plugins.
The defaults used throughout this guide are:
Create the namespace and a shared Secret for RustFS, the bucket Job, and Substrate’s object-store plugins. Enter an access key and secret key of your choice; these become the RustFS credentials:
Edit deps/substrate/manifests/ate-install/kind/rustfs.yaml in place before
applying it. Despite the directory name, this manifest uses standard Kubernetes
resources and can run on a cluster with a suitable StorageClass.
In the rustfs-data PersistentVolumeClaim, increase the upstream 1Gi request
to your chosen capacity:
If the cluster has no default StorageClass, add spec.storageClassName with
your class name. Check available classes with kubectl get storageclass.
Snapshots can include actor memory and writable filesystems. Retain the PVC
across RustFS restarts; deleting it can remove the snapshots stored there.
This single-server setup depends on that volume’s availability.
In the rustfs Deployment’s container env, replace the two literal credential
entries with Secret references. Preserve the other environment entries:
In the rustfs-bucket-init Job’s container env, replace its two literal
credential entries with the same Secret. Keep its endpoint, region, and
ate-snapshots bucket name:
Apply the edited manifest and wait for the bucket:
On a fresh installation, the apply and wait commands end with output like:
When you reapply the manifest, resources can report unchanged or configured
instead of created.
The Job logs can include startup retries before the bucket is available. A successful log excerpt looks like:
The success message can also be bucket ate-snapshots already exists.
For the 50 GiB allocation above, expect PVC output like:
The volume name, storage class, and age depend on your cluster. Confirm that
the rollout succeeded, the Job completed, and the PVC is Bound before
continuing. The completed Job confirms that the endpoint accepts the configured
credentials and the bucket exists. If a command fails or times out, inspect
the Job logs and the PVC and Pod events.
Configure the installation
Open the checked-in
substrate-install.yaml
and edit it for your cluster. Continue working from
examples/agents/harbor-substrate.
For an existing installation, update its managed configuration instead of replacing it with a new document.
ate-setup --config reads this installer configuration. Do not apply it with
kubectl. The file uses bundled PostgreSQL and the RustFS bucket created above:
To use an external database, follow Substrate’s PostgreSQL setup. Supply connection credentials through the installer’s supported environment settings.
For a shared installation, select administrator principals using the
authorization guide.
The development owner lets kubectl-ate use its default ate-client
ServiceAccount token. Anyone who can mint that token has global ownership.
If you replace the owner, authenticate subsequent CLI commands with the
configured administrator’s token using --token-file.
To have the egress gateway supply credentials, configure a provider using egress credential injection.
Before deploying, also check sandbox asset caching for your runtime’s binaries and node cache, and complete the certificate API/projection checks above. Verify projected files after deployment using authentication setup.
Point Substrate at RustFS
The installer does not expose S3 endpoint or credential fields in
SubstrateInstall. Before deploying, edit these files in the pinned checkout:
deps/substrate/manifests/ate-install/ate-api-server.yamldeps/substrate/manifests/ate-install/atelet.yaml
In each workload’s spec.template.spec.initContainers, find the container
named objectstore-plugin. Change its existing ATE_STORAGE_BACKEND entry
to s3 and add the following AWS entries to its env list. Preserve its
other environment entries, arguments, and mounts:
These settings belong on the plugin sidecars. The ate-api-server and atelet
main containers do not read S3 credentials. The API server’s plugin manages
stored snapshots; the node plugin uploads and downloads their contents.
For another S3 service, update storage.snapshotLocation, AWS_ENDPOINT_URL,
AWS_REGION, path-style addressing, and the credentials or workload identity
to match that service. For Amazon S3, omit AWS_ENDPOINT_URL when using the
standard regional endpoint. For GCS, follow the upstream
snapshot storage instructions
with a gs:// location and real transfer-service-account settings; keep
ATE_STORAGE_BACKEND=gcs and omit the AWS entries.
Configure image-pull authentication
Atelet pulls actor images directly. The supplied manifest uses GKE’s node
credential-provider paths. Before deploying, edit the DaemonSet in
deps/substrate/manifests/ate-install/atelet.yaml for your registry:
Public images
Private images
For images that allow anonymous pulls, remove:
- The
--image-credential-provider-configand--image-credential-provider-bin-dirarguments from theateletcontainer. - The
image-credential-provider-configandimage-credential-provider-binentries from the container’svolumeMounts. - The entries with those same names from the Pod’s
volumes.
Atelet will pull images without authentication and will no longer require the host credential-provider files.
Check optional router monitoring
The upstream manifests include a PodMonitoring resource for
Google Managed Service for Prometheus.
It configures scraping of the router’s Envoy metrics. This service can run on
GKE or other Kubernetes clusters, so check API availability before deployment:
If the output lists PodMonitoring with API version
monitoring.googleapis.com/v1, keep the monitoring resource.
If it does not, edit
deps/substrate/manifests/ate-install/base/kustomization.yaml and remove this
entry from resources:
Keep the referenced YAML file. The installer no longer applies that resource, so it does not require the Google monitoring API. Substrate still runs, and the Envoy metrics endpoint remains available for another scraper.
Monitor Substrate
uses a Prometheus Operator ServiceMonitor to scrape an OpenTelemetry
Collector. That setup collects Substrate’s native metrics and Collector
self-metrics without the Google API. It does not scrape Envoy’s separate
/stats/prometheus endpoint.
Deploy the control plane
Pass the current Kubernetes context explicitly so the installer uses your existing cluster credentials. Run the installer inside its source checkout: it locates deployment assets relative to that repository. The subshell returns you to your chosen working directory afterward:
The installer reads the plugin settings edited above before it waits for readiness. No post-deployment S3 patch is needed.
Verify the installation
Wait for the control plane and node runtime:
For a port-443 permission error, see Resolve low-port permission failures.
Check Pod readiness and placement:
Service Pods should be ready, with one atelet on each eligible sandbox node.
The bucket-initialization Job should be Completed.
Check certificate issuance:
Requests should reach Issued=True. Completed requests are eventually removed,
so an empty list alone does not indicate failure.
Check the published Substrate trust bundles:
Verify that admitted API-server and atelet Pods retain both certificate projections:
Both path columns must be populated. These checks verify issuance, published
trust bundles, admitted projection configuration, and readiness; they do not
inspect projected file contents. If a check fails, inspect the affected Pod’s
events with kubectl -n ate-system describe pod <pod-name>.
To configure and run a coding-agent workload, continue with Run SWE-bench Verified with Harbor, Pi, and Substrate.
Resolve low-port permission failures when needed
Fix a port-443 permission error
If an API-server or worker Pod reports bind: permission denied on port 443,
add this Pod-level setting to the affected Deployment:
For an existing API-server deployment, apply the same setting directly:
Keep the API-server manifest change so subsequent installer runs retain it.
The rc1 WorkerPool API does not expose Pod sysctls in workerpool.yaml.
Patch the generated Deployment, which has the same name as its WorkerPool.
For the Harbor tutorial:
Reapply the worker patch if the controller recreates the Deployment or overwrites the setting. The sysctl affects all containers in the affected Pod, not the host.