Adaptive Cache Governor (ACG)

Use the Adaptive Cache Governor (ACG) when repeated LLM requests contain stable prompt sections that can benefit from provider prompt caching.

ACG decomposes LLM requests into Prompt IR, scores block stability across observed runs, and plans provider-specific prompt-cache breakpoints. The acg section is optional. Omit it to keep cache planning disabled.

plugins.toml Example

Add the following Adaptive Cache Governor configuration to plugins.toml:

1
version = 1
2
3
[[components]]
4
kind = "adaptive"
5
enabled = true
6
7
[components.config]
8
version = 1
9
agent_id = "planner"
10
11
[components.config.state.backend]
12
kind = "in_memory"
13
14
[components.config.telemetry]
15
subscriber_name = "adaptive.telemetry"
16
learners = ["acg"]
17
18
[components.config.acg]
19
provider = "anthropic"
20
observation_window = 100
21
priority = 50
22
23
[components.config.acg.stability_thresholds]
24
stable_threshold = 0.95
25
semi_stable_threshold = 0.50
26
min_observations_for_full_confidence = 20

This configuration enables adaptive telemetry and configures ACG to plan cache breakpoints for Anthropic-style request surfaces after it has enough observed prompt samples.

Plugin Configuration

Use plugin configuration when the application should let NeMo Relay own the Adaptive Cache Governor (ACG) runtime lifecycle. The following examples configure and activate ACG through each supported language binding.

validate() checks only the supplied in-memory object. initialize() also layers discovered plugins.toml configuration. For effective file-backed validation, refer to Plugin Configuration Files and run the gateway with the same configuration path that production uses.

1
import asyncio
2
3
import nemo_relay
4
5
adaptive_config = nemo_relay.adaptive.AdaptiveConfig(
6
    agent_id="planner",
7
    state=nemo_relay.adaptive.StateConfig(
8
        backend=nemo_relay.adaptive.BackendSpec.in_memory(),
9
    ),
10
    telemetry=nemo_relay.adaptive.TelemetryConfig(learners=["acg"]),
11
    acg=nemo_relay.adaptive.AcgConfig(provider="anthropic"),
12
)
13
14
plugin_config = nemo_relay.plugin.PluginConfig(
15
    components=[nemo_relay.adaptive.ComponentSpec(adaptive_config)]
16
)
17
18
report = nemo_relay.plugin.validate(plugin_config)
19
if any(diagnostic["level"] == "error" for diagnostic in report["diagnostics"]):
20
    raise RuntimeError(report["diagnostics"])
21
22
async def main():
23
    await nemo_relay.plugin.initialize(plugin_config)
24
    try:
25
        # Run instrumented application work here.
26
        pass
27
    finally:
28
        nemo_relay.plugin.clear()
29
30
asyncio.run(main())

Manual API

Use the manual runtime API when an integration needs to own adaptive lifecycle directly instead of activating the top-level plugin component.

1
import asyncio
2
3
import nemo_relay
4
5
adaptive_config = nemo_relay.adaptive.AdaptiveConfig(
6
    agent_id="planner",
7
    state=nemo_relay.adaptive.StateConfig(
8
        backend=nemo_relay.adaptive.BackendSpec.in_memory(),
9
    ),
10
    telemetry=nemo_relay.adaptive.TelemetryConfig(learners=["acg"]),
11
    acg=nemo_relay.adaptive.AcgConfig(provider="anthropic"),
12
)
13
14
runtime = nemo_relay.adaptive.AdaptiveRuntime(adaptive_config.to_dict())
15
asyncio.run(runtime.register())
16
try:
17
    # Run instrumented application work here.
18
    runtime.wait_for_idle()
19
finally:
20
    asyncio.run(runtime.shutdown())

Fields

The following table describes Adaptive Cache Governor settings:

Use passthrough when you want ACG observations without provider-specific hint translation. Set provider to the backend API surface the agent actually calls when you are ready to apply cache planning.

Expected Output

When ACG is active, instrumented LLM calls still return the same application result. ACG records observations and, when enough stable prompt structure is available, emits adaptive diagnostics and cache-planning decisions through the adaptive runtime.

Provider-specific cache hints are useful only when the request surface supports them. Validate against representative LLM traffic before enabling ACG in production.

Common Configuration and Runtime Issues

• provider is not one of passthrough, anthropic, or openai.
• Stability thresholds are outside the supported numeric range.
• ACG is enabled before the application emits managed LLM events.
• The configured provider does not match the real model API surface.