Open source

Building Intent-Aware Configuration for LLM Serving

Marcus Acosta

I extended mistral.rs so PagedAttention configuration represents an operator's deployment policy, not just a best-effort runtime toggle.

The contribution carries PagedAttention intent from the CLI and Rust SDK through device validation, model loading, cache initialization, and scheduler selection. Automatic configuration can still fall back safely when unsupported. Explicit configuration is enforced: the engine either realizes PagedAttention or returns a specific startup error.

The design problem

PagedAttention manages an LLM's KV cache, the short-term memory used during token generation. It affects how cache memory is allocated, the scheduler used to serve requests, and the number of concurrent sequences a deployment can support.

mistral.rs supports three PagedAttention modes:

auto: Enable PagedAttention when the runtime supports it.
on: Require PagedAttention for this deployment.
off: Do not use PagedAttention.

At first, this looks like a simple on-or-off setting. It is not.

These modes express three different policies:

auto: Try it if possible.
on: It must work.
off: Do not attempt it.

A Boolean cannot fully represent those policies once configuration moves beyond the CLI boundary.

For example, on CUDA, both auto and on can initially become “enable PagedAttention.” If later code discovers that the model, device map, or pipeline cannot support it, a plain enabled = true value does not say whether fallback is acceptable.

That was the architectural gap I addressed.

Preserving intent

The CLI already represented the three modes:

auto → None
on   → Some(true)
off  → Some(false)

But later parts of the system mostly saw only whether a PagedAttention configuration existed. Once auto and on both created a configuration, loaders and scheduler code could not distinguish an optional optimization from a required serving policy.

I extended the existing configuration object with one field:

PagedAttentionConfig {
    required: bool,
}

The field defaults to false.

auto that enables PagedAttention
→ Some(PagedAttentionConfig { required: false })

on
→ Some(PagedAttentionConfig { required: true })

off
→ None

This is intentionally a small change. It does not introduce a new configuration hierarchy or require every loader to adopt a new interface. It preserves the missing policy signal in the object that already travels through the inference stack.

An NFL analogy

Think of PagedAttention as a goal-line package.

auto: Use the goal-line package if the needed players are available.
on: The goal-line package is required. Do not run the play without it.
off: Use the standard offense.

If the line between auto and on disappears after the coach makes the call, the staff cannot know whether it is allowed to substitute the normal offense.

The system needs the play call to retain its meaning until the snap.

That is what required: bool does for PagedAttention. It keeps the operator's original policy attached to the runtime configuration until the server is ready to accept requests.

Centralizing fallback decisions

Before this work, different model-loading paths could disable PagedAttention independently.

Some paths issued a warning. Others silently removed the PagedAttention configuration. These paths included unsupported models, mixed CPU and accelerator device maps, adapter configurations, GGML models, speech pipelines, embeddings, diffusion models, and other specialized loaders.

I introduced one shared policy boundary:

disable_paged_attention(
    &mut Option<PagedAttentionConfig>,
    reason,
) -> Result<()>

The loader supplies the compatibility reason. The shared helper applies the operator's policy.

That separates responsibilities cleanly:

Loaders know whether a model or device arrangement is compatible.
Configuration knows whether fallback is allowed.
The shared helper combines the two.

A loader no longer decides independently whether a PagedAttention failure should be tolerated.

Example: mixed CPU and GPU placement

A deployment may place part of a model on the GPU and part on the CPU to fit limited VRAM. In this configuration, the relevant PagedAttention path cannot be used because it does not provide a CPU KV-cache path.

With auto, the correct behavior is:

PagedAttention disabled: device mapping includes CPU and PagedAttention
has no CPU KV cache.

The server can continue using eager KV caching because the operator allowed a fallback.

With on, the correct behavior is different:

PagedAttention was explicitly requested (--paged-attn on),
but device mapping includes CPU and PagedAttention has no CPU KV cache.

The server does not start.

The difference is not whether the model can run. The difference is whether the runtime can meet the deployment policy the operator requested.

Enforcing policy at the device boundary

The first place the system can determine whether PagedAttention is possible is the device and build check.

I updated this path to return a decision that includes:

enabled
required
reason

That produces predictable behavior:

Existing defaults remain unchanged:

CUDA auto: Attempts to enable PagedAttention.
Metal auto: Keeps PagedAttention disabled by default.
CPU auto: Keeps PagedAttention disabled.

The change does not add new CUDA kernels, change Metal defaults, or introduce NCCL PagedAttention support. It makes the existing configuration behavior explicit and enforceable.

Verifying realized state

A user request is not the same as a realized system state.

The user may request PagedAttention, but model loading can still fail to create the required cache configuration. The scheduler must therefore validate the final pipeline metadata before serving requests.

I added a shared scheduler decision function that compares:

Requested configuration:
Did the operator request PagedAttention?

Realized configuration:
Did the loaded pipeline create a PagedAttention cache?

The resulting policy is:

This is the final backstop.

Even if a future loader accidentally loses a PagedAttention cache configuration, the scheduler will not quietly accept a required request and start with DefaultScheduler.

Why the scheduler matters

PagedAttention is connected to how the engine allocates KV-cache memory and schedules concurrent requests. It is commonly used with block-based cache allocation, prefix caching, and continuous batching in high-throughput LLM serving systems. mistral.rs README

If the engine silently selects a non-paged scheduler after an explicit PagedAttention request, the deployment may still start and generate output. But its memory profile, concurrency behavior, and benchmark results no longer represent the intended serving configuration.

The scheduler guard makes that mismatch visible before the server begins accepting traffic.

Extending the Rust SDK contract

The Rust SDK had the same policy issue.

Its with_paged_attn method could silently ignore an explicit request when the current build did not support PagedAttention. I changed it to preserve the request as required configuration.

The result is an intentional behavior change:

Before:
Explicit SDK request on an unsupported environment
→ Request may be ignored.

After:
Explicit SDK request on an unsupported environment
→ Loading returns an actionable error.

This is stricter, but it is more honest. An explicit API request should not disappear without the caller's knowledge.

Testing the policy without GPUs

PagedAttention is tied to accelerator-backed inference, but the core behavior in this contribution is configuration policy and scheduler selection.

I extracted those decisions into testable helpers and added CPU-only regression coverage for:

• CLI mapping of auto, on, and off
• Default and required configuration construction
• Automatic fallback versus required failure in disable_paged_attention
• CPU behavior for all three modes
• Unsupported-build behavior
• Scheduler rejection when a required request has no realized cache
• Automatic fallback to DefaultScheduler
• Successful selection of PagedAttentionMeta when cache metadata exists

This made it possible to validate the critical contract without downloading models or requiring CUDA or Metal hardware.

Scope

This contribution improves configuration semantics and serving reliability. It does not attempt to solve every PagedAttention limitation.

It does not:

• Add NCCL PagedAttention support.
• Change CUDA or Metal defaults.
• Update the Python paged_attn and no_paged_attn API.
• Add Windows CUDA PagedAttention support.
• Modify external KV-cache connectors.
• Add live GPU integration coverage.
• Remove the engine-level no_kv_cache path that can still remap a PagedAttention scheduler to DefaultScheduler.

Those are separate follow-up areas. Keeping them out of this change made the design reviewable and the tests focused on one contract.

What I learned

Configuration values should preserve intent, not just state.

A setting can look like a Boolean while carrying a policy:

Try this when possible.
Require this to be true.
Never do this.

If that policy is reduced too early to “enabled” or “disabled,” later components cannot make the correct decision.

This contribution adds a small field, a shared disable helper, and a scheduler-level guard. Together, they turn PagedAttention into an intent-aware serving policy rather than a best-effort toggle.

The final operator contract is straightforward:

--paged-attn on means PagedAttention is active, or mistral.rs does not start.