SDK 0.3.0 regression: per-turn mutating state in <system_reminder> block defeats Anthropic prompt cache

[pmella]

Summary

The Copilot CLI now auto-injects mutating session state (SQL tables list, todo status, added/removed tools, previous/new model) into a <system_reminder> block inside every user message. Because this content is part of the cached prefix and changes turn-by-turn within a session, it defeats Anthropic prompt-cache prefix matching for any consumer relying on cache_control: ephemeral for cross-turn cache reuse.

This is a regression in 0.3.0 vs 0.2.2 and has not been fixed through 1.0.0-beta.4 / CLI 1.0.47.

Observed in production

We measured a sustained 9 percentage-point drop in Anthropic prompt-cache hit rate for Claude Opus 4.5 traffic in our service immediately after upgrading from @github/copilot-sdk 0.2.2 (+ @github/copilot 1.0.34) to 0.3.0 (+ 1.0.40):

• Pre-upgrade (months stable): ~89% cache hit rate
• Post-upgrade (sustained through today): ~80% cache hit rate

The drop appeared within the typical container rollout window after merging the SDK bump. Cache rate has not recovered.

Behavior diff between 0.2.2 and 0.3.0 (reproduced locally)

Same input, same Claude Opus 4.5 model, same conversation prompt sequence:

SDK 0.2.2 user-message transformedContent (turns 1 and 2 identical, modulo timestamp):

<current_datetime>...</current_datetime>

Hello, can you help me with my spreadsheet?

<reminder>
<sql_tables>No tables currently exist. Default tables (todos, todo_deps) will be created automatically when you first use the SQL tool.</sql_tables>
</reminder>

SDK 0.3.0 user-message transformedContent:

Turn 1:

<current_datetime>...</current_datetime>

Hello, can you help me with my spreadsheet?

<system_reminder>
<sql_tables>No tables currently exist. Default tables (todos, todo_deps) will be created automatically when using the SQL tool for the first time.</sql_tables>
</system_reminder>

Turn 2 (same session):

<current_datetime>...</current_datetime>

Now do the same for B1 through B10.

<system_reminder>
<sql_tables>Available tables: todos, todo_deps, inbox_entries</sql_tables>
</system_reminder>

Two changes:

1. Tag rename: <reminder> -> <system_reminder> (one-time invalidation on upgrade)
2. Auto-mutating state: the SDK now tracks session state internally and injects it per turn. In 0.2.2 the <reminder> block was static across turns; in 0.3.0 the <system_reminder> block mutates as the SDK observes tool usage, model changes, todo updates, etc.

The second change is the ongoing cache-rate drag.

Why this matters

Anthropic prompt caching (cache_control: ephemeral) requires byte-exact prefix matching. When the cached prefix includes content that changes turn-by-turn within a session:

• Cache writes still happen every turn (paying the cache-write surcharge of 1.25 input rate)
• Cache reads find a shorter matching prefix because the mutating block ends the byte-match
• Net effect: paying for cache writes that never get fully read back

For high-volume API consumers, the 9-pp drop translates to a measurable input-token cost increase and lost end-user latency benefits.

Where in the code

yUr function in the CLI bundle, around node_modules/@github/copilot/sdk/index.js line 3942 in CLI 1.0.40:

yUr = ({customAgentPrompt, problemStatement, capabilities, hasImages,
        ..., sqlTables, todoStatus, addedTools, removedTools,
        previousModel, newModel, ...}) => {
    let E = sqlTables !== void 0
        ? `<sql_tables>Available tables: ${tablesList}</sql_tables>`
        : `<sql_tables>No tables currently exist...</sql_tables>`;
    // ...similar for todoStatus, addedTools, removedTools, model changes...
    return template.with({
        ...,
        current_datetime: q0t(),
        additional_instructions: assembled,
    }).asXML().trim();
};

These state fields are passed in by the CLI's internal session state machine and are not exposed to SDK consumers.

Proposed fixes (ranked)

1. Session config to opt out of auto state injection (preferred)

Expose a session option like:

client.createSession({
    autoInjectSessionState: false,  // disables <system_reminder> auto-population
    // ...
});

When false, the SDK omits the <system_reminder> block from user messages, or includes only static defaults. Consumers who want the state-tracking behavior keep the default; consumers optimizing for prompt caching opt out.

2. Move mutating state out of the cached prefix

Restructure the user message so mutating state lives in a separate content block placed after any consumer-set cache_control marker. Multi-block user messages with structured content arrays are supported by Anthropic; the SDK could emit them.

3. Document the cache impact

At minimum, document that <system_reminder> content is per-turn dynamic, so consumers can plan around it (e.g., avoid setting cache_control on the user-message block until this is opt-out-able).

Evidence and reproduction

We have a self-contained reproduction harness:

• Two minimal Node.js test apps, one per SDK version
• Same input prompt sent through each
• Captures transformedContent from user.message session events
• Side-by-side byte-level diff

Stage 1 (transformation diff) confirms the wrapper change. Stage 2 (5 independent sessions of the same prompt per version) confirms cross-session first-turn caching is consistent in both versions; the regression is specifically in intra-session multi-turn caching.

Happy to extract and share the harness publicly if helpful.

Related SDK behaviors worth flagging while we're here

While investigating, we also noted:

1. <current_datetime> is prepended at byte 0 of every user message, with sub-second precision. This single field guarantees that cache_control on the user-message block never reuses across requests. A flag to disable or move it would help.
2. Single-block user message means cache_control is all-or-nothing. A documented multi-block user-message API would let consumers mark stable portions (user profile, session metadata) with cache_control separately from per-request content.

These are separate from the regression above and could be addressed in follow-up issues if you prefer.

Versions

• Repro'd in: @github/copilot-sdk 0.3.0 + @github/copilot 1.0.40
• Last verified unfixed in: SDK 1.0.0-beta.4 + CLI 1.0.47

[github-actions]

Investigation findings

Thanks for the detailed report and the production telemetry data — this is well-documented.

What I investigated

• nodejs/src/types.ts — all SessionConfig / InfiniteSessionConfig public options
• nodejs/src/client.ts and nodejs/src/session.ts — session creation and event handling
• nodejs/src/generated/rpc.ts and session-events.ts — wire protocol and event shapes (including user.message / transformedContent)
• test/harness/replayingCapiProxy.ts — the replay proxy's normalizeUserMessage function
• docs/ — all feature docs and README for any mention of <system_reminder>, per-turn state injection, or prompt caching guidance

What I found

The <system_reminder>/<sql_tables> injection is intentional CLI behavior. The SDK wrapper (nodejs/src/) does not perform this injection. It originates inside the @github/copilot CLI bundle (the yUr function the reporter identified), where the CLI's internal session state machine assembles per-turn context (SQL tables, todo status, added/removed tools, model changes) and appends it to every user message.

This is confirmed by the SDK's own replay proxy, which explicitly strips <system_reminder> blocks before snapshot comparison (replayingCapiProxy.ts line 1038) — the SDK team was already aware of and designed around these blocks.

The behavior did change between CLI 1.0.34 (paired with SDK 0.2.2) and CLI 1.0.40 (paired with SDK 0.3.0). The injected content went from static per-session defaults to per-turn mutable state. That change is intentional — the CLI now actively tracks session state to give the model richer, up-to-date context.

The current SessionConfig API has no opt-out. There is no autoInjectSessionState or equivalent field in SessionConfig or InfiniteSessionConfig. The injection behavior is entirely internal to the CLI and is not configurable through the SDK today.

No documentation exists for this behavior or its prompt-caching implications. The docs don't mention that user messages include per-turn mutable state, or that cache_control: ephemeral strategies will be affected.

Classification: Enhancement

The per-turn state injection is working as designed — it's a deliberate feature that helps the model reason about current session context. This is not a bug.

However, you're raising a legitimate and measurable concern: the side effect of this design is that byte-exact prefix matching for Anthropic prompt caching breaks across turns within a session, at real cost. An opt-out is a reasonable new capability to add, which is why I've labeled this as enhancement rather than bug.

Suggested paths forward

The three proposals in the issue are all reasonable. For prioritization:

1. autoInjectSessionState: false in SessionConfig — cleanest for cache-optimized consumers; requires CLI-level support to honor the flag. This is the right long-term fix.
2. Move mutable state out of the cached prefix — structurally harder but benefits all consumers, not just those who opt out.
3. Documentation — the minimum viable action; clarify in the SDK docs that <system_reminder> content is per-turn dynamic and advise consumers targeting Anthropic prompt cache to be aware of this interaction. This should happen regardless of which other fix is chosen.

The <current_datetime> sub-second precision note (related items #1) is a separate and also valid concern for the same reason — worth a follow-up issue if you'd like to track it separately.

Generated by Bug Handler for issue #1296 · ● 723.9K · ◷

[pmella]

Thanks for the thorough investigation and confirming our findings on the source location.

Pushing back gently on the enhancement classification:

We agree the per-turn state injection is working as designed. The concern is that the behavior change between CLI 1.0.34 and CLI 1.0.40 was a breaking change for cache-optimized consumers, with no opt-out and no documentation. From our side it looked like a regression:

• The prior behavior (static <reminder> block) was cache-friendly. Every turn within a session matched the previous turn's prefix bytes.
• The new behavior (per-turn-mutating <system_reminder>) breaks byte-exact prefix matching turn-by-turn.
• Production telemetry shows a sustained 9 percentage-point drop in our cache hit rate the day the CLI bump rolled out. That's the definition of a regression: pre-existing functionality (cache reuse) was reduced without notice or migration path.

We understand "regression" vs "enhancement" affects prioritization. We'd be happy to keep the enhancement label if it helps internal triage, but want to flag that for consumers on Anthropic models the cost is real and ongoing.

Concretely, the most impactful thing for us right now would be option 3 (documentation) within days, paired with a commitment on option 1 (opt-out flag) on a reasonable timeline. We can plan around documented behavior; we can't around silent behavior change.

We'll file the follow-up issue on <current_datetime> byte-0 placement separately as suggested.

Thanks again for the prompt response.

[pmella]

Update: the regression has a second symptom besides cache rate.

Calls per interaction (LLM call count per user query) also jumped on the same May 7 inflection date, same Arcadia-only signature.

Data: Claude Opus 4.5, MSIT audience (stable user cohort)

	Pre-cliff (May 1-6)	Post-cliff (May 7-14)	Change
Arcadia (SDK 0.3.0)	5.11 calls/interaction	6.55	+28%
Aurora (different harness, control)	5.94	5.74	flat
Arcadia p90	12	14	tail moved too

Aurora is the cleanest control: same scenario, same audience, same model, different SDK path. It doesn't budge across the cliff.

Combined cost impact per user query

Calls +28%, cache miss share ~+30%, combined: ~1.67x input-token spend per Arcadia query vs pre-May-7 baseline.

Possibly the same root cause

If the per-turn mutating <system_reminder> content (SQL tables list, todo, tools, model) also nudges the model toward more verify / re-check / list-state tool calls, both symptoms could trace to one SDK behavior change. Speculative without source-level confirmation, but worth checking.