Skip to content

feat(shadow-requests): send shadow events under $ld:ai-shadow:* keys - #314

Open
XieX wants to merge 1 commit into
mainfrom
xie/aic-3954-shadow-event-keys
Open

XieX wants to merge 1 commit into
mainfrom
xie/aic-3954-shadow-event-keys

Conversation

@XieX

@XieX XieX commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Implements AIC-3954 (security design review item DC-1, SEC-9915): shadow traffic moves off the shared $ld:ai:* keys, so no consumer of AI events counts it unless it reads the new keys on purpose. The prefix is $ld:ai-shadow:, not $ld:ai:shadow:, because systems match on $ld:ai:.

Key rule

On a shadow run:

The run would send Sent as
$ld:ai:<rest> $ld:ai-shadow:<rest>
a key already under $ld:ai-shadow: unchanged, never prefixed twice
any other key, such as a judge's custom metric key my-relevance $ld:ai-shadow:my-relevance

The rule applies to:

  • B's generation success or error, duration and token events
  • each of B's judges' own model-call events (generation, duration, tokens)
  • each judge's score event

It applies on the background path, run_shadow() (deferred) and agui.stream(shadow_config(...)), including the timeout and handler-failure paths.

The seam

  • The rule lives in experimental/shadow_requests.py as _shadow_event_key(key) -> str. It's the only place that knows the prefix.
  • Core gets a generic, optional event_key: Callable[[str], str] | None mapper. Core never names shadows. Every client.track(...) key is passed through the mapper, and the default sends each key as is:
    • execute_and_track (public; it already took span_attributes and feature_flag_event for shadows). It passes the mapper on to wrap_tool_handlers.
    • wrap_tool_handlers (public). This keeps the invariant "the mapper covers every event execute_and_track sends". A shadow has no tools, so it sends no tool calls today.
    • judges._run_judges (private). It applies the mapper to each judge call's events and to both score paths: evaluationMetricKey and the typesafe eventKey.
  • How the shadow path selects it: shadow_requests._run passes event_key=_shadow_event_key to execute_and_track for B, and to _run_judges for B's judges. The background and deferred paths both go through _run, so they can't diverge. A's calls never pass a mapper, so A's keys are untouched.

Track data

  • B's run events and judge score events: unchanged. They keep B's standard fields, A's runId and the five shadow fields.
  • A shadow judge's own model-call events: these now carry the judge config's own track data and run id, as on A, plus the five shadow fields (isShadow, primaryVariationKey, shadowRequestId, ownerKind, ownerId). They come from a new judge_extra_track_data argument on _run_judges. Before this change those events had no shadow fields at all.

Unchanged

  • A's events and A's judge events.
  • The experimental.shadowConfig / experimental.runShadow usage events. The caller's call reports them, not the shadow run, so they stay on $ld:ai:sdk:usage.

Tests

The new TestShadowEventKeys class in tests/test_shadow_requests.py:

  • Key rule: unit cases, including $ld:ai:judge:*, a custom key, an already-shadow key, and $ld:ai:shadow:x.
  • Background, with one judge, for success, failure and timeout: records every track call after invoke() returns.
    • No shadow key starts with $ld:ai:.
    • Each key maps by the rule to the key list config().invoke() sends for the same outcome on B. A timeout is compared with a handler failure.
    • A's keys equal config().invoke()'s on A, and none is under $ld:ai-shadow:.
  • run_shadow, for the same three outcomes: the same assertions. The only $ld:ai: event is the experimental.runShadow usage event.
  • Background and deferred send identical keys.
  • Judge score key: my-relevance, $ld:ai:judge:relevance and $ld:ai-shadow:judge:relevance, on both paths, against A's judge score in the same invoke().
  • A shadow judge's own model call: its generation, duration and token events use $ld:ai-shadow: keys and carry the five shadow fields plus the judge config's own fields and run id. A's judge call in the same invoke() uses $ld:ai: keys with no shadow fields.
  • agui.stream(shadow_config(...)): every shadow key is under $ld:ai-shadow:, and A's stream has none.
  • Usage events stay on $ld:ai:sdk:usage.
  • wrap_tool_handlers applies event_key to regular and native tool calls (tests/test_tracking.py).
  • Existing shadow tests now expect the new keys.

Mutation check: dropping the mapper on the judge path, or dropping the judge shadow fields, makes the new tests fail.

make test, make lint, make format-check, make typecheck and make api-report-check all pass.

Notes for review

  • Public API report diff: execute_and_track and wrap_tool_handlers each gain a trailing optional event_key=None. The change is additive and doesn't change behaviour for existing callers. It ships under the shadow-requests scope because shadows are its only user.
  • Package README: updated where it describes shadow events and shadow judge scores.

🤖 Generated with Claude Code


Note

Overview
Shadow traffic is isolated from normal AI Config metrics by remapping every shadow telemetry key to the $ld:ai-shadow: namespace instead of $ld:ai:. The mapping lives in experimental _shadow_event_key; served variation events and $ld:ai:sdk:usage are unchanged.

Core tracking gains an optional event_key hook on execute_and_track and wrap_tool_handlers, and judges accept event_key plus judge_extra_track_data so shadow runs rewrite generation, token, duration, tool-call, and judge score keys (including custom metric names) and attach the five shadow fields on judge model-call events.

README and the generated API report document the new behavior; TestShadowEventKeys and updated shadow/tracking tests lock down key mapping across background, deferred, streaming, and judge paths.

Reviewed by Cursor Bugbot for commit a9b7c45. Bugbot is set up for automated code reviews on this repo. Configure here.

A shadow run now sends every event under the $ld:ai-shadow: namespace,
so no consumer of $ld:ai:* events counts shadow traffic unless it reads
the new keys on purpose. $ld:ai:<rest> is sent as $ld:ai-shadow:<rest>,
any other key (such as a judge's custom metric key) is prefixed, and a
key already under $ld:ai-shadow: is sent unchanged.

This covers the shadow's generation, duration and token events, each of
its judges' own model-call events and each judge's score, on the
background path, run_shadow and agui.stream(shadow_config(...)). A
shadow judge's own model-call events now also carry the five shadow
fields. The served call's events and the usage events are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@XieX
XieX requested a review from a team as a code owner November 20, 2026 18:14
@XieX
XieX requested a review from jeffdupont November 20, 2026 20:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant