Repository navigation
Conversation
A shadow run now sends every event under the $ld:ai-shadow: namespace, so no consumer of $ld:ai:* events counts shadow traffic unless it reads the new keys on purpose. $ld:ai:<rest> is sent as $ld:ai-shadow:<rest>, any other key (such as a judge's custom metric key) is prefixed, and a key already under $ld:ai-shadow: is sent unchanged. This covers the shadow's generation, duration and token events, each of its judges' own model-call events and each judge's score, on the background path, run_shadow and agui.stream(shadow_config(...)). A shadow judge's own model-call events now also carry the five shadow fields. The served call's events and the usage events are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements AIC-3954 (security design review item DC-1, SEC-9915): shadow traffic moves off the shared
$ld:ai:*keys, so no consumer of AI events counts it unless it reads the new keys on purpose. The prefix is$ld:ai-shadow:, not$ld:ai:shadow:, because systems match on$ld:ai:.Key rule
On a shadow run:
$ld:ai:<rest>$ld:ai-shadow:<rest>$ld:ai-shadow:my-relevance$ld:ai-shadow:my-relevanceThe rule applies to:
It applies on the background path,
run_shadow()(deferred) andagui.stream(shadow_config(...)), including the timeout and handler-failure paths.The seam
experimental/shadow_requests.pyas_shadow_event_key(key) -> str. It's the only place that knows the prefix.event_key: Callable[[str], str] | Nonemapper. Core never names shadows. Everyclient.track(...)key is passed through the mapper, and the default sends each key as is:execute_and_track(public; it already tookspan_attributesandfeature_flag_eventfor shadows). It passes the mapper on towrap_tool_handlers.wrap_tool_handlers(public). This keeps the invariant "the mapper covers every eventexecute_and_tracksends". A shadow has no tools, so it sends no tool calls today.judges._run_judges(private). It applies the mapper to each judge call's events and to both score paths:evaluationMetricKeyand the typesafeeventKey.shadow_requests._runpassesevent_key=_shadow_event_keytoexecute_and_trackfor B, and to_run_judgesfor B's judges. The background and deferred paths both go through_run, so they can't diverge. A's calls never pass a mapper, so A's keys are untouched.Track data
runIdand the five shadow fields.isShadow,primaryVariationKey,shadowRequestId,ownerKind,ownerId). They come from a newjudge_extra_track_dataargument on_run_judges. Before this change those events had no shadow fields at all.Unchanged
experimental.shadowConfig/experimental.runShadowusage events. The caller's call reports them, not the shadow run, so they stay on$ld:ai:sdk:usage.Tests
The new
TestShadowEventKeysclass intests/test_shadow_requests.py:$ld:ai:judge:*, a custom key, an already-shadow key, and$ld:ai:shadow:x.trackcall afterinvoke()returns.$ld:ai:.config().invoke()sends for the same outcome on B. A timeout is compared with a handler failure.config().invoke()'s on A, and none is under$ld:ai-shadow:.run_shadow, for the same three outcomes: the same assertions. The only$ld:ai:event is theexperimental.runShadowusage event.my-relevance,$ld:ai:judge:relevanceand$ld:ai-shadow:judge:relevance, on both paths, against A's judge score in the sameinvoke().$ld:ai-shadow:keys and carry the five shadow fields plus the judge config's own fields and run id. A's judge call in the sameinvoke()uses$ld:ai:keys with no shadow fields.agui.stream(shadow_config(...)): every shadow key is under$ld:ai-shadow:, and A's stream has none.$ld:ai:sdk:usage.wrap_tool_handlersappliesevent_keyto regular and native tool calls (tests/test_tracking.py).Mutation check: dropping the mapper on the judge path, or dropping the judge shadow fields, makes the new tests fail.
make test,make lint,make format-check,make typecheckandmake api-report-checkall pass.Notes for review
execute_and_trackandwrap_tool_handlerseach gain a trailing optionalevent_key=None. The change is additive and doesn't change behaviour for existing callers. It ships under theshadow-requestsscope because shadows are its only user.🤖 Generated with Claude Code
Note
Overview
Shadow traffic is isolated from normal AI Config metrics by remapping every shadow telemetry key to the
$ld:ai-shadow:namespace instead of$ld:ai:. The mapping lives in experimental_shadow_event_key; served variation events and$ld:ai:sdk:usageare unchanged.Core tracking gains an optional
event_keyhook onexecute_and_trackandwrap_tool_handlers, and judges acceptevent_keyplusjudge_extra_track_dataso shadow runs rewrite generation, token, duration, tool-call, and judge score keys (including custom metric names) and attach the five shadow fields on judge model-call events.README and the generated API report document the new behavior;
TestShadowEventKeysand updated shadow/tracking tests lock down key mapping across background, deferred, streaming, and judge paths.Reviewed by Cursor Bugbot for commit a9b7c45. Bugbot is set up for automated code reviews on this repo. Configure here.