Skip to content

fix(holdouts): treat a repeated holdoutIds entry as one holdout - #25

Closed
joalves wants to merge 4 commits into
mainfrom
fix/dedup-duplicate-holdout-ids
Closed

joalves wants to merge 4 commits into
mainfrom
fix/dedup-duplicate-holdout-ids

Conversation

@joalves

@joalves joalves commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

What

holdoutIds listing the same holdout more than once (adjacent or not) now resolves to one holdout, as in javascript-sdk, which dedups at install with insertUniqueSorted.

Draft, owner decision needed before merge: whether dedup is the cross-SDK contract is not yet confirmed (holdouts/switchbacks plan §7, open question 7). JS is the reference and dedups, so cross-sdk-tests records that behaviour (scenario 500, gated on a holdoutIdsDedup capability). This PR makes Java match it if the owner confirms.

Why

  • Before: Java kept the duplicate in the applicable-holdout set. It still queued one holdout exposure, because the cached holdout assignment is shared and guarded by its CAS. But the duplicate counted in holdoutSetMatches.
  • Effect: a refresh that only removed the duplicate looked like a coverage change, so the covered experiment was re-resolved and published a second exposure.

Change

  • resolveApplicableHoldouts skips a holdout that is already in the list. It is O(n²) over one experiment's holdoutIds, which is a handful of entries, and it is computed once per setData.
  • No public API change. Java 6 source still compiles; core-api-java6 is generated.
  • Tests in ContextHoldoutTest:
    • {11, 11}: one holdout exposure; a refresh to {11} re-exposes nothing; the publish carries the experiment and one holdout exposure.
    • {11, 12, 11, 12}: a refresh to {11, 12} re-exposes nothing, and the publish carries the experiment and one exposure per holdout. This kills mutants that only drop adjacent repeats or only repeats of the first id.
  • CHANGELOG: a new Unreleased → Fixed section, since 1.9.0 is already released.

Verification

  • ./gradlew test -PskipJdk25 on JDK 11: core-api 754 tests and core-api-java6 755 tests, 0 failures. Both new tests fail on main.
  • cross-sdk-tests: the java wrapper built from this branch reports holdoutIdsDedup: true and passes all 313 scenarios, including 500.
  • Spotless could not run locally because the sandbox cannot download the Eclipse formatter. CI runs it.

Linked: absmartly/cross-sdk-tests#23 (H1-H8; java lists holdoutIdsDedup as a reviewed exception until this merges).

🤖 Generated with Claude Code

Summary by CodeRabbit

  • Bug Fixes
    • Duplicate holdout IDs are now treated as a single holdout, preventing repeated evaluation or exposure.
    • Removing duplicate IDs without changing holdout coverage no longer triggers another resolution or exposure; existing assignments remain unchanged.

lux worker and others added 4 commits November 17, 2026 14:29
javascript-sdk dedups holdoutIds at install (insertUniqueSorted); java-sdk
kept the duplicate in the applicable-holdout set. One holdout exposure was
still queued (the cached holdout assignment's CAS), but the duplicate counted
in holdoutSetMatches, so a refresh that only dropped it re-resolved the
covered experiment and published a second exposure. Pinned by cross-sdk-tests
Holdout Conformance scenario 500.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nder Unreleased

A mutant that only drops a repeat of the last resolved holdout passed the
suite, since the only test repeated ids back to back. {11, 12, 11} now pins
that any repeat counts once. 1.9.0 is already released, so the fix is listed
under a new Unreleased section.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…larify the changelog

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A dedup that only compared against the first kept holdout passed the suite.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

Explain Change →

No actionable comments were generated in the recent review.

Walkthrough

The holdout resolver now adds each resolved holdout only once, while continuing to ignore missing or malformed references and sort results by ID. New tests check exposure counts for adjacent and non-adjacent duplicate IDs. They also check that removing duplicates during refresh preserves assignments and pending exposure counts. The changelog records the change for core-api and core-api-java6.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related PRs

  • absmartly/java-sdk#12: Introduces the holdout references and applicable-holdout tracking that this change updates.

Suggested reviewers: marcio-absmartly

Merge Risk: Minimal · up to eb9d4

Repeated holdout IDs are now treated as one holdout, which avoids a duplicate exposure after a refresh removes the repeat. The change is small and covered by tests. No merge-blocking risk was found.

Pre-merge checks: Passed 4

Check name Status Explanation
Title check Passed The title 'fix(holdouts): treat a repeated holdoutIds entry as one holdout' matches the main change. The raw summary and objectives show that resolveApplicableHoldouts now deduplicates holdouts, so re…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Full details: Title check

Explanation

The title 'fix(holdouts): treat a repeated holdoutIds entry as one holdout' matches the main change. The raw summary and objectives show that resolveApplicableHoldouts now deduplicates holdouts, so repeated holdoutIds entries count once, with tests and a changelog entry supporting it. The title is specific, concise, and follows a conventional commit style. It does not describe the refresh-related exposure fix in detail, but that is expected.

Finishing Touches

Generate unit tests (beta)

  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

A rabbit counts each holdout once,
Then checks the list from dusk till dawn.
The duplicate names fade away,
While exposures keep their count in play.
Refresh leaves assignments where they belong.

Recent review info

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Essentials
  • Run ID: cafcbffa-1817-4f6a-91c9-51ba12cc4e83
Commits

Reviewing files that changed from the base of the PR and between 07807b5 and eb9d467.

Files selected for processing: 3

  • CHANGELOG.md
  • core-api/src/main/java/com/absmartly/sdk/Context.java
  • core-api/src/test/java/com/absmartly/sdk/ContextHoldoutTest.java

Included review availability: This review used your included allowance. 0 included reviews remain after this review. Your included PR review attempts over the past 7 days set your current allowance at 1 review per hour.

@joalves

joalves commented Nov 17, 2026

Copy link
Copy Markdown
Contributor Author

Independent sdk-review on eb9d467: APPROVE, 3 P2 fixed across 4 reviewer passes (a4da49c, 1a2fee6, 1eab5ae, eb9d467). Remaining P3s (an unnumbered release heading, and a reverse-refresh test) are tracked below and will be addressed in the review loop.

Final reviewer report (eb9d467), with evidence log

APPROVE at eb9d467: the fix is a one-line install-time dedup that matches javascript-sdk. It leaves the public and private API byte-identical, kills every dedup mutant I tried, and turns the java-wrapper holdoutIdsDedup probe (scenario 500) from FAIL to PASS. What remains is two P3s, one owner decision (whether dedup is the cross-SDK contract) and gates I could not run offline (Spotless, Animal Sniffer, JaCoCo). Those gates must be green in CI on this exact head before it is pushed for human review.

Compared published com.absmartly.sdk:core-api / core-api-java6 1.9.0 (Maven Central POMs return 200; tag v1.9.0 = 853feb3) → merge base 07807b5 (origin/main, which adds the switchback merge #15 on top of the release) → head eb9d467 (4 commits: a4da49c, 1a2fee6, 1eab5ae, eb9d467). Parity against javascript-sdk 311cfca (src/context.ts _init insertUniqueSorted, src/algorithm.ts) and the cross-sdk-tests 86115a0 behavioural self-tests for holdoutIdsDedup, run through both SDKs. Only this PR's delta is a finding.

Must fix

None. No P1 or P2.

Tests needed (current implementation correct)

10 full-suite mutations (core-api + core-api-java6, JDK 11): 9 killed, 1 survived.

  • The survivor is M6 (drop the Collections.sort in resolveApplicableHoldouts). The PR did not add this code: the sort is unchanged from main and v1.9.0, and the PR only rewords its comment. Pre-existing test gap, track separately. Not a finding of this PR.
  • Every mutant aimed at the new dedup line was killed: none (M1), adjacent-only (M2), first-id-only (M3), first-element-only (M4) and dedup by the wrong key (M5). M2 and M3 are killed only by nonAdjacentDuplicateHoldoutIdIsOneHoldout with {11, 12, 11, 12}, so that test is load-bearing.

Compatibility / parity / forward-compat evidence

  • javap -public -s over every class: main vs head diff = 0 lines for both core-api (1824 lines) and core-api-java6 (1900 lines). javap -p -s Context: main vs head diff = 0 lines. No signature changed, so no consumer or NoSuchMethodError risk.
  • Released v1.9.0 test suite against the head library: 607 tests / 7 failed (core-api) and 608 / 7 (core-api-java6). Against the main library: identical counts and an identical failure set. All 7 are NoSuchMethodException: Context.getAssignment(String) from reflective helpers in the release's tests: a private method that main (switchback feat(switchback): serve R33 cluster-defined switchbacks #15) changed. Pre-existing on main, not this PR.
  • Head suite: core-api 754/754, core-api-java6 755/755 pass. Main suite: 752/752 and 753/753. The new 2 tests fail on main's production code (duplicateHoldoutIdIsOneHoldout at ContextHoldoutTest.java:648, and nonAdjacentDuplicateHoldoutIdIsOneHoldout), so they can fail on base.
  • Behaviour change is limited to experiments whose holdoutIds repeats an id. For a list with no repeat, applicable.contains(holdout) is always false, so the resolved array is the same as before. That is a static argument backed by the full suite passing unchanged; I ran no no-rule UID sweep (the diff touches no assignment or hashing code).
  • Parity: the cross-sdk-tests java-wrapper HoldoutIdsDedupSelfTest (adjacent [901,901], non-adjacent [902,903,902,903], then a refresh that drops the duplicates) was compiled outside Docker against each tree. Main: FAIL ("refresh dropping the adjacent duplicate … got exposures=[{id=801…}]", which is the bug). Head: PASS. The javascript-wrapper runHoldoutIdsDedup against local javascript-sdk 311cfca: PASS. Java head and JS now agree.
  • Reverse direction (a refresh that adds a duplicate, {11} → {11, 11}): a scratch test fails on main (pending 3, expected 2) and passes on head. Head is right; the direction is just not pinned (P3 below).
  • Java 6: the generated core-api-java6 Context.java contains the change (1 match). Class major version is 50. The change uses only List.contains.
  • Forward compat: the decoder and models are untouched (json/Experiment.java has no diff), so key injection was not re-run.

Performance

Static finding; no timing claimed. The dedup runs once per experiment at data install (setData) and is O(k²) in holdoutIds.length, using Experiment.equals. It is not on the read path: cached getTreatment/peek/variableValue and holdoutSetMatches are unchanged. k is a handful of holdouts. No new cache, so no counter test is needed.

P3 / notes

  • P3, version and section placement (CHANGELOG.md:3, build.gradle:15). The entry sits under an unnumbered ## Unreleased heading, and VERSION (and so the X-Agent) is still 1.9.0. 1.9.0 is published (tag v1.9.0, Maven Central 200). A build of this head would report 1.9.0 while behaving differently from the 1.9.0 artifact. The entry changes customer-visible exposure counts, so it arguably belongs under ### Changed with its "who moves" sentence, rather than ### Fixed. Please pick the release number (1.9.1 looks right for a bug fix) when this is released and bump VERSION with it. Owner call on Fixed vs Changed.
  • P3, cheap test hardening (ContextHoldoutTest.java:631). Only the "refresh drops the duplicate" direction is pinned. Please add the mirror case: install {11}, refresh to {11, 11}, and assert that pending stays 2. Measured: it fails on main (3) and passes on head.
  • Owner decision: whether dedup is the cross-SDK contract. The cross-sdk-tests README and WRAPPER_API_SPEC still mark holdoutIdsDedup as pending confirmation. The PR should stay a draft as planned. After merge, cross-sdk-tests must move holdoutIdsDedup from java's reviewed exception to required (orchestrator/expected-capabilities.json:286) and repin the lane. This is a release gate on the cross-sdk side, not on this PR.
  • Experiment.equals is used by contains. It is safe because holdoutsById yields one object per id, so equality reduces to identity here. An id comparison would be cheaper but is not needed.

Version and release notes

Head VERSION = "1.9.0", the same as the published release. The released sections are untouched by this diff (only lines are added above them). The ## 1.9.0 (unreleased) heading for an already-published version comes from main (see below). No migration notes are needed: there is no API change, assignments are unchanged, and the only behaviour change is that fewer duplicate exposures are sent.

Pre-existing on main (not this PR)

  • The CHANGELOG heading reads ## 1.9.0 (unreleased), but v1.9.0 is tagged and on Maven Central. Release gate for the next release.
  • 7 released-suite tests break against main's library through a private reflective helper (Context.getAssignment(String)). These are test-only reflection breaks, not public API.
  • Mutant M6 (holdout sort removed) survives the whole suite, so the sort/exposure-order contract is not pinned. Track separately.

Verification limits / provenance: Spotless (formatter download), Animal Sniffer (org.codehaus.mojo.signature:java18:1.0 unresolvable offline) and JaCoCo verification (org.jacoco.ant:0.8.12 unresolvable offline) were NOT RUN. They failed the same way on main, for resolution reasons only. There is no PR and no CI run on eb9d467 yet, so these gates are unverified until CI is green on this exact head. The added Java lines are at most 117 columns (lineSplit 120) and git diff --check is clean, but that is not Spotless. JDK 25 lane NOT RUN (-PskipJdk25). No Docker: the cross-sdk scenario 500 itself was not run through the orchestrator; I ran its wrapper self-tests directly against each tree instead. No publish dry-run. All counts come from Gradle JUnit XML on JDK 11 (arm64).


sdk-review evidence: java-sdk (branch fix/dedup-duplicate-holdout-ids, no PR yet)

  • Head: eb9d467a45d5617401d452fdbdad3042faaed293 (worktree /workspace/repos/java-sdk-dedup, clean before and after)
  • Mode: independent reviewer (read-only); scratch /tmp/sdkrev-jahd (worktrees head, main, rel, removed afterwards)
  • Previous pass / review: none for this branch; commits since base 07807b5: a4da49c, 1a2fee6, 1eab5ae, eb9d467
  • Finished (UTC): 2026-11-17T18:05Z

1. Trees

Tree Identifier How obtained Verified
Released core-api / core-api-java6 1.9.0; tag v1.9.0 = 853feb3 curl -w %{http_code} repo1.maven.org/.../1.9.0/*.pom → 200, 200; git rev-parse v1.9.0^{commit} jar-to-tag byte match NOT RUN (offline jar download not attempted); tag source used
Main / merge base 07807b5 git merge-base origin/main HEAD main = v1.9.0 + switchback #15 (git log v1.9.0..origin/main)
Head eb9d467 git rev-parse HEAD
Version source / X-Agent / CHANGELOG heading: build.gradle VERSION 1.9.0; new CHANGELOG heading "Unreleased" (unnumbered). Not bumped above the published 1.9.0 → P3.

2. Backwards compatibility

  • API diff: javap -public -s over all classes in build/classes/java/main, main vs head → 0 diff lines (core-api, core-api-java6). javap -p -s com.absmartly.sdk.Context → 0 diff lines.
  • Consumer probe: NOT RUN as a separate project because the signature diff is 0. The java-wrapper probe classes (an external package) compiled and ran against both main and head.
  • Released tests vs head lib: v1.9.0 tests + head src/main → core-api 607/7 failed, java6 608/7. Same with main src/main → 607/7, 608/7; diff of the failure sets → identical. Cause: NoSuchMethodException Context.getAssignment(String), main-era.
  • No-rule behaviour sweep: NOT RUN (the diff touches no hashing or assignment; for a list with no repeat the branch is a no-op by construction; the full suite is unchanged).
  • Error/null/logging contract: unchanged (no throw sites or logger calls in the diff).
  • Public fields/hooks whose effect changed: none (Experiment.holdoutIds is still read the same way).
  • Entry points: the change is in resolveApplicableHoldouts, called only from setData (Context.java:2341), which serves create-with-data, fetch and refresh. The tests cover with-data + refresh; the wrapper probe covers createContextWith + refresh.

3. Parity

  • JS reference javascript-sdk 311cfca: insertUniqueSorted (algorithm.ts:1-18) returns early on an equal id, so it dedups by id at install (context.ts:1504-1508). Java now dedups and then sorts by id. Equivalent.
  • Run through both: cross-sdk-tests 86115a0 holdoutIdsDedup self-test. JS (node holdoutSelfTest.runHoldoutIdsDedup, NODE_PATH → local javascript-sdk): PASS. Java main: FAIL ("refresh dropping the adjacent duplicate … exposures=[{id=801…}]"). Java head: PASS. holdouts battery PASS on both trees.
  • Numeric edge table: N/A (no numeric parsing in the diff).
  • cross-sdk-tests: scenario 500 gated on holdoutIdsDedup. Java is listed as a reviewed exception (expected-capabilities.json:286); the orchestrator was not run (no Docker), so no eligible/skip count is claimed.

4. Tests and mutation

  • Full suite head (./gradlew test -PskipJdk25 --offline, JDK 11): core-api 754 tests / 0 failed, core-api-java6 755 / 0; about 28 s wall.
  • Main: 752 / 0, 753 / 0. Head tests on main's production code: 81 run, 2 failed (both new tests).
    | # | Mutant (Context.java) | Killed by | Result |
    |---|---|---|---|
    | M1 | dedup removed (if (holdout != null)) | duplicateHoldoutIdIsOneHoldout, nonAdjacent… | killed |
    | M2 | dedup only against the last element (adjacent only) | nonAdjacentDuplicateHoldoutIdIsOneHoldout | killed |
    | M3 | dedup only against the first element | nonAdjacentDuplicateHoldoutIdIsOneHoldout | killed |
    | M4 | dedup only repeats of index 0 | both new tests | killed |
    | M5 | dedup keyed on iteration instead of identity | nonAdjacent… + 8 union tests | killed |
    | M6 | Collections.sort removed (pre-existing code) | none | SURVIVED (pre-existing, not PR) |
    | M7 | holdoutSetMatches ignores per-entry ids/iterations | refreshWithIterationBumpReassignsToExactlyOneNewArm | killed |
    | M8 | holdoutSetMatches ignores iteration | refreshWithIterationBump… | killed |
    | M9 | last distinct holdout dropped after dedup | nonAdjacent… + 8 union tests | killed |
    | M10 | single-id holdoutIds ignored | 60+ holdout tests | killed |
    Totals: 10 run / 9 killed / 1 survived (M6, pre-existing; not a PR finding). Script: /tmp/sdkrev-jahd/runmut.sh; each run used ./gradlew test --continue over both modules, then git checkout of src/main.
  • Extra probe: a mirror test (refresh {11} → {11,11}) fails on main (expected 2, was 3) and passes on head. It is not in the PR → P3.
  • Fix-commit audit:
    | Commit | New state | Where else set/cleared/read | Entry points skipping it | Mutant |
    |---|---|---|---|---|
    | a4da49c | dedup in resolveApplicableHoldouts (no new field) | sole producer of ContextExperiment.holdouts; switchbacks reset it to null afterwards | none: all data install goes through setData | M1-M5 |
    | 1a2fee6 | non-adjacent test, CHANGELOG moved to Unreleased | n/a | n/a | M2, M3 |
    | 1eab5ae | publish assertion for the non-adjacent case | n/a | n/a | M9 |
    | eb9d467 | test data {11,12,11,12} | n/a | n/a | M3 |
  • Test hygiene: Mockito.timeout(5000) is bounded; no sleeps; the asserts are concrete values and published bodies.

5. Performance

Path main head probe counter test
cached reads unchanged code unchanged code static n/a
setData install O(k) add O(k²) contains (k = holdoutIds length, small) static, no timing n/a (no cache added)

6. Forward compatibility

  • NOT RUN: no decoder or model change (git diff origin/main --stat touches only Context.java, the test and CHANGELOG). Existing unknown-field tests pass in the full suite.

7. Build / CI

  • ./gradlew check -x spotlessCheck -x spotlessJava --offline on main and head: animalsnifferMain FAILED (cannot resolve org.codehaus.mojo.signature:java18:1.0); jacocoTestCoverageVerification FAILED (cannot resolve org.jacoco.ant:0.8.12). The failure is identical on main, so it is an offline resolution problem, not a code problem. NOT RUN in effect.
  • Spotless: NOT RUN (formatter download). Proxy only: max added Java line 117 columns (tab=4) vs lineSplit 120; git diff --check clean.
  • Floors: java6 bytecode major 50 (javap -v); generated java6 source contains the change.
  • JDK 25 lane: NOT RUN (-PskipJdk25).
  • CI on this exact head: none (no PR; gh pr list --head fix/dedup-duplicate-holdout-ids → []). Must be green before human review.
  • Pins: n/a for java-sdk. cross-sdk-tests must later repin and move holdoutIdsDedup to required for java.

8. Docs / CHANGELOG

  • Section/version: "## Unreleased / ### Fixed" above "## 1.9.0 (unreleased)"; VERSION is still 1.9.0, but 1.9.0 is published → P3.
  • Migration notes: none needed (0 API diff vs main; the release API is not affected by this PR).
  • Added vs Changed: the entry is under Fixed and changes exposure counts → P3 (owner call).
  • Stale strings: the code comment (Context.java:2289-2301) matches the behaviour; no version strings touched.

Findings

Sev Title Evidence line above Fixed in
P3 Unnumbered "Unreleased" section, VERSION still 1.9.0 (published); consider Changed §1, §8 open
P3 Reverse-direction refresh ({11} → {11,11}) not pinned §4 extra probe open
owner Dedup as cross-SDK contract (holdoutIdsDedup pending) §3 n/a

Verification limits

  • Spotless, Animal Sniffer, JaCoCo, JDK 25 lane, publish dry-run, orchestrator scenario 500 (Docker), no-rule UID sweep, forward-compat injection, released-jar byte check: NOT RUN for the reasons above.

@jervasion-absmartly

jervasion-absmartly commented Nov 17, 2026 •

Copy link
Copy Markdown

Review submitted

The automated review has been submitted. See the review for the verdict and any findings.

Head: eb9d467a45d5 | Updated: 2026-11-17 18:50 UTC

@jervasion-absmartly

Copy link
Copy Markdown

Question Q1 — Confirm duplicate-holdout contract before merge

Has the responsible holdout/cross-SDK owner confirmed that repeated holdoutIds must count as one holdout, so this Java behavior change is intended to merge? The PR description explicitly leaves this decision open. Please confirm the intended contract (or link the owner's decision).

Context: PR description explicitly says “owner decision needed before merge” on dedup as the cross-SDK contract; joalves's evidence comment repeats that it is pending. Context.java:2302 now deduplicates applicable holdouts, and tests pin unchanged exposures when refresh removes duplicate IDs. Checked all four commits, PR conversation and existing reviews; no confirmation is recorded.

Why I'm asking: The implementation and tests can be verified independently, but approval would choose an explicitly unresolved product contract. The owner's answer determines merge readiness, not merely test adequacy.

Suggested answers (free text is always welcome):

  • Confirmed: duplicate holdout IDs must count once; this change is intended to merge.
  • Not confirmed yet; keep the PR draft pending the owner's decision.
  • Duplicates should retain different semantics; this implementation should not merge.

@joalves, please include Q1 or this comment's link in your reply. Free-text answers are always welcome. I'll keep checking the rest of the change in the meantime.

@joalves

joalves commented Nov 17, 2026

Copy link
Copy Markdown
Contributor Author

Q1 (#25 (comment)): Not confirmed yet; keep the PR draft pending the owner's decision.

No owner decision is recorded. This is open question 7 in the holdouts plan. The PR stays a draft until the owner confirms. Please review the implementation and tests on their merits: whether this fix makes Java match javascript-sdk, which already dedups at install, and does nothing else. The merge itself waits for that decision.

@jervasion-absmartly jervasion-absmartly left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❓ Needs a human decision

No blocking implementation defect found. The change deduplicates applicable holdouts at installation, consistent with javascript-sdk, and prevents duplicate-only refreshes from generating an extra experiment exposure. Merge readiness still depends on the explicitly open cross-SDK contract decision.

Next step: Keep this PR draft until the holdout/cross-SDK owner decides whether duplicate holdout IDs must count once.

Suggestions (optional)

  • [nit] Three short comments repeat the adjacent code or test setup; optional cleanup is listed in the review details.

Human review required

Question: Should repeated holdoutIds count as one holdout across SDKs?

Why it matters: This head implements that behavior; an approval would endorse a contract that the PR explicitly leaves undecided. This is not a code-defect finding.

Evidence checked: Installation and refresh/cache consumers, both added regression tests, javascript-sdk's insertUniqueSorted installation path at 311cfca, the PR description, and joalves's Q1 answer, which confirms that no owner decision is recorded.

Needed from: The holdout/cross-SDK owner, now asked directly by the author.

What I checked

  • ✅ Both artifacts pass their full suites on JDK 11 and JDK 25: 3,018 test executions, zero failures or skips.
  • ✅ Adjacent and non-adjacent duplicates preserve pending exposure counts across refresh and publish the expected experiment/holdout exposures. Both new tests fail in both artifacts when only the deduplication condition is reverted.
  • ✅ Independent error-handling review found no introduced failure-path defect; missing/malformed references and refresh failure propagation remain intact.
  • ✅ Exact-head CI passed its full build, including formatting, compatibility and coverage gates.
Review details

Scope and intent

  • Head: eb9d467a45d5617401d452fdbdad3042faaed293. Base and merge-base: 07807b516df5ad0d7d92e5b682041982802cd871. Initial review, round 1; resumed after Q1 was answered. Head/base remain unchanged.
  • Reviewed all four PR commits and the complete three-file diff: runtime resolver, ContextHoldoutTest, and changelog. No API signatures, decoder, hashing or assignment algorithms change. Java 6 sources/tests are generated from the same source path.
  • The author reports JDK 11 tests and 313 cross-SDK scenarios passing. This review independently ran the repository suites and a production-hunk revert; it did not rerun the external cross-SDK orchestrator. JavaScript parity was additionally checked against its install-time ID comparator and unique-insertion implementation.
  • Verified: the ID-keyed map supplies the same object for a repeated ID. Experiment.equals returns immediately for identity and rejects distinct IDs before comparing nested fields. Thus contains deduplicates IDs without collapsing distinct holdouts. Unique-reference inputs produce the same resolved array as before. Sorting and cache comparison remain unchanged.
  • Work shape: per experiment with k holdout references and u resolved unique holdouts, installation adds up to k list scans of at most u entries (worst-case quadratic); cached reads are unchanged. The PR describes these lists as a handful of entries. No measured latency claim is made.

Validation

  • Verified Java/Javac 11.0.33; Gradle ran with JAVA_HOME="${JAVA_HOME_11:?Select a verified JDK 11 first}".
  • ./gradlew build --no-daemon: local execution stopped during Spotless configuration because download.eclipse.org could not resolve. This was an environment failure, not a formatting result. Full-build coverage comes from exact-head CI below.
  • ./gradlew :core-api:test :core-api-java6:test :core-api:testJdk25 :core-api-java6:testJdk25 --no-daemon: passed. XML totals: core-api 754 on each runtime; core-api-java6 755 on each runtime; zero failures, errors or skips.
  • Scratch revert changed only if ((holdout != null) && !applicable.contains(holdout)) back to if (holdout != null). Command: ./gradlew :core-api:test :core-api-java6:test --tests '*ContextHoldoutTest.duplicateHoldoutIdIsOneHoldout' --tests '*ContextHoldoutTest.nonAdjacentDuplicateHoldoutIdIsOneHoldout' --continue --no-daemon. The core-api task collected 754 tests and failed only the two new tests; java6 collected two and failed both. Failures were attributable to post-refresh pending counts: expected 2/3, actual 3/4. Tests were unmodified.
  • git diff --check: passed.
  • CI: Java-SDK run 44144165280, exact head, completed successfully; build job 134254623494, including successful Build Java SDK step. Workflow runs ./gradlew -Porg.gradle.java.installations.fromEnv=JAVA_HOME_25_X64 clean build. Commit statuses contain no individual statuses; their empty aggregate pending is not a running check.

Passes

Passes (non-blocking): run: core runtime/API/data (resolver and refresh/exposure consumers); error handling/messages (mandatory independent pr-review-errors pass completed for installation, malformed references and refresh failure propagation; no material coverage gap); tests/CI/packaging (two new tests, observable exposure behavior and generated Java6 artifact); cross-cutting simplicity (local audit of the one-condition runtime change); comment hygiene; diff composition.

Not run: deployment/operations/observability (no manifests, release procedures or telemetry emissions changed); shared-component/frontend (no frontend source changed); behaviour-bearing data/policy (no runtime configuration, schemas, fixtures or policy changed).

Comment hygiene

Comment hygiene (non-blocking): 10 comment / 52 code lines, ratio ~1:5.2, 3 blocks flagged.

threshold T=5; restatements counted: yes (10 × C ≥ E).

  • core-api/src/test/java/com/absmartly/sdk/ContextHoldoutTest.java:638: own exposure + one holdout exposure — restates the assertion/expected payload.
  • Same file, line 660: A repeat need not be adjacent or of the first id: {11, 12, 11, 12} is holdouts 11 and 12. — restates the named test and fixture.
  • Same file, line 670: own exposure + one per holdout — restates the assertion/expected payload.

Three qualifying added lines are below T. Preserved the comments explaining cross-SDK semantics and the non-obvious relationship between duplicate references, cache matching and re-exposure.

Diff composition

Diff composition (non-blocking): every file is required for the change; none flagged.

Ran the pinned introduced_blobs.py helper successfully for the full baseline-to-head range: nine content-write occurrences across four commits and three paths. Reviewed their commit deltas as well as the head diff; no scratch artifacts or credential-bearing content found.

Simplifications

Simplifications (non-blocking): none. The existing list plus a membership guard is a local solution without new state or abstraction.

Author questions

  • Q1: answered by joalves; confirmation is still absent, and the PR remains draft pending the owner's decision. The answer is consistent with the PR description and does not resolve the contract itself.
  • Unanswered author questions: none. The outstanding owner decision is recorded above under Human review required.

@joalves
joalves removed the request for review from marcio-absmartly November 17, 2026 18:50
@joalves

joalves commented Nov 17, 2026

Copy link
Copy Markdown
Contributor Author

Closing: the owner confirmed the backend never sends duplicate holdoutIds, so there's no behaviour to change.

@joalves joalves closed this Nov 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants