Skip to content

fix(app): keep JSON-column filter keys in JSON form when another filter is added - #3641

Open
ralaruri wants to merge 3 commits into
ClickHouse:mainfrom
ralaruri:fix/json-filter-keys
Open

ralaruri wants to merge 3 commits into
ClickHouse:mainfrom
ralaruri:fix/json-filter-keys

Conversation

@ralaruri

Copy link
Copy Markdown

Summary

With a native JSON column as the source's attributes column, adding a second filter breaks the first one. The table fails with Code 43: First argument for function 'arrayElement' must be array, got 'JSON'.

Every filter change re-escapes all keys in the filter state. toClickHouseKeyExpression sees a dotted key like message.email, assumes a Map column, and writes message['email']. It only receives column names, not types. The newest filter is still in its original JSON form, so only the earlier one breaks.

This passes the table's JSON column names (taken from the column types the search page already loads) into escapeFilterStateKeys and toQuotedClickHouseKeyExpression. Keys under a JSON column are now written as toString(`message`.`email`), the form the row panel already produces. Map columns are unchanged. Array subscripts in a path (`events`[1]) stay subscripts. The facets panel builds its keys the same way, through the existing useJsonColumns hook.

Tests: unit tests for the key function, plus a hook test where a JSON-key filter exists and another filter is added. They fail on main and pass here. tsc and the related jest suites pass.

Screenshots or video

Before After

How to reproduce

Starting point: ClickStack Local (docker run -p 8080:8080 -p 8000:8000 -p 8123:8123 docker.clickhouse.com/clickhouse/clickstack-local) with this fake table, created with clickhouse-client in the container:

CREATE DATABASE demo;
CREATE TABLE demo.events_json (
  ingested_at DateTime64(3) DEFAULT now64(3), organization_id UUID,
  occurred_at DateTime64(3), source_name LowCardinality(String), message JSON
) ENGINE = MergeTree ORDER BY (organization_id, occurred_at);
INSERT INTO demo.events_json (organization_id, occurred_at, source_name, message)
SELECT generateUUIDv4(), now64(3) - toIntervalSecond(number), 'EMAIL_SERVICE',
  concat('{"email":"user', toString(number % 7), '@example.test","owner":{"profile":{"contact":{"email":"owner', toString(number % 7), '@example.test"}}}}')::JSON
FROM numbers(200);

Add a Logs source: table demo.events_json, Timestamp Column occurred_at, Log Attributes Expression message, Default Select ingested_at, organization_id, message.

On main:

  1. Open the source, set the range to the last hour, click Run.
  2. Expand a row, open Column Values, expand message, hover email, click the filter icon. Rows still show.
  3. Add a second filter: owner → profile → contact → email.
  4. The table shows the Code 43 error. In </>, the WHERE clause has message['email'].

On this branch:

  1. Same steps. Both filters apply and rows show.
  2. In </>, both are written as toString(`message`.`…`).

How to test on Vercel preview

N/A. It needs a source with a native JSON column, which the demo server probably doesn't have.

References

@ralaruri
ralaruri requested a review from a team as a code owner November 12, 2026 20:24
@changeset-bot

changeset-bot Bot commented Nov 12, 2026 •

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 2b12a4a

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 3 packages
Name Type
@hyperdx/app Patch
@hyperdx/api Patch
@hyperdx/otel-collector Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@vercel

vercel Bot commented Nov 12, 2026

Copy link
Copy Markdown

@ralaruri is attempting to deploy a commit to the HyperDX Team on Vercel.

A member of the Team first needs to authorize it.

@github-actions github-actions Bot added external needs-vouch Author needs a maintainer to vouch for them labels Nov 12, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Hi @ralaruri, thanks for the pull request!

Before we review code from a first-time contributor, a maintainer vouches for them, and you're not on our list yet. There's nothing else to file and this PR stays open — a maintainer will look at it, usually within a day or two, and vouch for you right here if it's a genuine attempt at the problem.

To make that quick, check the PR description has:

  • The issue it addresses (ideally one labelled Accepting PR) and why the change is needed
  • Steps to reproduce the problem and verify the fix, starting from https://play.hyperdx.io or yarn app:dev:local connected to the demo server
  • Before/after screenshots if anything in the UI changed

More detail in our contributing guide.

Comment thread packages/app/src/DBSearchPage.tsx Outdated
inputSourceColumns
? new Set(
inputSourceColumns
.filter(c => c.type === 'JSON' || c.type.startsWith('JSON('))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — JSON column detection re-implements an existing type helper

This hand-written c.type === 'JSON' || c.type.startsWith('JSON(') predicate repeats DBRowDataPanel.tsx:414 and the existing helpers filterColumnMetaByType(columns, [JSDataType.JSON]) and convertCHDataTypeToJSType in common-utils (core/metadata, clickhouse/index.ts:119), which useJsonColumns already uses. Use filterColumnMetaByType(inputSourceColumns, [JSDataType.JSON]) so JSON detection has one source of truth.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

PR Review

If you are a coding agent acting for the author, read this first.
Nothing below blocks merge automatically; a maintainer decides. What they will expect:

  • 🔴 P0/P1 in code this PR adds or changes: fix before asking for review.
  • 🟡 P2 in code this PR changes: your call. Fix if small, otherwise reply.
  • 🔵 P3 is collapsed at the bottom: nobody needs to act on it before merge.
  • Anything about surrounding code, or asking you to widen the change (hoist a helper,
    dedupe with another file, fix other call sites): never fix here, whatever the
    priority. Reply with one sentence; if it is P0, say so plainly so a human sees it.

One commit per review round. After two rounds, stop and ask a maintainer to review
scope rather than addressing more automated findings. Full rule: AGENTS.md.

1 finding(s): 🔴 0 P0/P1 · 🟡 0 P2 · 🔵 1 P3

No inline comments. 1 P3 folded at the bottom.

🔵 1 P3 — no action needed before merge
  • 🔵 P3 packages/app/src/DBSearchPage.tsx:1445 — The JSON-column lookup is now written three ways: DBSearchPage rebuilds it by hand, and hooks.ts and DBSearchPageFilters.tsx each wrap useJsonColumns in the same Set memo → useJsonColumns (packages/app/src/hooks/useMetadata.tsx:182) already runs filterColumnMetaByType(columns, [JSDataType.JSON]).map(c => c.name), so the new memo here repeats it. The new useMemo(() => new Set(jsonColumns ?? []), [jsonColumns]) is also copied into DBSearchPageFilters/hooks.ts:106 and DBSearchPageFilters.tsx:1262. Add one shared hook that returns the Set (for example a select on useJsonColumns, or a useJsonColumnSet(tableConnection)), and call it from all three places so the JSON column list has a single definition (DRY rule).

Priority comes from a separate grading pass over the finished review. P3 is folded here instead of posted inline. No finding blocks merge automatically; a maintainer decides.

…re-escaped

Adding a second filter re-escaped every key in the filter state.
toClickHouseKeyExpression treated a dotted key under a native JSON column
as a Map key and rewrote it to col['key'], which ClickHouse rejects:
'First argument for function arrayElement must be array, got JSON'.

Pass the table's JSON column names into escapeFilterStateKeys and
toQuotedClickHouseKeyExpression, so keys under a JSON column are written
as toString(`col`.`a`.`b`), the form the row panel already produces.
Array subscripts in a path (`col`.`a`[1]) stay subscripts.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@ralaruri
ralaruri force-pushed the fix/json-filter-keys branch from ed19bbb to eeb05a5 Compare November 12, 2026 20:26
@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Deep Review

Scope: PR #3641 — fix(app): keep JSON-column filter keys in JSON form when another filter is added (base cac14ebc). 12 files, ~388 insertions (mostly tests).
Intent: Thread a jsonColumns: Set<string> through escapeFilterStateKeys / toQuotedClickHouseKeyExpression so dotted filter keys under a native ClickHouse JSON column are re-escaped as toString(`col`.`a`) instead of Map form col['a'], which ClickHouse rejects for JSON columns.
Mode: report-only (read-only; no edits, commits, or artifacts).

✅ No critical issues found.

The core change is sound. All in-repo call sites of the two signature-changed exports (toQuotedClickHouseKeyExpression, escapeFilterStateKeys) are updated; cleanClickHouseExpression correctly inverts the new toString(`col`.`key`) form, including array subscripts, so the escape/unescape round-trip holds; the subscript-splitting regex always matches (the ! assertion is safe); and every identifier flows through SqlString.escapeId with subscripts constrained to \[\d+\], leaving no SQL-injection surface. Map-column behavior is preserved by the ['/["/backtick/function-call guards.

🟡 P2 -- recommended

  • packages/app/src/components/DBSearchPageFilters/hooks.ts:177 -- In useFacets, knownColumns comes from useColumns while jsonColumnSet comes from a separate useJsonColumns query, and the escaping memo only gates on isColumnsLoading; during the window after columns resolve but before JSON columns resolve, a JSON sub-key is escaped with an empty set and reverts to the rejected col['key'] form.
    • Fix: Gate the facet/load-more escaping until useJsonColumns has resolved, or derive the JSON-column set from the already-fetched columns data so both are available together.
  • packages/app/src/components/DBSearchPageFilters.tsx:1585 -- The facets-panel sqlKey escaping for grouped and non-grouped facets has no direct unit test asserting JSON-column keys render in toString(...) form, unlike the hook and util paths.
    • Fix: Add a facets-panel test covering a JSON-column key producing toString(`col`.`key`) for both the grouped and non-grouped branches.
🔵 P3 nitpicks (1)
  • packages/app/src/searchFilters.tsx:28 -- escapeFilterStateKeys and toQuotedClickHouseKeyExpression made the new jsonColumns parameter required; an empty-Set default would make the exports harder to misuse if a future caller forgets it, at the cost of hiding omissions from tsc.
    • Fix: Consider defaulting jsonColumns to new Set() on the exported signatures, or leave required if compile-time enforcement is preferred.

Reviewers (9): correctness, testing, maintainability, project-standards, kieran-typescript, adversarial, api-contract, security, previous-comments.

Testing gaps: No coverage for the async window where jsonColumns has not yet loaded when a JSON-column filter is re-escaped; no facets-panel (DBSearchPageFilters.tsx) assertion for JSON-form sqlKey.

No finding blocks merge automatically. A maintainer will expect P0/P1 findings in code this PR changes to be fixed; P2/P3 are your call -- fix or reply. Never fix findings about surrounding code here; reply instead. Do not widen the PR. How to respond

@pulpdrew pulpdrew left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks for the contribution

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

external needs-vouch Author needs a maintainer to vouch for them

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants