Skip to content

fix(membase): bound a graph query's total time, let the caller cancel it, and stop retrying timeouts - #1508

Merged
yileicn merged 5 commits into
SciSharp:masterfrom
yuyixg:fix/membase-query-time-budget
Nov 20, 2026
Merged

yileicn merged 5 commits into
SciSharp:masterfrom
yuyixg:fix/membase-query-time-budget

Conversation

@yuyixg

@yuyixg yuyixg commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Problem

MembaseGraphDb.ExecuteQueryAsync retries transient failures (502/503/504 and non-query 500s) three times with 2 s, 4 s and 8 s back-off. When Membase keeps returning 500, one query stays busy for 14 s of back-off alone. A caller that waits with its own deadline has given up long before, and the retries keep hitting Membase while it is already failing. A caller also had no way to stop a query whose result it no longer needs.

The per-attempt timeout (TimeoutSecond / 3) never fired. Polly timeouts are optimistic and only cancel through the CancellationToken passed to the delegate, but IMembaseApi.CypherQueryAsync took no token. A hanging call therefore ran until the 90 s HttpClient timeout.

Change

  • IMembaseApi.CypherQueryAsync takes a CancellationToken, and the policy passes its token through, so timeouts and cancellation now reach the HTTP call.
  • One timeout covers the whole query, back-off included. It is GraphQueryExecuteOptions.Timeout (new, optional) when the caller sets it, and Membase:TimeoutSecond otherwise.
  • The per-attempt timeout is removed, and timeouts are no longer retried. Once it actually fired, a healthy query slower than TimeoutSecond / 3 was cut, retried and timed out every time: about 54 s and a failure, instead of a slower success. A query that runs out of time is usually just slow, so another attempt does not help. Only transient HTTP failures are retried.
  • GraphQueryExecuteOptions.CancellationToken (new, [JsonIgnore]) lets the caller cancel the query, retries included. A typical use is passing HttpContext.RequestAborted. A caller's cancellation is not treated as a Membase error:
    • It is logged at Information level, without the query text.
    • It surfaces as OperationCanceledException, even when the HTTP client wraps the cancellation in its own exception.
  • MembaseController.ExecuteGraphQuery takes a CancellationToken action parameter (bound by MVC to RequestAborted) and passes it on, so an aborted /membase/{graphId}/query request stops its query.
  • The Refit client timeout is Math.Max(90 s, TimeoutSecond) instead of a fixed 90 s, so a larger TimeoutSecond is not cut at 90 s. The 90 s floor stays for calls made on IMembaseApi directly, which have no other timeout.
With the default TimeoutSecond = 30 Before After
Healthy query that takes 12 s succeeds at 12 s succeeds at 12 s
Call that hangs fails at 90 s (HttpClient) fails at 30 s
Consecutive 500s fails after ~14 s, 3 retries same, within 30 s
Caller sets Timeout = 5 s (no such option) fails at 5 s
Caller cancels (not possible) stops at once, no retry

Behaviour change to note

A healthy query that takes between TimeoutSecond and 90 s used to succeed and now times out. Raise Membase:TimeoutSecond if a deployment runs queries that slow.

Testing

Tests live in the downstream app that consumes this plugin. They use a mocked IMembaseApi and check six cases:

  • Consecutive 500s fail within a 1 s budget, after a single call.
  • A hanging call with a 0.5 s budget sees its token cancelled.
  • With no budget and TimeoutSecond = 1, a hanging call fails in about 1 s, with no retry.
  • With TimeoutSecond = 3, a 1.5 s query succeeds on its first attempt.
  • A caller cancellation stops the query at once, with no retry.
  • A caller cancellation that the HTTP client wraps still surfaces as OperationCanceledException.

The full downstream suite passes against this branch.

🤖 Generated with Claude Code

…ncluded

ExecuteQueryAsync retried transient failures with 2/4/8 s back-off, so a Membase
outage kept each query busy for well over its callers' own deadlines. The
per-attempt timeout never fired either: Polly timeouts are optimistic and need a
cancellation token, but CypherQueryAsync took none, so a hanging call ran to the
90 s HttpClient timeout.

- GraphQueryExecuteOptions.Timeout: optional upper bound for the whole query,
  applied as the outermost policy so it also cuts the back-off short. Null keeps
  today's behaviour.
- IMembaseApi.CypherQueryAsync takes a CancellationToken, and the retry policy
  passes its token through, so both timeouts now cancel the HTTP call.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@qodo-code-review

Copy link
Copy Markdown
Contributor

Qodo reviews are paused for this user.

Troubleshooting steps vary by plan Learn more →

On a Teams plan?
Reviews resume once this user has a paid seat and their Git account is linked in Qodo.
Link Git account →

Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center?
These require an Enterprise plan - Contact us
Contact us →

…rying timeouts

The per-attempt timeout (TimeoutSecond / 3) started firing once the previous
commit passed the token through. A healthy query slower than that limit was then
cut and retried, timed out the same way each time, and failed after ~54 s
instead of succeeding.

A query that times out is usually just slow, so retrying it does not help. Now
one timeout covers the whole query, back-off included: the caller's
GraphQueryExecuteOptions.Timeout, or Membase:TimeoutSecond when none is given.
Only transient HTTP failures are retried.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@yuyixg yuyixg changed the title fix(membase): let callers bound a graph query's total time, retries included fix(membase): bound a graph query's total time, and stop retrying timeouts Nov 20, 2026
GraphQueryExecuteOptions.CancellationToken (JSON-ignored) cancels the query,
retries and back-off included, e.g. when the request that needs the result is
aborted. MembaseGraphDb hands it to the retry policy, so it reaches the HTTP
call.

A caller's cancellation is not a Membase error: it is logged at information
level without the query text, and surfaced as OperationCanceledException even
when the HTTP client wraps it in its own exception.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@yuyixg yuyixg changed the title fix(membase): bound a graph query's total time, and stop retrying timeouts fix(membase): bound a graph query's total time, let the caller cancel it, and stop retrying timeouts Nov 20, 2026
@yileicn
yileicn requested a lite review from Copilot November 20, 2026 07:23

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Two moderate issues remain: request cancellation is not propagated through the public query endpoint, and the hard-coded 90-second Refit timeout can override configured or caller-supplied longer budgets.

2 open issues
What changed in this PR

This PR makes Membase graph queries cancellable and bounds their total execution time, including retry backoff, while preventing timeout-triggered retries.

Changes:

  • Adds cancellation-token support to the Membase API and propagates it through Polly to Refit.
  • Replaces per-attempt timeouts with one configurable whole-query timeout.
  • Restricts retries to transient HTTP failures and normalizes caller cancellation as OperationCanceledException.
File Description
src/​Plugins/​BotSharp.Plugin.Membase/​Interfaces/​IMembaseApi.cs Adds cancellation-token support to the Membase Cypher API call.
src/​Plugins/​BotSharp.Plugin.Membase/​GraphDb/​MembaseGraphDb.cs Applies total query timeouts, propagates cancellation, and adjusts retry behavior.
src/​Infrastructure/​BotSharp.Abstraction/​Graph/​Options/​GraphQueryOptions.cs Adds configurable timeout and caller-cancellation options for graph queries.

Review effort: Lite

…nd exceed 90 s

- MembaseController.ExecuteGraphQuery passes HttpContext.RequestAborted, so an
  aborted /membase/{graphId}/query request stops its query and retries.
- The Refit client timeout was a fixed 90 s, which cut any longer
  Membase:TimeoutSecond short. It is now the larger of 90 s and TimeoutSecond.
  The 90 s floor stays because calls made on IMembaseApi directly have no other
  timeout.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues were identified, and the changes consistently propagate cancellation while enforcing a bounded whole-query timeout and preserving appropriate transient-failure retries.

2 open issues

Review effort: Lite

…n parameter

MVC binds a CancellationToken action parameter to HttpContext.RequestAborted,
and a direct call without an HTTP context gets CancellationToken.None instead of
a null HttpContext.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The query action still converts caller-request cancellation into a 500 error instead of rethrowing it, leaving an active moderate-severity issue.

No open issues 🎉

1 previously missed issue

In code that hasn't changed since last review

Medium severity Propagate request cancellation instead of returning a 500

src/​Plugins/​BotSharp.Plugin.Membase/​Controllers/​MembaseController.cs:87

The token is now propagated into MembaseGraphDb, but this action still catches the resulting OperationCanceledException in the broad handler below and converts caller/request cancellation into a 500 ErrorResult. That treats a disconnected /membase/{graphId}/query request as a server error (and prevents cancellation from surfacing to in-process callers of the action); rethrow cancellation when cancellationToken.IsCancellationRequested before handling actual failures.

2 resolved since last review

Review effort: Lite

@yileicn

yileicn commented Nov 20, 2026

Copy link
Copy Markdown
Member

reviewed

@yileicn
yileicn merged commit 27aa167 into SciSharp:master Nov 20, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants