Skip to content

fix(membase): retry and avoid connection resets on graph queries - #1509

Merged
Oceania2018 merged 2 commits into
masterfrom
fix/membase-retry-transport-errors
Nov 20, 2026
Merged

Oceania2018 merged 2 commits into
masterfrom
fix/membase-retry-transport-errors

Conversation

@Oceania2018

@Oceania2018 Oceania2018 commented Nov 20, 2026 •

Copy link
Copy Markdown
Member

Problem

Production callers regularly log Refit.ApiRequestException → HttpRequestException → SocketException (104): Connection reset by peer from MembaseGraphDb.ExecuteQueryAsync, and the query fails on its first attempt (surfacing as a 500 from callers such as /proposal-tool/search-rescode). Server-side telemetry for the reported trace ids shows the requests never reached Membase; the service was healthy (CPU < 10%, no 5xx) and neighbouring queries returned 200.

Root causes

  1. No retry. BuildRetryPolicy handles HttpRequestException and ApiException. Since Refit 10, transport failures are thrown as ApiRequestException, which derives from ApiExceptionBase — a sibling of ApiException, not a subclass — wrapping the HttpRequestException. The policy matched neither, so transient network failures were never retried. (This repo pins Refit 8.0.0, but hosts that resolve a newer Refit hit this.)
  2. Stale pooled connections. Membase's Azure front end closes keep-alive connections idle for ~2–3 minutes (measured: reuse after 125 s succeeds, after 185 s fails). NATs, firewalls and proxies on the caller's path may drop flows sooner, and a reused connection whose flow has been forgotten is answered with a RST.

Fix

  • Retry transport failures: treat an exception as retryable when it is an HttpRequestException, or a non-ApiException whose inner exception is one. Written without referencing ApiRequestException, so it compiles against Refit 8 and works with newer versions. Client timeouts are still not retried.
  • Dedicated SocketsHttpHandler for the Membase client: PooledConnectionIdleTimeout = 30s, PooledConnectionLifetime = 5min, and TCP keepalive (first probe after 30 s idle, then every 10 s, 3 probes) via ConnectCallback, so idle connections are retired before anything on the path forgets them and long-running queries keep their flow alive.

Testing

  • BotSharp.Plugin.Membase builds (0 errors).
  • Verified the Refit 13.1.0 hierarchy by reflection: ApiRequestException : ApiExceptionBase, ApiException : ApiExceptionBase.
  • Ran the same handler configuration against api.membase.dev: socket options applied (KeepAlive=1, TcpKeepAliveTime=30), two requests returned 200 over one reused connection.
  • No automated test added: the plugin has no test project.

🤖 Generated with Claude Code

Newer Refit (10+) wraps transport failures in ApiRequestException, which is a
sibling of ApiException rather than a subclass, and no longer throws the bare
HttpRequestException. The retry policy matched neither, so a connection reset
on a pooled connection failed the query on the first attempt. Match the
wrapper by its inner HttpRequestException; Refit 8 behaviour is unchanged and
client timeouts are still not retried.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@qodo-code-review

Copy link
Copy Markdown
Contributor

Qodo reviews are paused for this user.

Troubleshooting steps vary by plan Learn more →

On a Teams plan?
Reviews resume once this user has a paid seat and their Git account is linked in Qodo.
Link Git account →

Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center?
These require an Enterprise plan - Contact us
Contact us →

Membase's front end closes connections idle for about two minutes, and NATs,
firewalls and proxies on the caller's side may drop them sooner, which shows
up as "Connection reset by peer" on a reused pooled connection. Give the
Membase HTTP client its own SocketsHttpHandler that retires connections idle
for 30 s, recycles every connection after 5 minutes, and turns on TCP
keepalive (30 s, then every 10 s) so a long-running query's flow is not
forgotten mid-request.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Oceania2018 Oceania2018 changed the title fix(membase): retry graph queries whose connection was reset fix(membase): retry and avoid connection resets on graph queries Nov 20, 2026
@Oceania2018
Oceania2018 merged commit 11b5059 into master Nov 20, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant