Repository navigation
fix(membase): retry and avoid connection resets on graph queries - #1509
Merged
Merged
Conversation
Newer Refit (10+) wraps transport failures in ApiRequestException, which is a sibling of ApiException rather than a subclass, and no longer throws the bare HttpRequestException. The retry policy matched neither, so a connection reset on a pooled connection failed the query on the first attempt. Match the wrapper by its inner HttpRequestException; Refit 8 behaviour is unchanged and client timeouts are still not retried. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
Qodo reviews are paused for this user.Troubleshooting steps vary by plan Learn more → On a Teams plan? Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center? |
Membase's front end closes connections idle for about two minutes, and NATs, firewalls and proxies on the caller's side may drop them sooner, which shows up as "Connection reset by peer" on a reused pooled connection. Give the Membase HTTP client its own SocketsHttpHandler that retires connections idle for 30 s, recycles every connection after 5 minutes, and turns on TCP keepalive (30 s, then every 10 s) so a long-running query's flow is not forgotten mid-request. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Production callers regularly log
Refit.ApiRequestException→HttpRequestException→SocketException (104): Connection reset by peerfromMembaseGraphDb.ExecuteQueryAsync, and the query fails on its first attempt (surfacing as a 500 from callers such as/proposal-tool/search-rescode). Server-side telemetry for the reported trace ids shows the requests never reached Membase; the service was healthy (CPU < 10%, no 5xx) and neighbouring queries returned 200.Root causes
BuildRetryPolicyhandlesHttpRequestExceptionandApiException. Since Refit 10, transport failures are thrown asApiRequestException, which derives fromApiExceptionBase— a sibling ofApiException, not a subclass — wrapping theHttpRequestException. The policy matched neither, so transient network failures were never retried. (This repo pins Refit 8.0.0, but hosts that resolve a newer Refit hit this.)Fix
HttpRequestException, or a non-ApiExceptionwhose inner exception is one. Written without referencingApiRequestException, so it compiles against Refit 8 and works with newer versions. Client timeouts are still not retried.SocketsHttpHandlerfor the Membase client:PooledConnectionIdleTimeout = 30s,PooledConnectionLifetime = 5min, and TCP keepalive (first probe after 30 s idle, then every 10 s, 3 probes) viaConnectCallback, so idle connections are retired before anything on the path forgets them and long-running queries keep their flow alive.Testing
BotSharp.Plugin.Membasebuilds (0 errors).ApiRequestException : ApiExceptionBase,ApiException : ApiExceptionBase.api.membase.dev: socket options applied (KeepAlive=1,TcpKeepAliveTime=30), two requests returned 200 over one reused connection.🤖 Generated with Claude Code