Skip to content

Run kubo on go-libp2p v0.52.0 on every content and bootstrap node - #41

Merged
TakGN merged 3 commits into
mainfrom
kubo-libp2p-v052
Nov 17, 2026
Merged

TakGN merged 3 commits into
mainfrom
kubo-libp2p-v052

Conversation

@TakGN

@TakGN TakGN commented Nov 16, 2026 •

Copy link
Copy Markdown
Collaborator

Part of libp2p/go-libp2p#3564 and ipfs/kubo#11480. Follows #37.

Why

go-libp2p v0.52.0 (2026-11-14) is the first release with the inbound-slot leak fix, libp2p/go-libp2p#3568. Upstream asked us to move to the release and keep reporting the resource-manager gauges. No kubo release uses it yet, and one will take a while, so this moves every content and bootstrap kubo to our own build of it now. kubo v0.43.1 replaces go-libp2p with lidel/go-libp2p@9c4ee60, so a release-built kubo can't pick it up with go get; ipfs/kubo#11480 asks for a kubo release with it.

The fix-only canaries, 2026-11-11 to 2026-11-16

These are the two canaries #37 put on v0.43.1 plus only the five lines of #3568. Leaked inbound slots = rcmgr system inbound − transient − live inbound (swarm opened − closed). Hub Prometheus, 2026-11-16 14:43 UTC:

kubo build running since (UTC) leaked now last 24 h since 11-11 ~16:30
chic-1 bootstrap-b v0.43.1 + #3568 11-11 15:35 0 0 0
chic-1 bootstrap (same box) v0.43.1 11-02 13:38 7,598 +468 +2,028 (~410/day)
sing-1 bootstrap v0.43.1 11-02 13:36 8,633 +533 +2,305 (~470/day)
lith-1 bootstrap v0.43.1 11-02 13:31 5,147 +301 +1,327 (~270/day)
sing-1 content v0.43.1 + #3568 11-11 17:24 0 0 0
lith-1 content v0.43.1 11-11 17:42 3,802 +780 3,802 since restart (~780/day)
chic-1 content v0.43.1 11-11 17:45 3,685 +736 3,685 since restart (~750/day)

Over those five days the canaries read 0–1, the scrape-timing noise upstream told us to expect. Their traffic matches their controls': bootstrap-b holds 17,008 live inbound connections (9,708 QUIC, 6,330 TCP) against 17,515 on bootstrap next to it. Both open about 20 inbound connections a second. Neither canary restarted or fired an alert. The results are in libp2p/go-libp2p#3564 (18 h comparison); they are why the fix went into v0.52.0.

What else changes

Unlike the fix-only build, this one changes more than the fix. Moving from lidel's v0.49-based fork to v0.52.0, go mod tidy also raises shared dependencies:

  • quic-go 0.62 → 0.64.1 (QUIC carries 45–80% of inbound connections here);
  • golang.org/x/net, crypto and sys;
  • prometheus client_golang 1.25 and procfs 0.23;
  • klauspost/compress 1.20.1;
  • goupnp, go-nat-pmp and go-ssdp.

Upstream has only checked that kubo starts on it. The fix itself has five days of production behind it; the dependency bumps don't. So the rollout below goes one box at a time, starting with the two former canary nodes, and the rollback is one digest.

The image

ghcr.io/ipni/kubo:v0.43.1-libp2p-v0.52.0 (sha256:71d7b748…), amd64, pushed by hand, pullable without logging in.

  • Built from ipfs/kubo v0.43.1 (acc53c0) with kubo's own Dockerfile, unchanged. The only source change is in go.mod / go.sum: go mod edit -replace github.com/libp2p/go-libp2p=github.com/libp2p/go-libp2p@v0.52.0, then go mod tidy. The go 1.26.5 line is unchanged.
  • Checked: v0.52.0's p2p/net/upgrader/listener.go has the fix. go version -m against upstream ipfs/kubo:v0.43.1 shows the same Go 1.26.5, CGO_ENABLED=1, -trimpath and amd64 v1, with => github.com/libp2p/go-libp2p v0.52.0 replacing the lidel line, plus the dependency bumps above. The version string is unchanged (0.43.1-acc53c0).
  • Smoke test, run locally: the daemon reaches "Daemon is ready" with no errors and finds 323 peers in 20 s. /debug/metrics/prometheus still exports libp2p_rcmgr_connections and libp2p_swarm_connections_*, so the leak gauge and the *InboundSlotsLeaking alerts in ipni/infra keep working.
  • The build recipe is still not in a repo.

Change

  • roles/content/defaults/main.yml: content_kubo_image → the v0.52.0 build, for every content box. The comment records the build, the fix-only results and the upstream digest to roll back to.
  • roles/bootstrap/defaults/main.yml: bootstrap_default_kubo_image → the same digest, for every bootstrapper instance.
  • host_vars/sing-1/content.yml, host_vars/chic-1/bootstrap.yml: drop the canary overrides from Run a kubo with the go-libp2p inbound-slot leak fix on sing-1's content node and chic-1's bootstrap-b #37; both nodes now take the role default.
  • README.md: the Images note says what kubo runs and why.

Checked

  • ansible-playbook --syntax-check on content.yml and bootstrap.yml.
  • scripts/render-manifests.yml renders ghcr.io/ipni/kubo@sha256:71d7b748… for both k8s/content and k8s/bootstrap; no kubo_image override is left in host_vars or group_vars.
  • The image is amd64 only, like the boxes. scripts/recover-upstream-pinset.sh runs the content default locally, so on an arm64 machine it now runs under emulation.
  • Not run locally: yamllint, ansible-lint (CI runs them).

After merging

Merging deploys nothing. The baseline for every content and bootstrap kubo was taken on 2026-11-16 at 14:43 UTC from the hub Prometheus: leak, live connections by direction and transport, rcmgr blocks and streams, RSS, CPU, goroutines and fds.

Each run restarts that box's kubo(s), which clears the leaked slots. One box at a time, --check --diff first (expect only the kubo image lines to change), and a look at the box's kubo (pod up, peers, no alerts) before the next:

# The two former canary nodes first
ansible-playbook content.yml -l sing-1 --check --diff
ansible-playbook content.yml -l sing-1
ansible-playbook bootstrap.yml -l chic-1 --check --diff   # bootstrap and bootstrap-b
ansible-playbook bootstrap.yml -l chic-1

# Then the rest
ansible-playbook content.yml -l lith-1 --check --diff
ansible-playbook content.yml -l lith-1
ansible-playbook content.yml -l chic-1 --check --diff
ansible-playbook content.yml -l chic-1
ansible-playbook bootstrap.yml -l sing-1 --check --diff
ansible-playbook bootstrap.yml -l sing-1
ansible-playbook bootstrap.yml -l lith-1 --check --diff
ansible-playbook bootstrap.yml -l lith-1

Then, for a few days, compare against the baseline:

  • the leak gauge on every node (expect 0–1);
  • the QUIC/TCP split of live inbound connections;
  • rcmgr blocks, RSS, CPU and goroutines.

The *InboundSlotsLeaking alerts in ipni/infra stay as they are, as a guard.

Rollback: set both role defaults back to ipfs/kubo sha256:b293923d… (upstream v0.43.1; the full digest is in the content defaults comment), or to ghcr.io/ipni/kubo sha256:44f66498… (v0.43.1 plus only the fix), and re-run the same commands.

When kubo ships a release with go-libp2p ≥ v0.52.0 (ipfs/kubo#11480), move both defaults back to ipfs/kubo.

🤖 Generated with Claude Code

go-libp2p v0.52.0 is the first release with the inbound-slot leak fix
(libp2p/go-libp2p#3568). No kubo release uses it yet, so sing-1's content
node and chic-1's bootstrap-b run ipfs/kubo v0.43.1 rebuilt on it, in
place of v0.43.1 plus only the fix, before every box moves to it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@claude

claude Bot commented Nov 16, 2026

Copy link
Copy Markdown

Claude review

This PR changes only the kubo image digest for the two canaries (sing-1 content, chic-1 bootstrap-b), plus comments and one README sentence. The digest is the same full 64-hex value in both files. The old fix-only digest remains only in a comment, which is correct for the rollback note. The controls (lith-1, chic-1 bootstrap) are untouched, and the change deploys nothing until -l <box> runs, which keeps the blast radius to one kubo per box. I could not check the image contents or the registry digest from here, so that rests on the PR description (a manual push, pullable anonymously). I found no correctness, rollout or compatibility problems in the diff.

No issues found.

claude-sonnet-5-5 at high effort · requested by TakGN · re-run with @claude review [sonnet|opus|fable] [low|medium|high|xhigh|max]

No kubo release ships go-libp2p v0.52.0 yet, and one will take a while.
Move the content and bootstrap role defaults to our v0.43.1 rebuild on
v0.52.0, which fixes the inbound-slot leak, and drop the canary overrides
on sing-1 and chic-1.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@TakGN TakGN changed the title Move the kubo leak canaries to go-libp2p v0.52.0 Run kubo on go-libp2p v0.52.0 on every content and bootstrap node Nov 17, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@TakGN
TakGN merged commit a410f35 into main Nov 17, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant