Repository navigation
Report a hung pinset fetch instead of failing content.yml - #44
Conversation
A `kubectl exec` that hangs past ipfs's own --timeout hit the task timeout, which Ansible fails whatever failed_when says, so content.yml stopped on chic-1 instead of listing the entry as unretrievable. Run each fetch under coreutils timeout on the box, so a hang comes back as rc 124, and keep the task timeout as a backstop that no longer stops the run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
| - name: Find each missing root and estimate its size # noqa ignore-errors | ||
| ansible.builtin.command: | ||
| argv: [k3s, kubectl, --namespace=content, exec, deploy/content, -c, kubo, --, | ||
| argv: [timeout, --kill-after=10, "{{ content_root_timeout | int + 30 }}", |
There was a problem hiding this comment.
Nit: --kill-after does not give rc 124 in the case it exists for. In GNU coreutils, if the command ignores the TERM and timeout has to send KILL, it exits 137 (128+9) on purpose, "to distinguish a forcibly killed command". 124 is returned only when TERM was enough. This applies to both fetches (here and at line 639).
A kubectl exec that is still hung 10 s after TERM therefore returns rc 137. The entry is still treated as a failed fetch, because rc != 0. But the reported error is the empty string, which is the symptom this PR sets out to remove. The 'timed out' if r.rc | default(0) == 124 checks (four places) don't match 137.
Cheapest fix: test in [124, 137] in all four places. Alternatively, drop --kill-after, which leaves a hang that ignores TERM to the task-timeout backstop.
There was a problem hiding this comment.
Fixed in 98f28ae: all four checks now match in [124, 137], and the comment says 137 is what a fetch that needed the KILL returns. Confirmed locally: a command that ignores TERM gives 137 under timeout --kill-after. Re-ran the decision task with a hung root returning 137; it now reports root not found: timed out.
Claude reviewThe change does what it says. Findings
claude-sonnet-5-5 at high effort · requested by TakGN · re-run with |
timeout --kill-after exits 137, not 124, when the command ignored TERM, which is the case the KILL is there for. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Why
On 2026-11-17
content.yml -l chic-1failed at "Measure each missing entry under the size gate":That entry is cluster.ipfs.io's new DNSLink root (
bafybeidrs7w…). It has not been retrievable since 2026-11-11. The run on all three boxes earlier the same day reported it as it should ("UNRETRIEVABLE cluster.ipfs.io … DAG not fetched: … promise channel was closed") and carried on. This time thekubectl exec … ipfs dag stathung past ipfs's own--timeout=120s, and the task'stimeout:(180 s) fired.The role means a fetch timeout to be reported ("rc is absent when the task timeout fired, so default it to a failure"), but Ansible fails a task that hits
timeout:whateverfailed_whensays, so the host failed and the run stopped before the report. Reproduced locally with ansible-core 2.19.4: a loopedcommandwithfailed_when: falseandtimeout: 3onsleep 10fails the host.Change
roles/content/tasks/main.yml, the two fetches of a missing pinset entry ("Find each missing root and estimate its size", "Measure each missing entry under the size gate"):timeout --kill-after=10, 30 s past ipfs's own--timeout. A hang is killed on the box (nokubectl execleft behind) and comes back as rc 124, or 137 if it ignored TERM and needed the KILL; the existing code already treats any non-zero rc as a failed fetch.timeout:stays as a backstop, moved to +90 s so it fires only after the inner kill.ignore_errors: true(with# noqa ignore-errors, as the role's other loud-failure tasks) means even the backstop is reported with the entry rather than stopping the run.The pin itself (
ipfs-cluster-ctl pin add --no-status) is left alone: it returns at once and has never hung.Checked
failed_when: false;timeoutgives rc 124 and the run continues; a backstop timeout withignore_errorscontinues with rc absent.dag stat→ unretrievable "DAG not fetched: timed out"; a hung root that needed the KILL (rc 137) → "root not found: timed out"; a backstop kill → "DAG not fetched: Task failed: Timed out after 210 second(s)."; a healthy entry is still pinned.ansible-playbook --syntax-check content.yml,yamllint .,ansible-lint roles/content/tasks/main.yml(0 failures).timeoutis part of coreutils, which every box has.After merging
Nothing to apply: the next
content.ymlrun uses it. cluster.ipfs.io's new root is still unretrievable; this only makes the run report it and go on.🤖 Generated with Claude Code