Skip to content

Report a hung pinset fetch instead of failing content.yml - #44

Merged
TakGN merged 2 commits into
mainfrom
content-pinset-fetch-timeout
Nov 18, 2026
Merged

TakGN merged 2 commits into
mainfrom
content-pinset-fetch-timeout

Conversation

@TakGN

@TakGN TakGN commented Nov 17, 2026 •

Copy link
Copy Markdown
Collaborator

Why

On 2026-11-17 content.yml -l chic-1 failed at "Measure each missing entry under the size gate":

failed: [chic-1] (item=cluster.ipfs.io (estimate 13861128 bytes))
msg: 'Task failed: Timed out after 180 second(s).'

That entry is cluster.ipfs.io's new DNSLink root (bafybeidrs7w…). It has not been retrievable since 2026-11-11. The run on all three boxes earlier the same day reported it as it should ("UNRETRIEVABLE cluster.ipfs.io … DAG not fetched: … promise channel was closed") and carried on. This time the kubectl exec … ipfs dag stat hung past ipfs's own --timeout=120s, and the task's timeout: (180 s) fired.

The role means a fetch timeout to be reported ("rc is absent when the task timeout fired, so default it to a failure"), but Ansible fails a task that hits timeout: whatever failed_when says, so the host failed and the run stopped before the report. Reproduced locally with ansible-core 2.19.4: a looped command with failed_when: false and timeout: 3 on sleep 10 fails the host.

Change

roles/content/tasks/main.yml, the two fetches of a missing pinset entry ("Find each missing root and estimate its size", "Measure each missing entry under the size gate"):

  • Run under coreutils timeout --kill-after=10, 30 s past ipfs's own --timeout. A hang is killed on the box (no kubectl exec left behind) and comes back as rc 124, or 137 if it ignored TERM and needed the KILL; the existing code already treats any non-zero rc as a failed fetch.
  • The task timeout: stays as a backstop, moved to +90 s so it fires only after the inner kill. ignore_errors: true (with # noqa ignore-errors, as the role's other loud-failure tasks) means even the backstop is reported with the entry rather than stopping the run.
  • The reported error reads "timed out" for rc 124 or 137, in the two fail-loudly tasks and in the pin decision, instead of an empty stderr.

The pin itself (ipfs-cluster-ctl pin add --no-status) is left alone: it returns at once and has never hung.

Checked

  • Locally, ansible-core 2.19.4:
    • as on main: the task timeout fails the host despite failed_when: false;
    • with the change: coreutils timeout gives rc 124 and the run continues; a backstop timeout with ignore_errors continues with rc absent.
  • The role's real "Decide which missing entries to pin" task, run on mocked results: a hung dag stat → unretrievable "DAG not fetched: timed out"; a hung root that needed the KILL (rc 137) → "root not found: timed out"; a backstop kill → "DAG not fetched: Task failed: Timed out after 210 second(s)."; a healthy entry is still pinned.
  • ansible-playbook --syntax-check content.yml, yamllint ., ansible-lint roles/content/tasks/main.yml (0 failures).
  • Not run against a box. timeout is part of coreutils, which every box has.

After merging

Nothing to apply: the next content.yml run uses it. cluster.ipfs.io's new root is still unretrievable; this only makes the run report it and go on.

🤖 Generated with Claude Code

A `kubectl exec` that hangs past ipfs's own --timeout hit the task
timeout, which Ansible fails whatever failed_when says, so content.yml
stopped on chic-1 instead of listing the entry as unretrievable. Run each
fetch under coreutils timeout on the box, so a hang comes back as rc 124,
and keep the task timeout as a backstop that no longer stops the run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- name: Find each missing root and estimate its size # noqa ignore-errors
ansible.builtin.command:
argv: [k3s, kubectl, --namespace=content, exec, deploy/content, -c, kubo, --,
argv: [timeout, --kill-after=10, "{{ content_root_timeout | int + 30 }}",

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: --kill-after does not give rc 124 in the case it exists for. In GNU coreutils, if the command ignores the TERM and timeout has to send KILL, it exits 137 (128+9) on purpose, "to distinguish a forcibly killed command". 124 is returned only when TERM was enough. This applies to both fetches (here and at line 639).

A kubectl exec that is still hung 10 s after TERM therefore returns rc 137. The entry is still treated as a failed fetch, because rc != 0. But the reported error is the empty string, which is the symptom this PR sets out to remove. The 'timed out' if r.rc | default(0) == 124 checks (four places) don't match 137.

Cheapest fix: test in [124, 137] in all four places. Alternatively, drop --kill-after, which leaves a hang that ignores TERM to the task-timeout backstop.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 98f28ae: all four checks now match in [124, 137], and the comment says 137 is what a fetch that needed the KILL returns. Confirmed locally: a command that ignores TERM gives 137 under timeout --kill-after. Re-ran the decision task with a hung root returning 137; it now reports root not found: timed out.

@claude

claude Bot commented Nov 17, 2026

Copy link
Copy Markdown

Claude review

The change does what it says. timeout kills a hung kubectl exec on the box and returns a non-zero rc. ignore_errors: true keeps the run going even if the Ansible task-timeout backstop fires. The existing consumers (rc | default(1) != 0, the loop filters and the decision template) already treat a missing or non-zero rc as a failed fetch. I checked the Jinja precedence of the new 'timed out' if … else a | trim or b | trim expressions, and they parse as intended. The inner timeout is 30 s past ipfs's own --timeout and the backstop is 90 s past it, so the order is correct for any content_*_timeout value. I did not run the playbook, and one claim in the PR is not quite right (below).

Findings

  1. Nit — roles/content/tasks/main.yml:590 (and :639): with --kill-after, coreutils timeout exits 137, not 124, when it has to send KILL because the command ignored TERM. So the "timed out" message is only produced when TERM was enough. A kubectl exec that survives TERM is still reported as a failed fetch, because rc is non-zero. Its error text would be empty, as before this PR. Match in [124, 137] in the four message expressions to cover both.

claude-sonnet-5-5 at high effort · requested by TakGN · re-run with @claude review [sonnet|opus|fable] [low|medium|high|xhigh|max]

timeout --kill-after exits 137, not 124, when the command ignored TERM,
which is the case the KILL is there for.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@TakGN
TakGN merged commit 411ad1e into main Nov 18, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant