You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
shipmill: the merge gate reads main red on a failed run a newer run of the same workflow passed #535
shipmill: pr_gate.py reads the default branch as red on a failed check that a newer run of the same workflow, on the same commit, already passed. Nothing lands on main until the next commit there, and the only rerun that clears it would publish again. #534 waits on it.
Reproduce
main is at 004111d. Two workflow_dispatch runs of Publish ran on that commit:
uv run python <github-pr-triage>/scripts/pr_gate.py 534 --bot-login 'shipmill-romamo[bot]' --agent (shipmill 0.145.0) on 2026-11-18 prints base: red: converge / converge failed on main@004111d and exits 1.
Touches
shipmill's github-pr-triage scripts/pr_gate.py: the base check (D-75)
Suspected code
run_entry keys each check run by its check suite ("workflowName": f"suite {suite['id']}"), so newest_entries keeps the newest run per suite, and two runs of one workflow are two suites. The failed run's job never gets superseded. The documented remedy, gh run rerun <id> --failed, reruns converge and then the dependent build and publish, which would publish a commit that is already published.
Proposed shape: for check runs from Actions, group by the workflow (the run's workflow_id or path) and the job name, keep the newest by started_at, and let a newer green run of the same job on the same commit clear an older failure. A rerun inside one suite keeps working as it does now.
One correction to the report: the failed run did not test main's code. Publish takes a tag input, and run 44109292844 converged refs/tags/v0.8.0rc76 (7741863, cut from 889271d), while the check is filed on main@004111d only because the run was dispatched from main. Run 44121225686 converged and published v0.8.0rc77 (2b29fc6).
So rerunning the failed jobs of 44109292844 would not republish an already published commit: it would converge rc76 again and, if that passes, publish 0.8.0rc76 to PyPI after 0.8.0rc77. The gate's fix also has to cover this case: a check from a workflow_dispatch run that checks out another ref says nothing about the default branch's head.
One more finding: rerunning run 44109292844 would not pass. Publish calls ./.github/workflows/converge.yml, which resolves at the dispatching ref (main), and checks out the code at refs/tags/v0.8.0rc76. Main's converge workflow has a domains apply probe-site step that relies on engine/tests/converge/project/domains/probe-site.yaml, and that file arrived between rc76 and rc77 (git diff --stat v0.8.0rc76 v0.8.0rc77). So Domain does not exist: probe-site happens on every rerun. Only a fix here or a new commit on main clears the base. #464 waits on it as well.
A second instance, on main@92481fc (2026-11-19): pr_gate.py 552 prints base: red: converge / converge failed on main@92481fc and exits 1.
The failed job is 135111539881 in the hotfix Release dispatch https://github.com/cloudfall-dev/cloudfall/actions/runs/44391818494 (20:31 UTC 2026-11-18). It used converge.yml@refs/heads/main but checked out 59bbeaa (the 0.7.1 hotfix branch, since deleted, in no tag), where engine/tests/converge/project doesn't exist: An error occurred trying to start process '/usr/bin/bash' with working directory '.../engine/tests/converge/project'. No such file or directory
converge / converge on the same commit passed afterwards in runs 44391773941 (21:01) and 44408318149 (22:06)
So a workflow_dispatch check that converged another ref blocks the base again, now for #552 too. A rerun can't clear it (the directory is still missing at 59bbeaa). Nothing lands on main until this is fixed or main gets a new commit.
It happened again on 2026-11-19, and it holds #541, which is green and mergeable.
main is at 92481fc. shipmill gh pr merge 541 (shipmill 0.149.2) is refused with base: red: converge / converge failed on main@92481fc. The failed job is in Release run https://github.com/cloudfall-dev/cloudfall/actions/runs/44391818494 (workflow_dispatch, failed at 20:31 UTC on 2026-11-18). Later runs on the same commit passed the same job:
Release run 44391773941: converge / converge passed at 21:08 UTC
Publish run 44408318149: converge / converge passed at 22:14 UTC
Rerunning the failed job (gh run rerun 44391818494 --failed) would also rerun the land jobs that depend on it. Until this is fixed, or main gets another commit, no pull request lands.
Activity
shipmill-romamo commentedon Nov 18, 2026
One correction to the report: the failed run did not test main's code. Publish takes a
taginput, and run 44109292844 convergedrefs/tags/v0.8.0rc76(7741863, cut from 889271d), while the check is filed on main@004111d only because the run was dispatched from main. Run 44121225686 converged and publishedv0.8.0rc77(2b29fc6).So rerunning the failed jobs of 44109292844 would not republish an already published commit: it would converge rc76 again and, if that passes, publish 0.8.0rc76 to PyPI after 0.8.0rc77. The gate's fix also has to cover this case: a check from a
workflow_dispatchrun that checks out another ref says nothing about the default branch's head.#428 and #504 wait on this too, as #534 does.
shipmill-romamo commentedon Nov 18, 2026
One more finding: rerunning run 44109292844 would not pass. Publish calls
./.github/workflows/converge.yml, which resolves at the dispatching ref (main), and checks out the code atrefs/tags/v0.8.0rc76. Main's converge workflow has adomains apply probe-sitestep that relies onengine/tests/converge/project/domains/probe-site.yaml, and that file arrived between rc76 and rc77 (git diff --stat v0.8.0rc76 v0.8.0rc77). SoDomain does not exist: probe-sitehappens on every rerun. Only a fix here or a new commit on main clears the base. #464 waits on it as well.shipmill-romamo commentedon Nov 19, 2026
A second instance, on main@92481fc (2026-11-19):
pr_gate.py 552printsbase: red: converge / converge failed on main@92481fcand exits 1.converge.yml@refs/heads/mainbut checked out 59bbeaa (the 0.7.1 hotfix branch, since deleted, in no tag), whereengine/tests/converge/projectdoesn't exist:An error occurred trying to start process '/usr/bin/bash' with working directory '.../engine/tests/converge/project'. No such file or directoryconverge / convergeon the same commit passed afterwards in runs 44391773941 (21:01) and 44408318149 (22:06)So a
workflow_dispatchcheck that converged another ref blocks the base again, now for #552 too. A rerun can't clear it (the directory is still missing at 59bbeaa). Nothing lands on main until this is fixed or main gets a new commit.shipmill-romamo commentedon Nov 19, 2026
It happened again on 2026-11-19, and it holds #541, which is green and mergeable.
main is at 92481fc.
shipmill gh pr merge 541(shipmill 0.149.2) is refused withbase: red: converge / converge failed on main@92481fc. The failed job is in Release run https://github.com/cloudfall-dev/cloudfall/actions/runs/44391818494 (workflow_dispatch, failed at 20:31 UTC on 2026-11-18). Later runs on the same commit passed the same job:converge / convergepassed at 21:08 UTCconverge / convergepassed at 22:14 UTCRerunning the failed job (
gh run rerun 44391818494 --failed) would also rerun thelandjobs that depend on it. Until this is fixed, or main gets another commit, no pull request lands.