Skip to content

shipmill: the merge gate reads main red on a failed run a newer run of the same workflow passed #535

Open

Description

@shipmill-romamo

shipmill: pr_gate.py reads the default branch as red on a failed check that a newer run of the same workflow, on the same commit, already passed. Nothing lands on main until the next commit there, and the only rerun that clears it would publish again. #534 waits on it.

Reproduce

main is at 004111d. Two workflow_dispatch runs of Publish ran on that commit:

uv run python <github-pr-triage>/scripts/pr_gate.py 534 --bot-login 'shipmill-romamo[bot]' --agent (shipmill 0.145.0) on 2026-11-18 prints base: red: converge / converge failed on main@004111d and exits 1.

Touches

shipmill's github-pr-triage scripts/pr_gate.py: the base check (D-75)

Suspected code

run_entry keys each check run by its check suite ("workflowName": f"suite {suite['id']}"), so newest_entries keeps the newest run per suite, and two runs of one workflow are two suites. The failed run's job never gets superseded. The documented remedy, gh run rerun <id> --failed, reruns converge and then the dependent build and publish, which would publish a commit that is already published.

Proposed shape: for check runs from Actions, group by the workflow (the run's workflow_id or path) and the job name, keep the newest by started_at, and let a newer green run of the same job on the same commit clear an older failure. A rerun inside one suite keeps working as it does now.

Activity

  1. shipmill-romamo commented on Nov 18, 2026

    @shipmill-romamo
    ContributorAuthor

    One correction to the report: the failed run did not test main's code. Publish takes a tag input, and run 44109292844 converged refs/tags/v0.8.0rc76 (7741863, cut from 889271d), while the check is filed on main@004111d only because the run was dispatched from main. Run 44121225686 converged and published v0.8.0rc77 (2b29fc6).

    So rerunning the failed jobs of 44109292844 would not republish an already published commit: it would converge rc76 again and, if that passes, publish 0.8.0rc76 to PyPI after 0.8.0rc77. The gate's fix also has to cover this case: a check from a workflow_dispatch run that checks out another ref says nothing about the default branch's head.

    #428 and #504 wait on this too, as #534 does.

  2. shipmill-romamo commented on Nov 18, 2026

    @shipmill-romamo
    ContributorAuthor

    One more finding: rerunning run 44109292844 would not pass. Publish calls ./.github/workflows/converge.yml, which resolves at the dispatching ref (main), and checks out the code at refs/tags/v0.8.0rc76. Main's converge workflow has a domains apply probe-site step that relies on engine/tests/converge/project/domains/probe-site.yaml, and that file arrived between rc76 and rc77 (git diff --stat v0.8.0rc76 v0.8.0rc77). So Domain does not exist: probe-site happens on every rerun. Only a fix here or a new commit on main clears the base. #464 waits on it as well.

  3. shipmill-romamo commented on Nov 19, 2026

    @shipmill-romamo
    ContributorAuthor

    A second instance, on main@92481fc (2026-11-19): pr_gate.py 552 prints base: red: converge / converge failed on main@92481fc and exits 1.

    • The failed job is 135111539881 in the hotfix Release dispatch https://github.com/cloudfall-dev/cloudfall/actions/runs/44391818494 (20:31 UTC 2026-11-18). It used converge.yml@refs/heads/main but checked out 59bbeaa (the 0.7.1 hotfix branch, since deleted, in no tag), where engine/tests/converge/project doesn't exist: An error occurred trying to start process '/usr/bin/bash' with working directory '.../engine/tests/converge/project'. No such file or directory
    • converge / converge on the same commit passed afterwards in runs 44391773941 (21:01) and 44408318149 (22:06)

    So a workflow_dispatch check that converged another ref blocks the base again, now for #552 too. A rerun can't clear it (the directory is still missing at 59bbeaa). Nothing lands on main until this is fixed or main gets a new commit.

  4. shipmill-romamo commented on Nov 19, 2026

    @shipmill-romamo
    ContributorAuthor

    It happened again on 2026-11-19, and it holds #541, which is green and mergeable.

    main is at 92481fc. shipmill gh pr merge 541 (shipmill 0.149.2) is refused with base: red: converge / converge failed on main@92481fc. The failed job is in Release run https://github.com/cloudfall-dev/cloudfall/actions/runs/44391818494 (workflow_dispatch, failed at 20:31 UTC on 2026-11-18). Later runs on the same commit passed the same job:

    • Release run 44391773941: converge / converge passed at 21:08 UTC
    • Publish run 44408318149: converge / converge passed at 22:14 UTC

    Rerunning the failed job (gh run rerun 44391818494 --failed) would also rerun the land jobs that depend on it. Until this is fixed, or main gets another commit, no pull request lands.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    upstreamA report waiting on another repo's work

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions