Skip to content

RabbitMQ service: a cluster of three, growing from one, and losing a majority (S-023-8, S-023-9, S-023-10, S-023-11) #508

Open

Description

@shipmill-romamo

Builds part of spec S-023, docs/specs/023-rabbitmq-service.md: the acceptance criteria below. Each needs a test that names its id (test_s023_<k>_..., or a proves: S-023-8 comment line).

Scope: three nodes: the node list, hostname resolution between nodes, inter-node TLS verified against the project CA, the port requirements, joining one node at a time, the healthy wait, growing a one-node service to three, and the PRECONDITION stop while the cluster has no majority.

  • S-023-8: Converging the declaration above on three hosts gives one cluster of three running nodes; a durable queue declared on h1 has three members; a message published to it through h1 with a publisher confirm is consumed through h3
  • S-023-9: Each server's ruleset admits ports 4369 and 25672 from the other two nodes only; a connection between nodes is encrypted and verified against the project CA; a node started with another cookie, or with a certificate from another CA, does not join
  • S-023-10: Adding two nodes to the service of S-023-3 and converging gives the cluster of S-023-8 with every queue and message the first node held, each quorum queue grown to three members with no step by a person, and no queue redeclared; a second converge changes nothing
  • S-023-11: With h2 stopped, a publish with a confirm to a quorum queue through h1 succeeds and audit reports the node as missing; when h2 returns it holds those messages with no converge. With h2 and h3 stopped, the same publish is not confirmed, audit reports the cluster without majority, and a converge stops with PRECONDITION without restarting h1; when a node returns, publishing works again with no step by a person

Depends on #506

Activity

  1. shipmill-romamo commented on Nov 17, 2026

    @shipmill-romamo
    ContributorAuthor

    Triage: implement. Serves: G-1, G-3.

    Builds S-023-8 to S-023-11 of spec 023 (docs/specs/023-rabbitmq-service.md, approved in #500), for #502: three nodes with inter-node TLS against the project CA, joining one node at a time, growing a one-node service to three, and the PRECONDITION stop without a majority, as the body above scopes it.

    It starts once #506 (one node converged) closes; its Depends on line holds it until then. I'll link the PR here.

  2. shipmill-romamo commented on Nov 18, 2026

    @shipmill-romamo
    ContributorAuthor

    PR: #541. Three nodes join one at a time from the home node, resolve each other through a managed /etc/hosts block, open 4369 and 25672 to the other two nodes only, and talk over TLS with certificates from the project CA; a health wait follows the last join, and a converge stops with PRECONDITION (service_rabbitmq_no_majority) before any change while no more than half the members run. audit reports a missing node and a cluster without majority. One node renders and behaves as before.

    Proven by tests: the rendered settings, rules, certificates, join and health conditions, the majority check's order and refusal, and the audit reports of S-023-8 to S-023-11. Not verified on a real host: a cluster actually forming, quorum queues growing to three members, confirmed publishes with a node stopped, and a node with another cookie or CA failing to join.

    The PR waits on one question about how nodes check each other's certificates (on the PR).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions