Skip to content

Converge a RabbitMQ cluster of three nodes (S-023-8 to S-023-11) - #541

Open
shipmill-romamo[bot] wants to merge 2 commits into
mainfrom
feat/508-rabbitmq-cluster
Open

shipmill-romamo[bot] wants to merge 2 commits into
mainfrom
feat/508-rabbitmq-cluster

Conversation

@shipmill-romamo

@shipmill-romamo shipmill-romamo Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Plan

  • Serves: G-1, G-3
  • Files: sdk/src/cloudfall/rabbitmq_cluster.py (new), requirements.py, elasticsearch_converge.py, lifecycle.py, audit.py, engine/src/cloudfall_engine/ansible_inventory.py, the cloudfall_rabbitmq role (tasks/{main,tls,cluster,majority}.yml, templates/{90-cloudfall-listeners.conf,inter_node_tls.config}.j2, filter_plugins/rabbitmq_cluster.py, library/cloudfall_rabbitmq_health.py, defaults, argument specs), services.yml, the cloudfall_inspect role, observed-server.schema.json, tests, changelog.d/508-rabbitmq-cluster.md
  • Order: the requirements and rendered cluster (ports, certificate, node list, home first), the certificate staging, then the role (majority check before any change, TLS and hosts, join wait, healthy wait), then the PRECONDITION answer, inspect and audit; the one-node path kept as it was; rebased on Serve RabbitMQ clients over TLS with clientAccess (S-023-19) #542 so three nodes and clientAccess share one certificate (tls.yml) and one definition of its paths (rabbitmq_access.py); rebased again on main after the backup and restore drill (Back up RabbitMQ definitions and drill their restore on a scratch node (S-023-16, S-023-17) #540, Make the RabbitMQ restore drill's scratch tree as the rabbitmq user (#544) #545): the drill's calendar check and the majority check both run before any change
  • Proof: S-023-8, S-023-9, S-023-10, S-023-11 in sdk/tests/test_s023_rabbitmq_cluster.py and engine/tests/test_s023_rabbitmq_cluster_role.py, three nodes with clientAccess (S-023-19) in the same files; S-023-2, S-023-3, S-023-6 and S-023-19 keep holding

Summary

A RabbitMQ service of three nodes now converges instead of being refused (spec S-023, docs/specs/023-rabbitmq-service.md).

  • Node list and resolution: the inventory renders cloudfall_rabbitmq_cluster on each node (its IPv4, every node's name, hostname and IPv4, its certificate paths). The role writes a managed /etc/hosts block, before Debian's 127.0.1.1 line, naming every node at its IPv4, so the nodes and the command-line tools reach each other there

  • Ports: each node gets PortRequirements for 4369 and 25672 from the other two nodes, and its firewall admits exactly those sources (ip saddr <other>/32 tcp dport 4369|25672 accept, plus each other node's IPv6 network as the other services' rules do). audit expects those rules too. epmd and the inter-node port leave loopback on the node's own IPv4 only; the listener check allows exactly those two

  • Inter-node TLS: nodes keep short names (rabbit@h2). Each node has a CertificateRequirement (usage server and client, /etc/rabbitmq/cloudfall/{node.crt,node.key,ca.crt}) naming its hostname, its IPv4 and its short hostname (h2 for h2.example.internal), the host part of its node name. The services converge has each node make its key and the project CA sign its certificate, as it does for MariaDB and Elasticsearch, and hands it to the RabbitMQ role alone (cloudfall_services_rabbitmq_delivered_files). The key is 0600 rabbitmq. inter_node_tls.config sets verify_peer against the project CA on both sides and fail_if_no_peer_cert on the server side, and leaves Erlang's default server_name_indication in force: inet_tls_dist sends the host part of the peer's node name (h2), and ssl checks the peer certificate's name against it, so the peer's name is checked. rabbitmq-env.conf runs the node and the tools (SERVER_ADDITIONAL_ERL_ARGS, CTL_ERL_ARGS) with -proto_dist inet_tls

  • Joining one at a time: the service's home is first in the rollout order, and services.yml is serial. A new node joins the running home through its classic-config node list, and the role waits until cluster_status lists it and the home as running. A node that started a cluster of its own stops the converge there

  • Healthy wait: once every declared node is a member, a read-only module (cloudfall_rabbitmq_health) waits up to an hour until every node is a running member, no alarm or partition is reported, and every quorum queue has a leader and a member online on every node. Otherwise it stops with what is missing

  • Growing one to three: no task declares, deletes or moves a queue. Continuous membership reconciliation, already on with target size 3, grows each quorum queue. The node that was alone restarts once for its new settings

  • No majority: before any task that could change or restart the node, the role reads cluster_status. When no more than half of the members run, it reports service_rabbitmq_no_majority to the services converge's report directory and fails. The converge answers with PRECONDITION, and no node is restarted

  • Audit: inspect reads a cluster's members on its home (rabbitmqCluster), and audit adds rabbitmq.cluster[<id>]: drift for a missing node, an undeclared member, and a cluster without majority. A service of one node gets no such check

  • With clientAccess (Serve RabbitMQ clients over TLS with clientAccess (S-023-19) #542): a node of three may declare clientAccess. The node's one certificate from the project CA serves the inter-node TLS and the 5671 client listener. tls.yml prepares it whenever either needs it, and its paths are defined once, in cloudfall.rabbitmq_access. Each node's ruleset holds the 5671 rule for each allowed source, plus 4369/25672 from the other two nodes only. The listener check allows exactly those three listeners on the node's own IPv4

One node is unchanged: its settings, rabbitmq-env.conf block, inventory, requirements (a one-node clientAccess certificate names its hostname and IPv4 only, as before) and extra vars render as before (tested), and every new task is skipped on it.

Test notes

  • New: sdk/tests/test_s023_rabbitmq_cluster.py (requirements, the short hostname among each node's certificate names, rendered firewall through the nftables template, rendered cluster and rollout order, certificates signed by the project CA for server and client auth with the short hostname as a DNS name, PRECONDITION on a no-majority report, audit) and engine/tests/test_s023_rabbitmq_cluster_role.py (rendered settings, TLS options with no server_name_indication, verify_fun or customize_hostname_check override, env block, hosts lines, listener check, majority filter, task order, and the majority tasks, the join condition and the health module run for real on localhost against the stand-in rabbitmq-diagnostics/rabbitmqctl)
  • engine/tests/rabbitmq_node.py's stand-in now answers cluster_status, list_vhosts and JSON list_queues, and reads the first non-option word as the command
  • Passed locally after the rebase: ruff check ., mypy, and the tests of the touched files (test_s023_*, test_pki, test_requirements, test_ansible_inventory, test_examples_render_unchanged, test_s019_mariadb_remote_access*, test_s012_elasticsearch_converge, test_domains_certificate). ansible-lint engine/ansible was run without ANSIBLE_CONFIG and reported only role-path syntax-check failures; CI's run with ANSIBLE_CONFIG decides it. The full suite is left to CI

Not verified (needs three live Debian 13 hosts)

No live multi-host run was made, so these parts are not claimed:

  • S-023-8: a message published through h1 with a confirm and consumed through h3, and RabbitMQ really forming one cluster through classic-config peer discovery
  • S-023-9: a TLS connection between real nodes, the peer's short hostname actually passing Erlang's name check, and a node with another cookie, another CA's certificate or a certificate for another name failing to join (the options and certificate names that make it so are tested, the handshake is not). The role reads inet_tls_dist's directory with erl on the host, which no test runs
  • S-023-10: the messages a single node held surviving the growth, and reconciliation actually adding members on a live cluster
  • S-023-11: a confirmed publish with h2 stopped, h2 catching up, and publishing resuming when a node returns. The PRECONDITION stop and the audit reports are tested; the broker behaviour is not

Fixes #508

🤖 Generated with Claude Code

@shipmill-romamo

Copy link
Copy Markdown
Contributor Author

@romamo Decision needed: may RabbitMQ nodes check each other's certificate against the project CA without checking the hostname in it (server_name_indication, disable), as this PR does?
CI on 72c0ffe: 2 checks green

  1. Accept it (recommended): S-023-9 asks for a connection "verified against the project CA", and this meets it; Erlang compares the short node name (h2) with a certificate naming h2.example.internal, so a hostname check fails on every cluster unless nodes run with long names. A node holding any certificate of the same environment's CA passes TLS, and still needs the Erlang cookie to join. This is the setup RabbitMQ's own inter-node TLS guide shows
  2. Check the hostname too: nodes run with long names (rabbit@h2.example.internal) and a hostname match function, so a certificate issued to another server of the environment is refused. It costs a rework of this PR (node names, the /etc/hosts block, the tools' arguments) and a node-name change that S-023-10's growth from one node would have to handle

Reply here with a number or your own answer; shipmill takes this up on the tick after your reply.

@romamo

romamo commented Nov 19, 2026

Copy link
Copy Markdown
Collaborator

Neither as written: keep short node names, add each node's short hostname (h2) to its certificate's names and drop server_name_indication, disable, so the name is checked.

@shipmill-romamo

Copy link
Copy Markdown
Contributor Author

Taken up on @romamo's answer: nodes keep short names (rabbit@h2), each node's certificate adds its short hostname (h2) to its subject alternative names, and server_name_indication, disable is dropped, so Erlang's default check of the peer's name runs against the project CA's certificate.

Next on this PR: rebase onto main (it conflicts now), make that change with a test that the rendered TLS config no longer disables the check and that the node certificate names the short hostname, then land it once CI is green.

@shipmill-romamo shipmill-romamo Bot removed the needs-decision A shipmill session asked a question here; waits for a reply label Nov 19, 2026
shipmill-romamo Bot and others added 2 commits November 19, 2026 19:56
A RabbitMQ service of three nodes now converges instead of being refused.
Each node runs node discovery and the inter-node port on its IPv4, which
its firewall admits from the other two nodes only; every node resolves the
others' hostnames to their IPv4 through a managed /etc/hosts block; and the
node and its command-line tools speak TLS between nodes with the node's
certificate from the project CA, verified against it both ways. The
services converge has each node make its key and the project CA sign its
certificate, and hands it to the RabbitMQ role alone. A node of three may
also declare clientAccess: one certificate per node serves the inter-node
TLS and the 5671 client listener, and the certificate paths are defined
once, in cloudfall.rabbitmq_access.

The service's home converges first and the other nodes join it one at a
time; once every declared node is a member, the role waits up to an hour
for the service to be healthy and otherwise stops with what is missing.
Growing a service of one to three redeclares no queue: membership
reconciliation grows each quorum queue. Before anything that could change
or restart a node, the role reads the cluster and stops with PRECONDITION
(service_rabbitmq_no_majority) when no more than half of its members run.
inspect reads a cluster's members on its home, and audit reports a missing
node and a cluster without majority. One node renders and converges as it
did.

Fixes #508

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Nodes keep short names (rabbit@h2). Each node of three's certificate from
the project CA now names its short hostname, the host part of its node
name, beside its FQDN and IPv4, and the inter-node TLS options no longer
set server_name_indication to disable. Erlang's inet_tls_dist then sends
the node name's host as server_name_indication, which ssl uses to verify
the peer certificate's name, so a node connecting to another checks it.
Both sides still verify the peer against the project CA, and the server
still refuses a peer without a certificate. A service of one node, with or
without clientAccess, keeps the certificate names it had.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@shipmill-romamo
shipmill-romamo Bot force-pushed the feat/508-rabbitmq-cluster branch from 72c0ffe to 449245c Compare November 19, 2026 18:03
@shipmill-romamo

Copy link
Copy Markdown
Contributor Author

CI is green on 449245c (checks and converge), and the pull request merges cleanly, but the merge gate refuses it: base: red: converge / converge failed on main@92481fc.

That failure is the job in Release run https://github.com/cloudfall-dev/cloudfall/actions/runs/44391818494 (workflow_dispatch, 2026-11-18 20:31 UTC, failed after 7 s). The same job passed on the same commit in Release run 44391773941 (21:08 UTC) and Publish run 44408318149 (22:14 UTC), and Converge run 44391772858 passed too. Rerunning the failed job would rerun that run's land jobs as well, so it is not rerun here.

On hold: waits on #535 (the merge gate keeps an older failed run of a workflow that a newer run on the same commit passed), or on the next commit to main, which clears the base check.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

RabbitMQ service: a cluster of three, growing from one, and losing a majority (S-023-8, S-023-9, S-023-10, S-023-11)

1 participant