Repository navigation
Converge a RabbitMQ cluster of three nodes (S-023-8 to S-023-11) - #541
shipmill-romamo[bot] wants to merge 2 commits into
Conversation
9fa9c0b to
72c0ffe
Compare
|
@romamo Decision needed: may RabbitMQ nodes check each other's certificate against the project CA without checking the hostname in it (
Reply here with a number or your own answer; shipmill takes this up on the tick after your reply. |
|
Neither as written: keep short node names, add each node's short hostname (h2) to its certificate's names and drop server_name_indication, disable, so the name is checked. |
|
Taken up on Next on this PR: rebase onto main (it conflicts now), make that change with a test that the rendered TLS config no longer disables the check and that the node certificate names the short hostname, then land it once CI is green. |
A RabbitMQ service of three nodes now converges instead of being refused. Each node runs node discovery and the inter-node port on its IPv4, which its firewall admits from the other two nodes only; every node resolves the others' hostnames to their IPv4 through a managed /etc/hosts block; and the node and its command-line tools speak TLS between nodes with the node's certificate from the project CA, verified against it both ways. The services converge has each node make its key and the project CA sign its certificate, and hands it to the RabbitMQ role alone. A node of three may also declare clientAccess: one certificate per node serves the inter-node TLS and the 5671 client listener, and the certificate paths are defined once, in cloudfall.rabbitmq_access. The service's home converges first and the other nodes join it one at a time; once every declared node is a member, the role waits up to an hour for the service to be healthy and otherwise stops with what is missing. Growing a service of one to three redeclares no queue: membership reconciliation grows each quorum queue. Before anything that could change or restart a node, the role reads the cluster and stops with PRECONDITION (service_rabbitmq_no_majority) when no more than half of its members run. inspect reads a cluster's members on its home, and audit reports a missing node and a cluster without majority. One node renders and converges as it did. Fixes #508 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Nodes keep short names (rabbit@h2). Each node of three's certificate from the project CA now names its short hostname, the host part of its node name, beside its FQDN and IPv4, and the inter-node TLS options no longer set server_name_indication to disable. Erlang's inet_tls_dist then sends the node name's host as server_name_indication, which ssl uses to verify the peer certificate's name, so a node connecting to another checks it. Both sides still verify the peer against the project CA, and the server still refuses a peer without a certificate. A service of one node, with or without clientAccess, keeps the certificate names it had. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
72c0ffe to
449245c
Compare
|
CI is green on 449245c ( That failure is the job in Release run https://github.com/cloudfall-dev/cloudfall/actions/runs/44391818494 ( On hold: waits on #535 (the merge gate keeps an older failed run of a workflow that a newer run on the same commit passed), or on the next commit to main, which clears the base check. |
Plan
sdk/src/cloudfall/rabbitmq_cluster.py(new),requirements.py,elasticsearch_converge.py,lifecycle.py,audit.py,engine/src/cloudfall_engine/ansible_inventory.py, thecloudfall_rabbitmqrole (tasks/{main,tls,cluster,majority}.yml,templates/{90-cloudfall-listeners.conf,inter_node_tls.config}.j2,filter_plugins/rabbitmq_cluster.py,library/cloudfall_rabbitmq_health.py, defaults, argument specs),services.yml, thecloudfall_inspectrole,observed-server.schema.json, tests,changelog.d/508-rabbitmq-cluster.mdclientAccessshare one certificate (tls.yml) and one definition of its paths (rabbitmq_access.py); rebased again on main after the backup and restore drill (Back up RabbitMQ definitions and drill their restore on a scratch node (S-023-16, S-023-17) #540, Make the RabbitMQ restore drill's scratch tree as the rabbitmq user (#544) #545): the drill's calendar check and the majority check both run before any changesdk/tests/test_s023_rabbitmq_cluster.pyandengine/tests/test_s023_rabbitmq_cluster_role.py, three nodes withclientAccess(S-023-19) in the same files; S-023-2, S-023-3, S-023-6 and S-023-19 keep holdingSummary
A RabbitMQ service of three nodes now converges instead of being refused (spec S-023,
docs/specs/023-rabbitmq-service.md).Node list and resolution: the inventory renders
cloudfall_rabbitmq_clusteron each node (its IPv4, every node's name, hostname and IPv4, its certificate paths). The role writes a managed/etc/hostsblock, before Debian's127.0.1.1line, naming every node at its IPv4, so the nodes and the command-line tools reach each other therePorts: each node gets
PortRequirements for 4369 and 25672 from the other two nodes, and its firewall admits exactly those sources (ip saddr <other>/32 tcp dport 4369|25672 accept, plus each other node's IPv6 network as the other services' rules do).auditexpects those rules too. epmd and the inter-node port leave loopback on the node's own IPv4 only; the listener check allows exactly those twoInter-node TLS: nodes keep short names (
rabbit@h2). Each node has aCertificateRequirement(usage server and client,/etc/rabbitmq/cloudfall/{node.crt,node.key,ca.crt}) naming its hostname, its IPv4 and its short hostname (h2forh2.example.internal), the host part of its node name. The services converge has each node make its key and the project CA sign its certificate, as it does for MariaDB and Elasticsearch, and hands it to the RabbitMQ role alone (cloudfall_services_rabbitmq_delivered_files). The key is0600 rabbitmq.inter_node_tls.configsetsverify_peeragainst the project CA on both sides andfail_if_no_peer_certon the server side, and leaves Erlang's defaultserver_name_indicationin force:inet_tls_distsends the host part of the peer's node name (h2), andsslchecks the peer certificate's name against it, so the peer's name is checked.rabbitmq-env.confruns the node and the tools (SERVER_ADDITIONAL_ERL_ARGS,CTL_ERL_ARGS) with-proto_dist inet_tlsJoining one at a time: the service's home is first in the rollout order, and
services.ymlis serial. A new node joins the running home through its classic-config node list, and the role waits untilcluster_statuslists it and the home as running. A node that started a cluster of its own stops the converge thereHealthy wait: once every declared node is a member, a read-only module (
cloudfall_rabbitmq_health) waits up to an hour until every node is a running member, no alarm or partition is reported, and every quorum queue has a leader and a member online on every node. Otherwise it stops with what is missingGrowing one to three: no task declares, deletes or moves a queue. Continuous membership reconciliation, already on with target size 3, grows each quorum queue. The node that was alone restarts once for its new settings
No majority: before any task that could change or restart the node, the role reads
cluster_status. When no more than half of the members run, it reportsservice_rabbitmq_no_majorityto the services converge's report directory and fails. The converge answers withPRECONDITION, and no node is restartedAudit:
inspectreads a cluster's members on its home (rabbitmqCluster), andauditaddsrabbitmq.cluster[<id>]: drift for a missing node, an undeclared member, and a cluster without majority. A service of one node gets no such checkWith
clientAccess(Serve RabbitMQ clients over TLS with clientAccess (S-023-19) #542): a node of three may declareclientAccess. The node's one certificate from the project CA serves the inter-node TLS and the 5671 client listener.tls.ymlprepares it whenever either needs it, and its paths are defined once, incloudfall.rabbitmq_access. Each node's ruleset holds the 5671 rule for each allowed source, plus 4369/25672 from the other two nodes only. The listener check allows exactly those three listeners on the node's own IPv4One node is unchanged: its settings,
rabbitmq-env.confblock, inventory, requirements (a one-nodeclientAccesscertificate names its hostname and IPv4 only, as before) and extra vars render as before (tested), and every new task is skipped on it.Test notes
sdk/tests/test_s023_rabbitmq_cluster.py(requirements, the short hostname among each node's certificate names, rendered firewall through the nftables template, rendered cluster and rollout order, certificates signed by the project CA for server and client auth with the short hostname as a DNS name, PRECONDITION on a no-majority report, audit) andengine/tests/test_s023_rabbitmq_cluster_role.py(rendered settings, TLS options with noserver_name_indication,verify_funorcustomize_hostname_checkoverride, env block, hosts lines, listener check, majority filter, task order, and the majority tasks, the join condition and the health module run for real on localhost against the stand-inrabbitmq-diagnostics/rabbitmqctl)engine/tests/rabbitmq_node.py's stand-in now answerscluster_status,list_vhostsand JSONlist_queues, and reads the first non-option word as the commandruff check .,mypy, and the tests of the touched files (test_s023_*,test_pki,test_requirements,test_ansible_inventory,test_examples_render_unchanged,test_s019_mariadb_remote_access*,test_s012_elasticsearch_converge,test_domains_certificate).ansible-lint engine/ansiblewas run withoutANSIBLE_CONFIGand reported only role-path syntax-check failures; CI's run withANSIBLE_CONFIGdecides it. The full suite is left to CINot verified (needs three live Debian 13 hosts)
No live multi-host run was made, so these parts are not claimed:
h1with a confirm and consumed throughh3, and RabbitMQ really forming one cluster through classic-config peer discoveryinet_tls_dist's directory witherlon the host, which no test runsh2stopped,h2catching up, and publishing resuming when a node returns. The PRECONDITION stop and the audit reports are tested; the broker behaviour is notFixes #508
🤖 Generated with Claude Code