brazier/cluster

Cluster formation: node discovery, reconnection and the peer check.

Brazier never turns distribution on itself: start the node with a name (--name / -sname), a cookie and, in production, TLS distribution (see docs/CLUSTERING.md). start then runs the cluster manager, which:

discovers peers with Static (a fixed list, retried with exponential backoff and jitter) or Dns (A/AAAA lookups every poll_ms), or only watches connections with NoDiscovery. After a netsplit Static retries and every Dns poll reconnects; with NoDiscovery nothing does, and OTP’s global (prevent_overlapping_partitions, kept at its default) may have cut connections the network never lost (docs/CLUSTERING.md, “Netsplits”);
logs node up/down and emits [brazier, cluster, node, up | down] and [brazier, cluster, connect, failed];
checks every connected peer: nodes exchange HMAC-SHA256(secret_key, “brazier-cluster-v1”) and a protocol version, where secret_key is the application’s own secret key (one key per deployment; the purpose label keeps the MAC apart from token signatures). A peer whose key differs, or that does not answer within peer_check_timeout, is rejected: it is not a member, the error is logged and reported with [brazier, cluster, peer, rejected], and it stays connected (disconnecting it would make OTP’s global partition the verified members; see docs/CLUSTERING.md). This catches two deployments sharing a cookie by mistake; it is not a defence against a node that holds the cookie, which can run any code here.
keeps the cluster’s membership: PubSub, replicated stores, presence and session revocation exchange messages only with peers that passed the check (members), and drop what any other node sends. Without a running manager no remote node is a member.

In production (BRAZIER_ENV=prod) start refuses plain TCP distribution, cookies shorter than 32 bytes and TLS distribution that does not verify peers (server verify_peer and fail_if_no_peer_cert, client verify_peer) unless the config carries allow_insecure_distribution(reason).

Discovery follows libcluster’s Epmd and DNSPoll strategies.

Types

ClusterConfig

opaque </>

pub opaque type ClusterConfig

ClusterError

</>

pub type ClusterError {
  DistributionNotStarted
  InsecureDistribution(reason: String)
  InvalidConfig(reason: String)
  NotStarted
  AlreadyStarted
  StartFailed(reason: String)
  StopFailed(reason: String)
}

Constructors

```
DistributionNotStarted
```
This node runs without distribution (no --name / -sname).
```
InsecureDistribution(reason: String)
```
Production refuses this distribution setup; see the reason.
```
InvalidConfig(reason: String)
```
```
NotStarted
```
The cluster manager is not running on this node.
```
AlreadyStarted
```
The cluster manager is already running on this node.
```
StartFailed(reason: String)
```
```
StopFailed(reason: String)
```

Discovery

</>

How the cluster manager finds peers.

pub type Discovery {
  Static(nodes: List(String))
  Dns(query: String, basename: String, poll_ms: Int)
  NoDiscovery
}

Constructors

```
Static(nodes: List(String))
```
Connect to each name@host node, retrying failed and lost connections with backoff (connect_backoff). This node may appear in the list.
```
Dns(query: String, basename: String, poll_ms: Int)
```
Every poll_ms, resolve query (A and AAAA records) and connect to basename@<address> for each address not yet connected. Nodes that leave the DNS answer are not disconnected: a node that is up stays connected until distribution reports it down.
```
NoDiscovery
```
Connect to nothing; only report and check nodes that connect. Nothing is ever reconnected: a member lost to a netsplit, or dropped by OTP’s global to close a partial partition, stays apart until other code connects it again (as libcluster’s Epmd strategy without timeout). Use Static or Dns unless something else forms the cluster.

Resolver

</>

Resolve a DNS name to address strings ("10.0.0.7", "fd00::7").

pub type Resolver =
  fn(String) -> Result(List(String), String)

Values

allow_insecure_distribution

</>

pub fn allow_insecure_distribution(
  c: ClusterConfig,
  reason: String,
) -> ClusterConfig

Start in production without TLS distribution or with a short cookie. The reason is logged as a warning on every start. Only for networks where every host that can reach the distribution ports is trusted.

connect_backoff

</>

pub fn connect_backoff(
  c: ClusterConfig,
  initial_ms initial_ms: Int,
  max_ms max_ms: Int,
) -> ClusterConfig

Retry a lost or failed static connection after a delay that starts at initial_ms and doubles to max_ms, taken uniformly from the upper half of the current delay so restarted nodes do not retry in lockstep.

describe

</>

pub fn describe(e: ClusterError) -> String

dns_resolver

</>

pub fn dns_resolver(
  c: ClusterConfig,
  resolve: fn(String) -> Result(List(String), String),
) -> ClusterConfig

Resolve Dns queries with resolve instead of the system resolver.

dns_timeout

</>

pub fn dns_timeout(
  c: ClusterConfig,
  timeout_ms: Int,
) -> ClusterConfig

How long a DNS lookup may take before it counts as failed.

max_discovered_nodes

</>

pub fn max_discovered_nodes(
  c: ClusterConfig,
  max: Int,
) -> ClusterConfig

The most node names DNS discovery may make over the life of the VM. Node names are atoms, which are never freed, so a DNS answer that keeps changing would otherwise grow the atom table without bound. Past the limit new addresses are not connected to (error log and [brazier, cluster, connect, failed]); addresses already named still are. The count survives manager restarts.

max_dns_nodes

</>

pub fn max_dns_nodes(c: ClusterConfig, max: Int) -> ClusterConfig

The most addresses one DNS answer may hold. A larger answer is refused as a whole (logged, [brazier, cluster, connect, failed]).

members

</>

pub fn members() -> Result(List(node.Node), ClusterError)

The peers that passed the peer check and are connected.

min_cookie_bytes

</>

pub const min_cookie_bytes: Int

Production cookies must be at least this long (a 32-byte random cookie is not guessable; short ones like “secret” are).

new

</>

pub fn new(discovery: Discovery) -> ClusterConfig

A cluster config with discovery. The peer check uses the application’s secret key, which start and supervised take (the application passes its own): every node of one deployment must use the same one, and the peer check rejects nodes that do not.

Defaults: reconnect backoff 1s doubling to 30s, peer check timeout 5s, the system DNS resolver with a 10s timeout, at most 256 nodes per DNS answer and 16384 DNS node names over the VM’s life.

peer_check_timeout

</>

pub fn peer_check_timeout(
  c: ClusterConfig,
  timeout_ms: Int,
) -> ClusterConfig

How long a newly connected peer has to pass the peer check.

resolve_dns

</>

pub fn resolve_dns(query: String) -> Result(List(String), String)

The system resolver: A and AAAA records of query. An error only when both lookups fail.

start

</>

pub fn start(
  c: ClusterConfig,
  secret_key secret_key: String,
) -> Result(Nil, ClusterError)

Check the config and this node’s distribution, then start the cluster manager, not linked to the caller. secret_key is the application’s secret key (at least config.min_secret_key_bytes).

stop

</>

pub fn stop() -> Result(Nil, ClusterError)

Stop the cluster manager started with start. Connections stay up; they are no longer reconnected or checked.

supervised

</>

pub fn supervised(
  c: ClusterConfig,
  secret_key secret_key: String,
) -> Result(supervision.ChildSpecification(Nil), ClusterError)

A child specification that starts the cluster manager under a supervisor. The checks run when the specification is built.