Distributed mode (multi-node cluster)

TinyS3 can run as a cluster of nodes that together serve one logical S3 endpoint — any node answers any request. Object data is erasure-coded and spread across the nodes with quorum durability, so the cluster keeps serving reads and writes while nodes are down.

TinyS3 ships as a single edition with the full feature set — server pools (including per-bucket pool affinity) and rack-aware placement are available to every deployment. There is no per-feature edition split. The only boundary is cluster size: a cluster of 2 nodes or fewer needs nothing, while a cluster of more than 2 nodes must set license.enterprise: true (or TINY_S3_LICENSE_ENTERPRISE=true) to acknowledge the commercial terms in LICENSE.enterprise, otherwise it refuses to boot.

/ This is a young feature. The object data path, listing, quorum coordination, and background self-healing work and are stress-tested; multipart-across-nodes and versioning are still in progress. /

How it works

• Grid placement — a large object is split into stripes of dataShards × 1 MiB; each stripe is erasure-coded into dataShards + parityShards shards and spread deterministically across a grid of nodes × disksPerNode slots, so losing one node costs the fewest shards of any stripe. Every node computes the same placement from config alone; there is no lookup service. A small object (below the inline threshold) is stored whole inside the replicated metadata instead of being striped.
• Write quorum — a write is durable once dataShards + 1 shard acks are collected; short of that it is rolled back, so a half-written object is never exposed.
• Read quorum — any dataShards shards reconstruct the object.
• Locks — cluster-wide serialization uses dsync-style majority leases (a lock needs a majority of nodes to grant it; a crashed holder's lease expires on its own).
• Transport — nodes talk over gRPC, authenticated with short-lived tokens signed by the root secret key and, optionally, mutual TLS (see Securing the cluster). The local node is called in-process; only cross-node hops use the network.
• Isolation — the distributed code is a separate module (tiny-s3-distributed) that plugs into the server through the ordinary FileSystem seam. The single-node storage path is untouched, so the two cannot break each other.

Configuration

Every node runs the same binary with a config that lists the whole cluster and points self at this node. The node's local disks and the erasure parameters come from storage.erasure.

server:
  port: 8000                 # this node's S3 port
auth:
  accessKey: "root"
  secretKey: "rootsecret"
  region: "us-east-1"
cluster:
  name: "prod-1"
  nodes:                     # gRPC address of every node, same list & order on all of them
    - "node1:9000"
    - "node2:9000"
    - "node3:9000"
    - "node4:9000"
  self: "node1:9000"         # which node this process is
  # No cluster secret needed: inter-node auth uses the root secret key (auth.secretKey) below.
  # Set `secret:` (or TINY_S3_CLUSTER_SECRET) only to rotate it independently of the root credential.
  tls:                       # optional mutual TLS for inter-node traffic (recommended in production)
    enabled: true
    certFile: "/etc/tinys3/cluster/node.crt"
    keyFile:  "/etc/tinys3/cluster/node.key"
    trustFile: "/etc/tinys3/cluster/ca.crt"   # CA (or bundle of peer certs) that signs every node
storage:
  mode: erasure
  erasure:
    dataShards: 8
    parityShards: 4          # tolerates the parity budget of shard/node loss
    disks:                   # THIS node's local disks (disksPerNode is their count)
      - "/data/disk1"
      - "/data/disk2"
      - "/data/disk3"

Constraint: nodes × disksPerNode ≥ dataShards + parityShards (enough slots to place a stripe). With 8+4 and 4 nodes, use 3 disks per node (12 slots); one node then holds 3 of the 12 shards, so a single node loss leaves 9 shards — still above both quorums.

Securing the cluster

The inter-node gRPC channel (shard read/write/delete, distributed locks) is trusted infrastructure — it is not the S3 API and carries no per-user authorization. Two controls protect it.

Inter-node authentication (automatic)

Inter-node auth uses the root secret key (auth.secretKey) — already shared identically by every node, so there is nothing extra to configure using the node's own root credentials. Every RPC carries a short-lived HS256 token signed with that secret and a 60-second expiry: the receiver verifies the signature and expiry, so the raw secret never travels on the wire and a captured token cannot be replayed once it expires.

To rotate the inter-node secret independently of the root credential, set cluster.secret (or the TINY_S3_CLUSTER_SECRET environment variable) to any shared value — it then becomes the token signing key on every node. A node refuses to start if it has neither a root secret key nor an explicit cluster secret.

Mutual TLS (recommended in production)

The secret alone travels in cleartext on a plaintext channel, so anyone who can observe inter-node traffic could capture it and impersonate a peer. Enable cluster.tls to encrypt the channel and authenticate peers by certificate — only a node holding a key trusted by trustFile can join.

Generate a CA and a per-node cert (or one shared cert whose SAN covers every node's hostname):

# A CA, once:
openssl req -x509 -newkey rsa:4096 -days 3650 -nodes \
  -keyout ca.key -out ca.crt -subj "/CN=tinys3-cluster-ca"

# A cert per node, signed by the CA. The SAN must match the host in `cluster.nodes`:
openssl req -newkey rsa:2048 -nodes -keyout node1.key -out node1.csr \
  -subj "/CN=node1" -addext "subjectAltName=DNS:node1"
openssl x509 -req -in node1.csr -CA ca.crt -CAkey ca.key -CAcreateserial \
  -days 825 -out node1.crt -copy_extensions copy

Point each node's cluster.tls at its own node.crt/node.key and the shared ca.crt as trustFile. Because TLS validates the certificate against the peer hostname, cluster.nodes must use names the certificates cover (not bare IPs, unless the SAN is an IP: entry).

Configuration reference

Setting	Env var	Default	Purpose
cluster.secret	TINY_S3_CLUSTER_SECRET	auth.secretKey	signing key for the inter-node tokens; defaults to the root secret key
cluster.tls.enabled	—	false	turn on mutual TLS for inter-node traffic
cluster.tls.certFile / keyFile	—	—	this node's PEM certificate and private key
cluster.tls.trustFile	—	—	CA (or peer-cert bundle) trusted to sign the other nodes

Monitoring

Every node can report the whole deployment, so a monitor only needs to reach one:

• GET /rest/api/v1/cluster (root-only admin API) returns the topology, each node's reachability, and whether read/write quorum is currently met:

  { "name": "prod", "selfIndex": 0, "writeQuorum": 3, "readQuorum": 2,
    "nodesTotal": 4, "nodesUp": 3, "hasWriteQuorum": true, "hasReadQuorum": true,
    "nodes": [ { "index": 0, "address": "host0:9101", "self": true, "reachable": true }, … ] }

• The web console has a Cluster tab (/console/cluster) that polls that endpoint and shows a live node grid, quorum banners, and how many nodes the deployment can currently lose — refreshing every few seconds. A node going down turns its card red and flips the quorum banner if a threshold is crossed. The tab appears only when the server is in cluster mode (the endpoint returns nothing otherwise). Open the console on any node (http://<any-node>:<port>/console/) and sign in with a root key.

Running

Point a client (or a load balancer) at any node's S3 port; they are interchangeable. Failure behavior for a 4 × 3, 8+4 cluster:

Failure	Reads	Writes
1 disk / 1 node	continue	continue (quorum met)
2 nodes	continue if ≥ dataShards shards reachable	fail quorum → 503
minority partition	—	no lock majority, no write quorum → no split brain

Self-healing

Erasure coding keeps the cluster serving through node loss (reads reconstruct from the surviving shards); healing is what makes it durable again afterwards — it puts the missing shards back so a second failure doesn't cross the quorum line. TinyS3 heals shards within an erasure set, not whole nodes by replication: it never copies a node's disk somewhere else. A missing shard is reconstructed from a read-quorum of the surviving shards and written back to the slot that owns it.

Each node runs its own healer on a background schedule. Because placement is deterministic, every shard has exactly one owning node, so each healer repairs only the slots it owns — no locking or coordination between nodes. A pass:

1. builds the cluster-wide object list (the union of every reachable node's listing, so even a node with a blank disk discovers what it should hold);
2. for each slot this node owns, restores the replicated metadata if missing, then rebuilds any missing stripe shard by reading dataShards surviving shards and erasure-decoding the gap;
3. leaves shards that are already present untouched.

The same path covers both recovery cases with no special handling:

• Node returns with its disks — only the shards it missed while down are rebuilt; everything else is already there, so the pass is cheap.
• Node replaced with a blank disk — it re-derives its whole shard set from the survivors, one object at a time, while reads keep working from the other nodes.

Healing is incremental and needs no cluster restart. It can only rebuild while a read quorum of shards survives; below that the shard is left for a later pass rather than fabricated.

Setting	Default	Meaning
-Dtinys3.cluster.healIntervalSeconds	60	seconds between background heal passes per node; 0 disables the healer

Bit-rot is caught too: each shard and metadata copy is stored with a CRC32, so a rotted shard fails its checksum and reads back as absent — the quorum reader reconstructs the object from the other shards and the healer rebuilds the bad one on its next pass. A read never decodes over corrupt bytes.

Repair is also eager: a read that reconstructs over a corrupt/missing shard enqueues a targeted rebuild at read time (a bounded, deduplicated per-node read-repair queue), so hot objects are fixed on the first degraded read rather than waiting for the next heal pass. Stale generations left by a missed overwrite are harmless (ignored on read) and are garbage-collected by the healer after a grace window.