Self-healing & data integrity

How TinyS3 keeps your objects intact on a distributed cluster: it detects missing and corrupted data, repairs it automatically, and reclaims the disk that overwrites leave behind — all in the background, without operator intervention. This page explains what the machinery does and what you can tune. For the step-by-step runbook (replacing a disk or node, reading the telemetry), see Operating a cluster; for the durability model it builds on, see Distributed mode.

The durability contract it protects

Every object is erasure-coded into dataShards + parityShards shards spread across the node×disk grid. Any dataShards shards reconstruct the object, so the cluster tolerates losing up to parityShards of them. That tolerance is only useful if lost shards come back — otherwise each failure permanently lowers the margin until an object drops below quorum. The three mechanisms below are what put the margin back.

Bit-rot detection

Every shard is written with a CRC32 frame. When a shard is read, the checksum is verified first; a rotted shard (a flipped bit on disk, a bad sector) fails the CRC and reads back as absent rather than as bad bytes. Corruption therefore folds into the same path as a missing shard — the reader gathers other shards to reconstruct, and healing rebuilds the rotted one.

For a stronger guarantee, each object's replicated metadata also stores a per-shard hash. On read, every gathered shard is verified against its expected hash, so a shard that was corrupted to a colliding CRC is still caught: the decoder discards it, pulls a parity shard instead, and decodes only from verified shards. The result of a read is always either the correct object or a clean below-quorum error — never silent garbage, and a read tolerates up to parityShards corrupt shards, not only missing ones.

• Verification can be turned off for latency-critical, trusted deployments with -Dtinys3.cluster.skipShardVerify=true — the CRC frame still catches accidental rot.

Read-repair (eager)

When a read has to reconstruct over a missing or corrupt shard, it has already done the expensive work — gathered a quorum and run the decode. Instead of throwing that away and waiting for the next background pass, TinyS3 repairs the shard on the spot: the owning node rebuilds and writes back the bad slot from the shards already in hand. A hot object with a lost shard is healthy again after its first read, not after the heal interval.

Repair work is bounded and rate-limited (a per-node queue that coalesces duplicates and sheds load under pressure), so a storm of degraded reads can't saturate disk I/O — the periodic healer is the backstop for anything the queue drops, and for cold objects that are never read.

Background healing

A healer runs on every node on a timer. Each node scans the cluster's objects, and for every shard it owns but is missing (a replaced disk, a node that was down, a rotted shard), it reconstructs the shard from a read-quorum of peers and writes it back. Nodes only heal their own slots, so heals never contend with each other, and reads keep serving from the surviving shards throughout.

This is what makes disk and node replacement hands-off: bring the replacement up at the same identity, and its healer re-derives the slots it owns and rebuilds every shard onto the fresh disks. TinyS3 heals shards within an erasure set — it never whole-disk-copies one disk onto another.

• Interval: -Dtinys3.cluster.healIntervalSeconds (default 60; 0 disables the background pass).
• Progress is exposed at GET /rest/api/v1/cluster/healing and on the console — watch it until a pass reports 0 unrecoverable before trusting a recovered cluster (see Operating a cluster).

Space reclamation (generation GC)

Each overwrite lands in a new generation (a fresh directory of shards) and the object's metadata is flipped to point at it. Keeping the previous generation briefly is deliberate — it lets a long-running reader that already opened the old version finish streaming while the overwrite installs the new one, so an overwrite never tears a read in progress.

Those superseded generations must be reclaimed or a frequently-rewritten key would accumulate disk without bound. The healer collects them as part of its pass, deleting any generation that is (1) no longer current and (2) older than a grace window — long enough that no read which could still be using it is in flight. This is why disk usage for a hot key may briefly exceed the live object size and then settle: the previous generation is waiting out its grace window.

• Grace window: -Dtinys3.cluster.gcGraceSeconds (default 300). Raise it if you serve very long downloads of frequently-overwritten objects; lower it to reclaim space faster.

What an operator actually tunes

Property	Default	Effect
tinys3.cluster.healIntervalSeconds	60	how often the background healer scans (0 = off)
tinys3.cluster.gcGraceSeconds	300	how long a superseded generation is kept before reclamation
tinys3.cluster.skipShardVerify	false	skip per-shard hash verification on read (CRC still applies)

All three are safe to leave at defaults. The one number to watch rather than set is unrecoverable objects in the healing telemetry — anything above zero means objects are below read quorum and a disk/node needs attention now.