File-system module

file-system/ (dev.totis.tinys3.filesystem) is TinyS3's storage engine — and it stands on its own. If you need fault-tolerant, self-healing, optionally encrypted file storage across a set of disks in a Java application, you can use it without any of the S3 layers.

Architecture: one facade, pluggable backends

TinyS3's storage stack is two layers, and the split is what keeps every deployment shape maintainable from one codebase:

1. S3FileOperations (in lib/) is the protocol-facing facade. Its single implementation, DefaultS3FileOperations, owns all S3 semantics exactly once — metadata sidecars, versioning and delete markers, multipart assembly, conditional requests, object locking, inline small objects. Nothing below this layer knows what a bucket policy or a version ID is.
2. FileSystem (this module) is the pluggable byte-storage interface underneath. A backend only implements file primitives — write, read (ranged), copy, delete, list — plus the conventions the facade relies on (temp-file staging with atomic promotion, and honoring the object-lock discipline).

The payoff: every storage backend gets the complete S3 feature set for free, because the hard semantics live above the interface. Writing objects as plain files without any erasure encoding is not a degraded mode — it is simply the DiskFileSystem backend, and it supports everything the erasure and cluster backends do. Conversely, a new backend never re-implements S3; it implements about a dozen file primitives.

The one obligation on a backend: it must keep the facade's assumptions true — a write is staged and atomically published (never partially visible), and concurrent mutations of the same key respect the object lock. Break those and the semantics above silently stop being safe.

The FileSystem interface

Everything is programmed against one interface:

public interface FileSystem {
  void writeFile(String path, byte[] data);
  void writeFileFromChannel(String path, ReadableByteChannel channel, long size);
  ReadableByteChannel readFileChannel(String path);
  ReadableByteChannel readFileChannel(String path, long position);   // ranged reads
  byte[] readTempFile(String path);
  void appendToFile(String path, byte[] data);
  void delete(String path);
  void copy(String source, String destination);
  FileEntry[] list(String path);
  long getSize(String path);
  FileTime getLastModifiedTime(String path);
  boolean exists(String path);
  void createDirectory(String path);
  // ... temp-file staging, directory listing, path resolution
}

Four implementations ship:

Implementation	Module	Backing
DiskFileSystem	file-system	A single local directory — plain files, no encoding, zero-copy reads via localFilePath. What .withStorageDir(...) and storage.mode: simple use
ErasureCodedFileSystem	file-system	Reed-Solomon shards striped across many disks, with healing and at-rest encryption
ClusterFileSystem	distributed	Shards quorum-replicated across the nodes of a cluster
PooledFileSystem	distributed	Composes several of the above into server pools (expansion, decommission)

They compose: a multi-pool cluster is a PooledFileSystem over ClusterFileSystems, each of which stripes locally like the erasure backend — while DefaultS3FileOperations above them is the same class in every case.

PooledFileSystem routing: an existing object always follows the pool that already holds it (it is never re-routed). A brand-new object goes to its bucket's pinned pool if the bucket was pinned at creation (bucket pool affinity), otherwise to the default pool (the first active pool). A bucket's objects are never striped across pools. While a pool is suspended (decommissioning), writes to a bucket still pinned to it are refused rather than spilling to another pool — until decommission migrates the bucket and rewrites its pin to the target pool, after which new writes land there.

Erasure-coded storage

import dev.totis.tinys3.filesystem.StorageConfig;
import dev.totis.tinys3.filesystem.metadata.ErasureConfig;
import dev.totis.tinys3.filesystem.erasure.ErasureCodedFileSystem;

StorageConfig config = StorageConfig.create(
    List.of("/disk1", "/disk2", "/disk3", "/disk4", "/disk5", "/disk6"),
    new ErasureConfig(
        4,      // data shards
        2,      // parity shards — survives this many disk losses
        4096)); // shard size multiple

FileSystem fs = new ErasureCodedFileSystem(config);

fs.writeFile("bucket/report.pdf", bytes);
byte[] back = fs.readTempFile("bucket/report.pdf");   // still works with 2 disks gone

How it stores data:

• Each write is split into 4 data + 2 parity shards (configurable) using the Backblaze Reed-Solomon codec and distributed across the disks. Any 4 of the 6 shards reconstruct the file.
• Objects at or below inlineThreshold (default 128 KB) skip striping and are stored inline — small-object performance without the shard overhead.
• Large files are processed in chunks (chunkSize, default 16 MB) so memory stays flat.
• Writes are staged per-generation and atomically promoted; concurrent writers to the same key are serialized with striped locks, and deletes are tombstoned so a half-failed delete never corrupts a live object.

StorageConfig.singleDisk(path) exists for tests; ErasureConfig.defaults() gives 4+2.

At-rest encryption & integrity

FileSystem fs = new ErasureCodedFileSystem(config, diskEncryptionKeyBase64);

With a key (32 bytes, base64), every shard is encrypted with AES-256-CTR before it touches a disk, and sealed with a HighwayHash keyed MAC (ShardIntegrity). Consequences:

• A lost or stolen disk reveals nothing.
• A tampered or bit-rotted shard fails its MAC and is treated as missing — read repair reconstructs it from parity instead of returning corrupt data.

New writes seal each shard with per-block MACs over 128 KiB blocks (ShardInfo.blockChecksums) rather than a single whole-shard MAC. This lets a ranged read fetch and verify only the blocks it overlaps instead of the whole shard, and localizes bitrot to a block. Shards written by older versions (and healed shards) keep the single whole-shard MAC and still verify.

Without a key, shards still carry the integrity MAC (default key), so corruption detection works either way; only confidentiality requires the key.

Self-healing

The healing package scans for missing or corrupted shards and rebuilds them from parity:

import dev.totis.tinys3.filesystem.healing.*;

HealingConfig healingConfig = HealingConfig.builder()
    .enabled(true)
    .scanInterval(Duration.ofHours(1))
    .maxConcurrentHeals(4)
    .healOnStartup(false)
    .build();                     // or HealingConfig.defaults()

HealingScheduler scheduler = new HealingScheduler(
    objectHealer,                 // ObjectHealer — DefaultObjectHealer over the erasure fs
    () -> listOfObjectKeys(),     // Supplier<List<String>> — what to scan
    healingConfig);
scheduler.start();
// scheduler.stop() on shutdown

A scan walks every object, verifies each shard's presence and MAC, and re-encodes anything recoverable. Objects with more shards lost than parity can cover are reported as unrecoverable rather than silently missing.

Using it under TinyS3

This is exactly what the server does in erasure mode — pass the file system to the builder:

S3Server server = new S3Server.Builder()
    .withPort(8000)
    .withCredentials(credentials)
    .withFileSystem(new ErasureCodedFileSystem(config))
    .build();

See Embedding TinyS3 for the full builder reference and Storage modes for the operational view.

Design notes

• Placement — shards are spread deterministically across disks (placement/), so a disk's contents are reconstructible knowing only the config.
• Metadata — each object records its erasure config with the data, so dataShards/ parityShards can change over time without breaking old objects.
• Buffering — a shared BufferPool recycles direct buffers on hot paths.
• Retry — disk operations run under a RetryPolicy so a transient I/O error is not an immediate object failure.
• Reserved object-key suffixes — the facade stores each object's metadata and version data alongside the object in the key space (<key>.meta.json, <key>.versions.json, <key>.v-<id>). Object keys whose final path segment ends in one of these (matched case-insensitively) are therefore rejected with 400 InvalidURI, so a client key can never resolve to another object's metadata or object-lock record. This is a deliberate divergence from AWS S3, which permits such keys; it applies at the request boundary and does not affect any already-stored object.