System Architecture & Module Boundaries ​

TinyS3 is designed as a modular, high-performance, strictly layered object storage engine. Each subsystem has clear ownership, zero upward dependencies, and well-defined contracts.

Architectural Diagram ​

Layer Responsibilities ​

1. User Interface Layer ​

  • console/: Embedded React and TypeScript single-page application. Manages buckets, IAM users, access keys, policies, replication jobs, and server metrics over standard REST APIs (/api/v1/...).

2. Application Entrypoint ​

  • app (:tiny-s3-app): The single server binary entrypoint (dev.totis.tinys3.Application). One edition, full feature set — standalone, erasure, and distributed cluster. It builds a single fat jar (tinys3-<version>-all.jar) and a single jib image. Server pools live in core/pools (:tiny-s3-pools) and are put on the runtime classpath via ServiceLoader (runtimeOnly), so pools and rack awareness are available to every deployment — no separate edition jar. See the single-edition model.
  • core/pools (:tiny-s3-pools; package dev.totis.tinys3.pools): Multi-pool topologies, rack/zone fault-domain placement, and pool decommission (ServerPoolCluster, ServerPoolClusterFactory, PooledFileSystem, PoolDecommissioner, PoolSelector).

3. Orchestration & Assembly ​

  • core/bootstrap (:tiny-s3-bootstrap): Orchestrates the server lifecycle. Parses command-line arguments, boots storage drivers, prepares encryption keys, and wires all internal services together.

4. API & Control Plane Services ​

  • core/rest (:tiny-s3-rest): Non-S3 REST management endpoints (/api/v1/pools, /api/v1/metrics, /api/v1/jobs, /api/v1/heal, etc.).
  • core/iam (:tiny-s3-iam): AWS-compatible IAM action router (CreateUser, AssumeRole, PutUserPolicy), role assumption, STS tokens, and resource policy evaluation.
  • core/replication (:tiny-s3-replication): Background async replication engine, scheduled transfer jobs, real-time chunk streaming with bandwidth counters (CountingReadableByteChannel).

5. S3 Protocol Core ​

  • core/lib (:tiny-s3-lib): The core S3 engine. Implements AWS SigV4 / SigV2 signature authentication, request canonicalization, bucket and object lifecycle, multipart upload assemblies, and bucket policy enforcement.

6. Clustering & Placement ​

  • core/distributed (:tiny-s3-distributed): Peer-to-peer gRPC mesh across nodes, distributed quorum leases, cluster tokens, and node health registries.
  • core/placement (:tiny-s3-placement): Deterministic Rendezvous (HRW) and Maglev hashing across drives, nodes, and fault domains.

7. Storage Engine ​

  • core/file-system (:tiny-s3-file-system): Pure I/O abstractions (FileSystem, DiskStorage), striped file locking (StripedLock), erasure chunking, background healing scheduler, and bitrot detection. Owns its own storage-level metadata (ErasureObjectMeta, ErasureMetaCodec, MetadataVersioning) with zero dependency on higher-level S3 protocol types. Supports constructor-injected metadata serialization (ErasureMetaCodec).
  • database/document-store (:tiny-s3-document-store): Key-value/document store with AES-GCM encryption for credentials, endpoints, and cluster states.
  • core/select (:tiny-s3-select): Streaming SQL query parser and projection engine for S3 Select (CSV, JSON; Parquet not yet supported).
  • core/JavaReedSolomon (:tiny-s3-javareedsolomon): SIMD vectorized Galois-Field arithmetic (VectorizedCodingLoop) for maximum-throughput erasure coding.

8. Foundational Subsystems ​

  • core/metadata (:tiny-s3-metadata): Schema models (ObjectMetadata, VersionManifest, IamIdentity, IamState, CredentialRecord, UserRecord) and the unified serialization boundary (MetadataCodec).
  • core/http (:tiny-s3-http): High-performance Netty HTTP/1.1 non-blocking transport engine, async streaming, and TLS configuration.
  • core/telemetry (:tiny-s3-telemetry): Prometheus metrics exporters, latency histograms, health probes, and optional OpenTelemetry (OTLP) request tracing.
  • core/kms (:tiny-s3-kms): Pluggable envelope-encryption SPI (KmsProvider) for SSE — a random per-object data key wrapped by a provider. Ships a local provider (wraps under the SSE master key) and a bitwarden provider (root key held in Bitwarden Secrets Manager); the storage core depends only on the SPI. See Encryption.

9. Integrations (Optional Plugins) ​

  • integrations/plugin-api (:tiny-s3-plugin-api): The ServerPlugin SPI and PluginManager — the extension point that server integrations hook into, discovered via ServiceLoader.
  • integrations/cloudflared (:tiny-s3-cloudflared): Optional plugin that exposes the server through a Cloudflare Tunnel via a native libcloudflared library driven with the Java Project Panama (FFM) API. Opt-in only; see Public endpoint.
  • integrations/mcp (:tiny-s3-mcp): Optional plugin that exposes TinyS3 to AI clients over the Model Context Protocol. Serves a stateless MCP server on the main server port under /mcp (registered through the HttpRouteRegistrar seam, the same path-dispatch the console uses — no second listener), authenticates clients with an OAuth 2.0 device flow, and executes each tool as a signed S3 call under the user's own credentials — so the server's normal authentication and authorization apply and the MCP layer never bypasses S3 access control. Opt-in with --mcp (--mcp-allow-write/-delete/-admin for mutating tools).

Single edition and the node-count license ​

TinyS3 ships as one edition with the full feature set. There is no community-vs-enterprise jar split: one build (:tiny-s3-app:fatJar → tinys3-<version>-all.jar), one jib image, one entry point dev.totis.tinys3.Application. Server pools and rack-aware placement are available to every deployment — they are wired in from core/pools via ServiceLoader and are not gated behind an edition.

The only boundary is cluster size, not features. A cluster of 2 nodes or fewer needs no license flag. A cluster of more than 2 nodes must set license.enterprise: true in the config (or the TINY_S3_LICENSE_ENTERPRISE environment variable) to acknowledge the commercial terms in LICENSE.enterprise. This is an honor-system config flag — not a cryptographic license. Setting it flips the reported edition label to enterprise and logs a notice that the cluster operates under the commercial terms; without it, a

2-node cluster refuses to boot. The gate lives in ClusterBootstrap.enforceEditionLimits; the label is surfaced through EditionCapabilities (core/lib, dev.totis.tinys3.spi) on GET /server/info and in the console.

Architectural Principles & Invariants ​

  1. Directed Acyclic compile graph, downward-leaning layers:

    • The compile-time dependency graph is acyclic. Modules lean downwards: Applications $\rightarrow$ Bootstrap $\rightarrow$ Services $\rightarrow$ S3 Core $\rightarrow$ Storage $\rightarrow$ Foundation.
    • Documented exceptions, none of which affect the compile DAG: lib has a test-scope dependency on iam (the conformance server wires IAM), so the module graph is acyclic only when test configurations are excluded; pools (core/pools) depends on bootstrap because it is loaded there as a ServiceLoader plugin; document-store depends on its sibling file-system within the Storage layer.

  2. S3 semantics stay in the facade, not the backends:

    • S3 request semantics (auth, ACLs, multipart, versioning) live in lib; the FileSystem backends move bytes and shards only. Naming conventions the backends must filter from listings (version files) are owned by the facade and injected via HiddenPathPolicy, so no backend re-encodes them.
    • The facade is split by S3 domain (see the Storage Facade box above): S3FileOperations is an empty union of ten per-domain role interfaces (ObjectOperations, MultipartOperations, VersioningOperations, BucketOperations, BucketConfigStore, ObjectAttributeStore, QuotaOperations, TieringOperations, IntegrityOperations, StorageWiring), so callers can depend on the narrow contract they use. DefaultS3FileOperations implements that union and composes focused collaborators rather than being a god-object — MetadataStore (object-metadata I/O over the shared ObjectMetadataCache), TieringGateway, IntegrityScanner, CustomDomainRegistry — all sharing one object lock and cache. The multipart-store and versioning read-modify-write spine deliberately stays in the facade, since the object path and the versioning path are the same path.
    • The HTTP transport (:tiny-s3-http) is protocol-agnostic: S3-specific error rendering and presigned-URL detection are injected from lib (ErrorBodyRenderer, ExpectContinuePolicy), not hardcoded in the engine.

  3. Pluggable & Decoupled Storage:

    • The :tiny-s3-file-system module is an independent storage engine that can be embedded into other applications.
    • Erasure coding metadata persistence supports constructor injection of ErasureMetaCodec, allowing custom serialization formats (JSON, Protobuf, CBOR, etc.).

  4. Unified Metadata Serialization Boundary:

    • Persisted state routes through MetadataCodec. The generic document store uses an S3-unaware codec variant (MetadataCodec.forDocuments()), so credential/job records carry no object-metadata schema knowledge.
    • :tiny-s3-metadata is a domain-model foundation: it holds the record schemas (including S3 and AWS/IAM shapes such as ARNs and policy documents) but performs no protocol logic. These are domain data, not protocol behavior, and higher layers depend on them, never the reverse.

TinyS3 is AGPLv3 licensed. Cloudflare and Cloudflare Workers are trademarks of Cloudflare, Inc. Amazon S3 and AWS are trademarks of Amazon.com, Inc. or its affiliates. MinIO is a trademark of MinIO, Inc. They are named only to describe compatibility; totis.ai is not affiliated with, endorsed by, or sponsored by any of them.

totis.ai