Feature status
Where TinyS3 stands against the S3 API and against MinIO, which implements roughly the same surface and is a useful yardstick for what "complete" looks like. Measured 2026-08-09.
Conformance
Every ceph/s3-tests suite, filtering AWS-irrelevant tests with -m "not fails_on_aws" (see S3 compatibility testing). test_s3 is measured on the 4-node cluster; the rest against a freshly started single node. test_iam runs against the IAM-enabled server (./gradlew :tiny-s3-iam:iamConformanceServer); the rest run without it, which is also how the effect of enforcement on the object API is measured:
| Suite | Passed | Failed | Skipped |
|---|---|---|---|
test_s3 | 596 | 3 | 4 |
test_headers | 33 | 15 | — |
test_s3select | 28 | 6 | — |
test_iam | 66 | 22 | — |
test_sts | — | — | — |
test_s3control | 0 | 1 | — |
test_sns | 0 | 5 | — |
STS AssumeRole and GetCallerIdentity are implemented (temporary session credentials from a role's sts:AssumeRole trust policy), but the ceph test_sts suite does not currently initialize against TinyS3 — every test errors in fixture setup (AccessDenied on a ListBuckets the fixture issues), so the row is left blank rather than reporting a number the harness never actually measured. The test_s3 row is the conformance baseline; its per-failure triage is maintained in docs/conformance-status.md, the single source of truth for that number. All suites above use the same -m "not fails_on_aws" filter.
No genuine AWS-compatibility failure remains in the object API. All 3 test_s3 failures are deliberate non-goals: test_list_buckets_paginated (a shared-account parallel-run artifact that passes serially) and two bucket-policy edge cases (AWS-vs-RGW owner semantics; a defective upstream test). The tenant tests, once counted as failures, now pass — tenant bucket namespacing is implemented (see docs/conformance-status.md). test_iam exercises accounts, user/role CRUD, groups, and inline plus managed policies (all implemented; see the IAM row below); the remaining failures are other IAM surface the suite covers. test_headers, test_s3control, and test_sns cover header edge cases, S3 Control, and SNS-target notifications that remain partial or unimplemented.
The six test_s3select failures are output formatting and error prose specific to Ceph; the values they compute are correct.
What works
Objects and buckets, multipart, versioning with delete markers, conditional requests, ACLs (canned, explicit grants, group and email grantees), bucket policies with condition keys, CORS, public access block, ownership controls, object lock and retention, tagging, lifecycle expiration, presigned URLs, SigV2 and SigV4, multi-user credentials, inline IAM user policies enforced against object operations, STS AssumeRole / GetCallerIdentity (temporary session credentials), S3 Select over CSV and JSON, server access logging, Reed-Solomon erasure coding with healing, TLS, bucket replication (durable, versioned, status-tracked — local or remote targets), webhook bucket notifications, per-bucket quotas, Prometheus metrics, a REST admin API and a web console.
Gaps against MinIO
Ordered by how much they matter for a server claiming S3 compatibility.
| Area | MinIO | TinyS3 |
|---|---|---|
| Encryption at rest | SSE-S3 and SSE-KMS via KES | SSE-S3, SSE-KMS and SSE-C all encrypt. The server key comes from tinys3.sse.masterKey; without it, requests asking for encryption are refused with 501 rather than silently stored in cleartext |
| IAM | Users, groups, policies, service accounts, LDAP and OpenID identity providers | Users, groups, and roles with full CRUD; inline and managed policies (customer and AWS-managed, with policy versions); access keys; attach/detach at user/group/role scope — all created and enforced through the AWS IAM API. No LDAP or OpenID identity-provider federation |
| STS | AssumeRole, WebIdentity, LDAP, certificate | AssumeRole and GetCallerIdentity (temporary session credentials via a role's sts:AssumeRole trust policy). No WebIdentity/OIDC, LDAP, or certificate federation |
| Replication | Bucket replication, site replication, bucket targets | Bucket replication (local or remote targets, durable queue, delete markers, replication status, resync, loop-safe two-way); no multi-site/sync/edge |
| Event notifications | Webhook, Kafka, AMQP, Redis, NATS, MySQL, Postgres, Elasticsearch, NSQ, MQTT | Webhook only — a NotificationConfiguration targeting an http(s):// URL gets async, best-effort event POSTs on object create/remove, with event-type and prefix/suffix filters. No queue/broker transports |
| Tiering | Lifecycle transitions to remote tiers, tier sweeper | Lifecycle expiration plus lifecycle transitions to a remote tier (admin-configured tier backends, managed through the /tiers admin API) |
| Batch operations | Replicate, expire, key rotation jobs | None |
| Quotas | Per-bucket quotas | Per-bucket hard byte quotas via the admin API and console; over-quota writes refused with 507 QuotaExceeded. Enforcement is per-node best-effort (short-TTL usage cache) |
| Storage classes | Multiple | STANDARD only |
| Metrics | Prometheus | Prometheus text format at /tinys3/metrics, JSON at /rest/api/v1/metrics |
| Distributed operation | Multi-node, pools, decommission | Multi-node erasure cluster at any node count with quorum + background self-healing. Server pools, pool decommission (drains a pool onto the survivors), rack awareness and per-bucket pool affinity (pin a bucket to a named pool at creation) are available in every deployment; clusters larger than 2 nodes set license.enterprise: true. |
| Select formats | CSV, JSON, Parquet | CSV, JSON |
Everything below IAM in that table is a scope decision rather than a compatibility defect — no S3 client breaks because site replication or tiering is absent. IAM and STS are different: they are separate AWS services, so their absence does not affect an S3 client either, but they are what stands between "an S3 server" and "a system you can hand to more than one team".
Server-side encryption
tinys3.sse.masterKey holds 32 base64 bytes. Configured, SSE-S3 and SSE-KMS encrypt the object with it and record the plaintext length so ranges and listings report the object's real size. Unconfigured, a request asking for encryption is refused with 501 NotImplemented, the way MinIO refuses when no KMS is present — the alternative, storing cleartext while answering AES256, tells a caller their data is protected when it is not.
Objects above MAX_SSEC_BUFFERED_SIZE are encrypted as the storage layer pulls them rather than in one piece, so a customer-encrypted upload is not limited by heap.
Losing the key loses the data. There is deliberately no fallback to a generated key stored beside the objects: it would make the feature appear to work while protecting nothing.
Known sharp edges
- Erasure overwrites keep two generations of shards. Each write lands in its own directory and metadata switches readers over, so a read during an overwrite sees one version or the other rather than a mixture. The generation just replaced is kept until the following write, because metadata is replicated across disks without being atomic and a reader can briefly still be handed the previous version. The cost is that an object occupies up to twice its size between writes, and a crash mid-write leaves an unreferenced directory behind.
- Upload buffering is bounded globally at a quarter of the heap by default (
tinys3.upload.memory.bytes); a deployment expecting very high concurrent-upload counts should set it explicitly. - Small-object latency sits around 0.62 ms/op against a ~0.09 ms filesystem floor, and throughput plateaus near 6,600 ops/s — roughly 5x of server-side headroom remaining.