TinyS3 Metadata Format Specification

Version:    1.2.0
Status:     Stable
Created:    2025-01-26

1. Introduction

This document specifies the metadata formats used by TinyS3 for persisting object, bucket, versioning, multipart, and erasure-coding state. It covers three distinct storage backends:

• Standard storage (single node, filesystem or in-memory).
• Erasure-coded storage (single node, Reed-Solomon across local disks).
• Distributed/cluster storage (shards spread across cluster nodes and disks).

These backends do NOT share one on-disk metadata format. The standard backend serializes JSON via a shared codec; the single-node erasure backend serializes a compact binary record (with transparent read of legacy JSON); the distributed backend uses a compact newline-delimited record. Each is specified separately below.

1.1 Terminology

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119.

1.2 Notation

• <path> - Variable path component
• integer - 64-bit signed integer
• string - UTF-8 encoded string
• boolean - true or false
• object - JSON object
• array - JSON array
• null - JSON null value

2. Format Versioning

2.1 Version Field

JSON metadata files (object, bucket, and version manifest) are serialized through a shared codec that injects a formatVersion field at the root of the JSON document. The value currently emitted is "1.0.0".

Single-node erasure metadata (Section 6) is serialized through its own storage-layer codec, BinaryErasureMetaCodec. This is a compact binary format (a 4-byte magic TS3M + a 1-byte structural version, currently 1), not JSON, and carries no formatVersion field. It is backward-compatible on read: a record that does not begin with the binary magic is decoded as legacy JSON (JsonErasureMetaCodec, whose files start with { and carry formatVersion "1.0.0"), so metadata written by older versions keeps reading while all new writes are binary. The erasure record versions independently of the shared codec — it is deliberately decoupled from the S3/object schema.

formatVersion = MAJOR "." MINOR "." PATCH

• MAJOR: Incompatible schema changes
• MINOR: Backward-compatible additions
• PATCH: Documentation or bug fixes

The distributed/cluster metadata record (Section 9) is NOT JSON and carries no formatVersion field; it versions itself structurally by appending optional trailing fields (Section 9.1).

2.2 Unknown Fields

All JSON metadata records are deserialized with unknown-property tolerance: a field present in the file but absent from the current schema is ignored on read. Because records are re-serialized from their in-memory form, unknown fields are NOT round-tripped and MUST NOT be relied upon to survive a read-modify-write.

2.3 Compatibility

Implementations SHOULD tolerate absent optional fields when reading older files. New fields SHOULD be added as OPTIONAL so that older readers ignore them and newer readers default them.

3. Object Metadata (Standard Storage)

Location: <object-path>.meta.json

Verified against dev.totis.tinys3.metadata.schema.ObjectMetadata and dev.totis.tinys3.DefaultS3FileOperations.

3.1 On-Disk Layout

The metadata file is written as a single line of JSON. When the object is stored inline (see Section 3.4), the raw object bytes follow immediately after a single newline character, avoiding a base64 round-trip through the JSON codec. Readers MUST split on the first newline: everything before it is JSON metadata, everything after it is the inline object payload.

3.2 Schema

Field	Type	Required	Default	Description
formatVersion	string	REQUIRED	"1.0.0"	Injected by the codec; see Section 2.1
contentType	string	OPTIONAL	"application/octet-stream"	MIME type of object content
userMetadata	object	OPTIONAL	{}	User-defined x-amz-meta-* key-value pairs
tags	object	OPTIONAL	{}	Object tagging key-value pairs
acl	string	OPTIONAL	"PRIVATE"	Canned ACL enum name; see Section 3.3
sseAlgorithm	string	OPTIONAL	null	Server-side encryption algorithm (AES256, aws:kms)
sseKmsKeyId	string	OPTIONAL	null	KMS key ID for SSE-KMS encryption
ssecAlgorithm	string	OPTIONAL	null	Customer-provided encryption algorithm
ssecKeyMD5	string	OPTIONAL	null	Base64 MD5 of customer-provided encryption key
ssecOriginalSize	integer	OPTIONAL	0	Plaintext size before SSE-C encryption
legalHold	boolean	OPTIONAL	false	Legal hold status for compliance
retentionMode	string	OPTIONAL	null	Object lock retention mode; see Section 3.4
retentionRetainUntilDate	string	OPTIONAL	null	ISO-8601 date until which object is locked
cacheControl	string	OPTIONAL	null	Cache-Control HTTP header value
contentDisposition	string	OPTIONAL	null	Content-Disposition HTTP header value
contentEncoding	string	OPTIONAL	null	Content-Encoding HTTP header value
expires	string	OPTIONAL	null	Expires HTTP header value
ownerId	string	OPTIONAL	null	Owner identifier
grantFullControl	string	OPTIONAL	null	Grantee ID for full control permission
etag	string	OPTIONAL	null	Entity tag; MD5 hash or multipart composite
partsCount	integer	OPTIONAL	0	Number of parts; > 0 marks a multipart-assembled object
partSizes	array	OPTIONAL	null	Array of individual part sizes in bytes
storageClass	string	OPTIONAL	null	S3 storage class
checksums	object	OPTIONAL	null	Additional checksum algorithm/value pairs (e.g. CRC32, SHA256)
aclGrants	string	OPTIONAL	null	Explicit ACL grants, PERMISSION:granteeId entries joined by ;
sseEncrypted	boolean	OPTIONAL	false	true when the stored bytes are ciphertext this server produced
sseOriginalSize	integer	OPTIONAL	0	Plaintext size recorded when sseEncrypted is true
kmsEnvelope	object	OPTIONAL	null	Envelope-encryption key material (see below). null for a legacy object encrypted with the derived-key scheme, or an unencrypted object

When present, kmsEnvelope records how the object's random data key was wrapped:

Field	Type	Required	Description
providerId	string	REQUIRED	KMS provider that wrapped the data key (local, bitwarden, …). The read path unwraps with the provider named here, so the object survives a later change of the default provider
keyId	string	OPTIONAL	The SSE-KMS key id the data key is bound to (null for SSE-S3 / default)
wrappedDek	binary (base64)	REQUIRED	The object's data key, wrapped by the provider. Only the wrapped form is ever persisted

The in-memory inlineData field is not serialized as JSON; inline bytes are appended after the JSON line as described in Section 3.1.

3.3 ACL Values

PRIVATE | PUBLIC_READ | PUBLIC_READ_WRITE | AUTHENTICATED_READ |
BUCKET_OWNER_READ | BUCKET_OWNER_FULL_CONTROL

3.4 Retention Mode Values

GOVERNANCE | COMPLIANCE

3.5 Inline Storage

Objects whose size is at most the inline threshold (default 131072 bytes, 128 KB) are stored entirely inside their .meta.json file (Section 3.1) with no separate data file on disk. Larger objects are written to a separate data file at <object-path> and the metadata is a JSON-only sidecar.

4. Bucket Metadata (Standard Storage)

Location: <bucket-path>/.bucket-metadata.json

Verified against dev.totis.tinys3.metadata.schema.BucketMetadata.

4.1 Schema

Field	Type	Required	Default	Description
formatVersion	string	REQUIRED	"1.0.0"	Injected by the codec; see Section 2.1
ownerId	string	OPTIONAL	null	Bucket owner identifier
versioningStatus	string	OPTIONAL	"UNVERSIONED"	Versioning state; see Section 4.2
acl	string	OPTIONAL	"PRIVATE"	Canned ACL enum name; see Section 3.3
policy	string	OPTIONAL	null	Bucket policy as JSON string
cors	string	OPTIONAL	null	CORS configuration as XML/JSON string
website	string	OPTIONAL	null	Static website configuration
lifecycle	string	OPTIONAL	null	Lifecycle rules configuration
objectLockConfiguration	string	OPTIONAL	null	Object lock settings
encryption	string	OPTIONAL	null	Default encryption configuration
tagging	string	OPTIONAL	null	Bucket tagging configuration
notification	string	OPTIONAL	null	Event notification configuration
replication	string	OPTIONAL	null	Cross-region replication configuration
publicAccessBlock	string	OPTIONAL	null	Public access block configuration
aclGrants	string	OPTIONAL	null	Explicit ACL grants, PERMISSION:granteeId entries joined by ;
ownershipControls	string	OPTIONAL	null	Object ownership (BucketOwnerEnforced / BucketOwnerPreferred / ObjectWriter)
location	string	OPTIONAL	null	Region reported by GetBucketLocation
logging	string	OPTIONAL	null	Server access logging configuration (stored LoggingEnabled body)
requestPayment	string	OPTIONAL	null	Who pays for requests: BucketOwner or Requester

4.2 Versioning Status Values

Value	Description
UNVERSIONED	Versioning has never been enabled
ENABLED	Versioning is currently active
SUSPENDED	Versioning was enabled but is now suspended

5. Version Manifest (Standard Storage)

Location: <object-path>.versions.json

Verified against dev.totis.tinys3.metadata.schema.VersionManifest.

5.1 Schema

Field	Type	Required	Description
formatVersion	string	REQUIRED	Injected by the codec; see Section 2.1
versions	array	REQUIRED	List of version entries; see Section 5.2

5.2 Version Entry Schema

Field	Type	Required	Description
versionId	string	REQUIRED	Version identifier ("null" for a pre-versioning implicit version)
size	integer	REQUIRED	Object size in bytes; 0 for delete markers
lastModified	string	REQUIRED	ISO-8601 timestamp of version creation
eTag	string	OPTIONAL	Entity tag; null for delete markers
isDeleteMarker	boolean	REQUIRED	true if this entry is a delete marker

5.3 Ordering

The versions array MUST be ordered most-recent-first. Index 0 is the current version.

5.4 Version Data Files

Individual version payloads and their metadata are stored as siblings of the current object, named <object-path>.v-<versionId> (data) and <object-path>.v-<versionId>.meta.json (metadata). The metadata sidecar of a version carries no inline payload.

6. Erasure Coding Metadata (Single-Node)

Location: <object-path>.erasure-meta.json

Note: despite the .json filename, new writes contain the compact binary record described in Section 2.1 (BinaryErasureMetaCodec); the extension is retained for path compatibility. Files written by older versions remain JSON and are still read. The fields below describe the logical record regardless of encoding.

Verified against dev.totis.tinys3.filesystem.metadata.ErasureObjectMeta, BinaryErasureMetaCodec, ErasureConfig, and ErasurePathBuilder.

The erasure record is a pure storage-layer type with no dependency on the S3/object schema. Higher-level protocols that need to travel alongside a shard set attach an opaque userPayload blob rather than embedding an object-metadata document.

6.1 Schema

Field	Type	Required	Default	Description
formatVersion	string	OPTIONAL	"1.0.0"	Present only in legacy JSON records; the binary record identifies itself by magic + structural version instead (Section 2.1)
size	integer	REQUIRED	-	Original unencoded file size in bytes
lastModified	integer	REQUIRED	0	Storage-layer modification time (epoch millis); 0 for single-node erasure writes
inline	byte[]	OPTIONAL	null	Base64 object bytes for inline objects; see Section 6.4
dataDir	string	OPTIONAL	null	Generation directory holding this object's shards; null when inline
previousDataDir	string	OPTIONAL	null	The generation this write superseded, retained one write for in-flight readers
stripeSize	integer	OPTIONAL	0	Stripe size for striped objects; 0 when not striped
shardSize	integer	OPTIONAL	0	Size of each shard; 0 for inline and chunked objects
dataShards	integer	REQUIRED	-	Number of data shards
parityShards	integer	REQUIRED	-	Number of parity shards
checksum	string	OPTIONAL	null	MD5 of original data; null for chunked/inline files
shards	array	OPTIONAL	null	Shard info for small (single-generation) files; see Section 6.2
chunkCount	integer	REQUIRED	-	Number of chunks; 1 for small/inline files
chunkSize	integer	REQUIRED	-	Maximum chunk size; default 16777216 (16 MB)
chunks	array	OPTIONAL	null	Chunk info for large files; see Section 6.3
shardHashes	string[][]	OPTIONAL	null	Per-generation shard hashes (distributed/cluster use)
version	integer	OPTIONAL	0	Object version counter for cluster convergence
tombstone	boolean	OPTIONAL	false	true when this record marks a delete
userPayload	byte[]	OPTIONAL	null	Opaque higher-level payload (e.g. serialized object metadata)

6.2 Shard Entry Schema

Field	Type	Required	Description
index	integer	REQUIRED	Shard index from 0 to (dataShards + parityShards - 1)
checksum	string	OPTIONAL	Whole-shard HighwayHash keyed MAC (16 hex characters). Written on legacy and healed shards; null on new writes, which use blockChecksums instead
size	integer	REQUIRED	Shard size in bytes
present	boolean	REQUIRED	true if shard was successfully written to storage
blockChecksums	string[]	OPTIONAL	Per-block HighwayHash MACs (16 hex each), one per 128 KiB block of the shard (last block may be shorter). Written on new writes to enable block-level ranged reads and finer bitrot localization; null on legacy shards, which carry only the whole-shard checksum

Every shard entry carries either checksum (legacy/healed, whole-shard) or blockChecksums (new writes, per-block); both verify the same HighwayHash keyed MAC, and readers accept either.

6.3 Chunk Entry Schema

Field	Type	Required	Description
chunkIndex	integer	REQUIRED	Chunk index from 0 to (chunkCount - 1)
size	integer	REQUIRED	Actual data size of this chunk in bytes
shardSize	integer	REQUIRED	Size of each shard within this chunk
shards	array	REQUIRED	Shard entries for this chunk; see Section 6.2

6.4 File Classification

• Inline: originalSize <= inlineThreshold (default 128 KB) - object bytes stored in inlineData; no shards, dataDir is null.
• Small: originalSize <= chunkSize (16 MB) and not inline - a single generation of shards, described by shards.
• Chunked: originalSize > chunkSize - split into chunks of chunkSize, each erasure-coded independently and described by chunks.

6.5 Generation Directories

Each write of an object stores its shards under a fresh generation directory named for that write (dataDir). Metadata references shards by generation, so an overwrite never mutates the shards a concurrent reader is using. The immediately preceding generation (previousDataDir) is retained on disk and removed one write later, once no reader can still name it.

6.6 Tombstones

A delete that cannot reach every disk writes a tombstone at <object-path>.erasure-meta.json.deleted on the reachable disks, so the object is treated as gone until stale replicas are cleaned up.

7. Multipart Upload Metadata

Multipart tracking depends on the deployment mode (verified against dev.totis.tinys3.DefaultS3FileOperations).

7.1 Standalone Mode (cache enabled)

Upload state is held in an in-memory map keyed by upload ID and is not persisted as a standalone metadata file. Part data is written to a node-local temporary directory.

7.2 Cluster Mode (cache disabled)

So that any node can accept a part or complete the upload — and so an in-flight upload survives a restart — upload state and part data are persisted through the object layer under a reserved system prefix:

.tinys3.sys/multipart/<uploadId>/upload.json      # upload metadata (Section 7.3)
.tinys3.sys/multipart/<uploadId>/part-<N>.json     # per-part metadata (Section 7.4)
.tinys3.sys/multipart/<uploadId>/part-<N>           # raw part data

The uploadId is a server-minted 32-character lowercase hex string.

7.3 Upload Metadata (upload.json)

Serialized with a plain JSON mapper (no formatVersion field).

Field	Type	Required	Description
bucket	string	REQUIRED	Target bucket name
key	string	REQUIRED	Target object key
initiated	integer	OPTIONAL	Initiation timestamp in epoch milliseconds
contentType	string	OPTIONAL	Content type for the final assembled object
userMetadata	object	OPTIONAL	User metadata for the final assembled object
initiatorId	string	OPTIONAL	Identifier of the upload initiator
initiatorDisplayName	string	OPTIONAL	Display name of the upload initiator
ssecAlgorithm	string	OPTIONAL	Customer-provided encryption algorithm
ssecKeyMD5	string	OPTIONAL	Base64 MD5 of customer-provided encryption key
checksumAlgorithm	string	OPTIONAL	Requested checksum algorithm
checksumType	string	OPTIONAL	Checksum type (e.g. FULL_OBJECT / COMPOSITE)
partChecksums	object	OPTIONAL	Base64 checksum per part, keyed by part number
objectLockMode	string	OPTIONAL	Object lock mode to apply on completion
objectLockRetainUntilDate	string	OPTIONAL	Retain-until date to apply on completion
objectLockLegalHold	boolean	OPTIONAL	Legal hold to apply on completion

7.4 Part Entry Schema (part-<N>.json)

Field	Type	Required	Description
partNumber	integer	REQUIRED	Part number from 1 to 10000
eTag	string	REQUIRED	MD5 hash of part data
originalSize	integer	REQUIRED	Plaintext size of the part; -1 if unknown
encryptedSize	integer	REQUIRED	Stored (possibly encrypted) size of the part; -1 if unknown

8. File Layout

8.1 Standard Storage

<storage>/
  <bucket>/
    .bucket-metadata.json
    <key>                       # data file (omitted for inline objects)
    <key>.meta.json             # metadata (+ inline bytes after a newline, when inline)
    <key>.versions.json         # version manifest (versioned buckets)
    <key>.v-<versionId>         # version data
    <key>.v-<versionId>.meta.json
  .tinys3.sys/
    multipart/
      <uploadId>/
        upload.json
        part-<N>.json
        part-<N>

8.2 Erasure Storage (Single-Node)

<disk-N>/
  <bucket>/
    .bucket-metadata.json
    <key>.erasure-meta.json          # replicated across disks
    <key>.erasure-meta.json.deleted  # tombstone (only when a delete was partial)
    <key>/
      .shards/
        <dataDir>/                   # one generation per write
          shard-<M>                  # small objects
          chunk-<C>-shard-<M>        # chunked objects

Shard files live under a per-write generation directory (<dataDir>), not as flat <key>.shard-N siblings. Staging writes use a .staging suffix and temporary work lives under a .temp/ directory.

8.3 Distributed / Cluster Storage

Verified against dev.totis.tinys3.distributed.DiskShardStore, ShardStore, and ObjectMeta.

Each node owns one or more disk roots. Within a disk root, every object occupies a single directory whose name is the object key percent-encoded into one ASCII path segment (/ and non-ASCII bytes become %XX). Keys too long for a single filesystem component are stored as a truncated prefix plus = plus 40 hex chars of the key's SHA-256; the full key is then recorded in a __key__/0 entry inside the object directory.

<disk-root>/
  =tmp/                    # in-flight write scratch, swept at startup
  <encoded-key>/
    __meta__/0             # replicated per-object metadata (Section 9)
    __key__/0              # original key record (hash-named objects only)
    <generation>/          # one directory per write (dataDir, or "<dataDir>-s<i>" per stripe)
      <shardIndex>         # framed shard bytes (Section 8.4)

The metadata directory name is __meta__ (ShardStore.META_DIR). Listing enumerates keys by scanning each disk root for object directories that carry a __meta__/0 entry; there is no central index. Stale generations are left behind for in-flight readers and collected by the healer past a grace window.

8.4 Shard Framing (Distributed)

Every shard and metadata file written by DiskShardStore is framed with an 8-byte header: a 4-byte magic tag S3C1 followed by a 4-byte big-endian CRC32 of the payload. A read that fails the magic or CRC check returns null (treated as a missing shard), which is the cluster's defence against silent bit-rot.

9. Distributed Object Metadata Record

Location: <disk-root>/<encoded-key>/__meta__/0 (framed, Section 8.4)

Verified against dev.totis.tinys3.distributed.ObjectMeta. This record is NOT JSON; it is a UTF-8, newline-separated list of fields, replicated to every metadata slot.

9.1 Field Order

Line	Field	Type	Description
0	size	integer	Original object byte length (used to trim erasure padding on read)
1	dataDir	string	Generation directory of the current shard set
2	lastModified	integer	Last-modified time in epoch milliseconds
3	inline	string	Base64 object bytes for small objects; empty when not inlined
4	stripeSize	integer	Data bytes per erasure stripe; 0 for inline/empty/legacy single-stripe objects
5	shardHashes	string	Per-stripe shard hashes; empty when absent

Lines 3 through 5 are OPTIONAL trailing fields; older records that omit them MUST still parse. A record MUST have at least the first three lines.

9.2 Inline and Striped Objects

• Inline: small objects store their bytes directly in the inline field (line 3); no shards are written.
• Striped: when stripeSize > 0, the object is split into independent erasure stripes stored under generations <dataDir>-s<i>, so read and write touch one stripe at a time.

9.3 Shard Hashes

shardHashes records, per stripe, a strong hash (128-bit truncated SHA-256, hex) of every erasure shard (data + parity), so a read can verify each gathered shard and discard a corrupt one — tolerating up to parityShards corrupt shards, not just missing ones. Encoding: stripes separated by ;, shard hashes within a stripe separated by ,.

10. Migration

10.1 Read Path

Implementations SHOULD tolerate older files by defaulting absent optional fields (Section 2.3). Parsed metadata MAY be cached; caches are disabled in cluster mode because another node can write the same object.

10.2 Write Path

Implementations MUST write metadata in the current format.

10.3 Atomicity

Metadata writes MUST be atomic with respect to concurrent readers. Implementations SHOULD write to a unique temporary file and atomically rename it into place (as DiskShardStore does), so a reader always sees either the old or the new file, never a missing one.

11. Limits

Limit	Value	Description
Max object size	5 TB	Maximum size of a single object (S3 protocol limit)
Max part size	5 GB	Maximum size of a multipart upload part (S3 protocol limit)
Min part size	5 MB	Minimum size of a non-final multipart part
Max parts per upload	10000	Enforced when completing a multipart upload
Standard inline threshold	128 KB	Objects at or below this size are stored inside their metadata file
Erasure inline threshold	128 KB	Default; objects at or below this size are stored inline in erasure metadata
Erasure chunk size	16 MB	Default chunk size for single-node erasure coding

12. Checksums

Algorithm	Usage	Format
MD5	ETag, erasure object checksum (whole-object)	32 lowercase hexadecimal characters
HighwayHash (keyed MAC)	Erasure shard integrity — whole-shard checksum and per-block blockChecksums (Section 6.2)	64-bit, 16 lowercase hexadecimal characters
CRC32	Distributed shard framing (Section 8.4)	4-byte big-endian header field
SHA-256 (truncated)	Distributed shard-hash verification (Section 9.3)	128-bit, 32 hex characters

13. Reserved Namespaces

• .tinys3.sys/ - Reserved system prefix (e.g. cluster multipart state, Section 7.2).
• __meta__, __key__, =tmp - Reserved directory names in the distributed disk layout (Section 8.3); object keys never collide with them because keys are percent-encoded and these names use characters the encoder does not emit.
• User metadata keys beginning with x-amz-meta- are carried verbatim in userMetadata. TinyS3 does not currently enforce a reserved sub-namespace within user metadata.