S3 compatibility testing

TinyS3 aims for compatibility with AWS S3, and the yardstick is the ceph/s3-tests suite. It is far stricter and has found real bugs — multi-tenancy leaks, a response-writer deadlock, authorization skipped on a fast path.

ALWAYS RUN THE FULL CONFORMANCE SUITE IN PARALLEL

The canonical S3-tests measurement is the 4-node cluster with pytest -n 4 — see Distributed conformance. That parallel cluster run is the pass/fail number; run it that way every time.

The single-node conformanceServer run in Running below is a quick-iteration convenience only — narrow it to one area with -k. Run it serially (no -n): xdist against a single node flaps the account/bucket-count tests, so it is never a valid measurement on its own.

Only AWS-relevant tests count

The suite targets Ceph RGW, which deliberately differs from AWS in places. Tests that AWS itself would fail are marked fails_on_aws, and they are not part of our goal:

pytest s3tests/functional/test_s3.py -m "not fails_on_aws"

That selects 603 of the 838 tests. A number quoted without this filter is not a compatibility measurement — it is measuring us against Ceph.

Setup

The suite is not vendored. Clone it anywhere outside the repo:

git clone --depth 1 https://github.com/ceph/s3-tests.git
cd s3-tests
python3 -m venv .venv
.venv/bin/pip install -r requirements.txt pytest-timeout

Create the config from the sample, changing only the port and the bucket prefix:

cp s3tests.conf.SAMPLE s3tests.conf
sed -i '' 's/^port = 8000/port = 8001/' s3tests.conf
sed -i '' 's/^bucket prefix = yournamehere-{random}-/bucket prefix = s3tests-{random}-/' s3tests.conf

Keep the rest of the sample intact. In particular do not delete the [iam] section even though TinyS3 implements no IAM — test_s3.py reads it at import time, and removing it fails collection before a single test runs.

Running

This is the single-node, quick-iteration path — run serially (no -n), and only to iterate on an area with -k. The official conformance number always comes from the parallel 4-node run in Distributed conformance.

Start a server with the three identities the suite expects:

./gradlew :tiny-s3-lib:conformanceServer -Pconformance.port=8001

Optionally pass -Pconformance.storage=/some/dir to keep the data; otherwise it uses a temporary directory. Then, from the s3-tests checkout:

TZ=UTC S3TEST_CONF=s3tests.conf .venv/bin/pytest s3tests/functional/test_s3.py \
    -q -m "not fails_on_aws" --timeout=120 --timeout-method=signal -p no:cacheprovider

Run it with TZ=UTC. test_lifecycle_expiration_header_tags_head builds its expected expiry from naive local time while S3 expiry dates are UTC, so a run that crosses local midnight in a non-UTC zone reports a lifecycle failure that looks exactly like a regression. Same code and same server: it passes under TZ=UTC and fails under CEST at 00:12.

The whole run takes about three minutes. Narrow to an area while iterating with -k, e.g. -k "lifecycle" or -k "acl".

Lifecycle runs in seconds, not days

The suite drives lifecycle tests by sleeping a few multiples of its lc_debug_interval (10s by default) and expecting day-based rules to have fired. conformanceServer therefore sets tinys3.lifecycle.debugIntervalSeconds=10, which shrinks a lifecycle "day" to ten seconds and runs the scheduler at the same period. Without it a lifecycle test can only fail, however correct the implementation is — which is exactly how one was misread as a broken feature.

Why those flags

• --timeout=120 — a hung request would otherwise stall the whole run. Do not lower it to 60: test_delete_marker_expiration legitimately waits about a minute for lifecycle processing and reports a timeout failure that looks exactly like a server hang.
• --timeout-method=signal — never use thread. It kills the session on the first hang.
• -p no:cacheprovider — stops pytest writing a cache directory into the checkout.

The three users

The suite authenticates as three identities and asserts on the owner it gets back, so they must match s3tests.conf exactly — not just the keys, but display_name, user_id and email. ConformanceServer (in lib/src/test/java/dev/totis/tinys3/) hard-codes them to match the sample config. A mismatch surfaces as confusing ownership and ACL failures rather than as an auth error.

Traps

These cost hours the first time round.

Run one server, and let it outlive the run. Restarting the server while a suite is in flight leaves pytest waiting on a connection to a process that no longer exists — it sits at 0% CPU indefinitely and the per-test timeout never fires, because no test is running. It reads exactly like a server hang and is not one.

Give each suite its own server. Anything left behind — scratch buckets from probing by hand, or buckets from a previous suite file — fails test_list_buckets_paginated, which asserts the account owns none. It reads like a regression and is not one. Restart with a fresh storage directory between suites.

Errors cascade. nuke_prefixed_buckets runs in an autouse fixture, so a single bucket that cannot be emptied errors every subsequent test. One run showed 308 errors of which 212 were this. Always fix the first failure before reading the rest of the report.

A quiet log is not a hang. With -q and output redirected, pytest block-buffers and flushes roughly every 72 tests. Check actual process CPU before concluding anything is stuck — and check the java/python PIDs, not the wrapper shell.

Rebuild the http module explicitly. If your test classpath resolves http/ from http/build/libs/tiny-s3-http-*.jar rather than a classes directory, run ./gradlew :tiny-s3-http:jar before restarting the server or you will silently test stale code. lib/ is a classes directory and picks up rebuilds normally.

Distributed (multi-node) conformance

The single-server run above measures the standalone build. The distributed module (multi-node quorum storage) has its own conformance run, and it has two hard rules: it runs against a 4-node cluster, and the suite runs in parallel. Both are mandatory — a serial single-node-style run does not exercise the code that distribution changes, and it is far too slow to iterate on.

Why parallel is mandatory

Serial is not just slower here, it is impractical: a full 603-test run against a co-located cluster takes upward of two hours, because every request coordinates across nodes over gRPC. Parallel (-n4) brings that down to roughly ten minutes — about a 20× speedup — which is the difference between iterating and not.

Parallel is safe only with quorum slack. All the server JVMs and the pytest workers share one machine, so a parallel run saturates CPU and some gRPC calls transiently miss their deadline. With a 4-node 2+2 layout a metadata write needs a majority (3 of 4), so one momentarily-starved node is tolerated and the write-quorum retry absorbs the blip. A 2-node 1+1 layout has no slack — every metadata write needs both nodes — so the same load produces spurious failures that look exactly like regressions. Never measure distributed conformance on 2 nodes.

Ports

Port range	Purpose
8101–8104	Per-node S3/HTTP endpoints. Any node answers any request, so the suite points at 8101.
9101–9104	Inter-node gRPC (quorum reads/writes, locks). Not for clients.
8300	Live test dashboard (below).

Start the cluster

Build the fat jar, then bring up 4 nodes:

./gradlew :tiny-s3-app:fatJar
core/distributed/testing/cluster-run.sh          # 4 nodes, 2+2 erasure, ports 8101-8104

cluster-run.sh is parameterised by environment (NODES, BASE_HTTP, DATA_SHARDS, PARITY_SHARDS, WORK, JAR, …); the defaults are the conformance layout. It prints the work directory it created — configs, per-node data disks and nodeN.log live there.

Point s3tests.conf at port = 8101 and keep the three-user section exactly as in the standalone setup above (the ownership/ACL assertions depend on it).

The cluster must be configured with all three identities, not one. cluster-run.sh writes them into each node's auth block (a primary credential plus an auth.users list), values matching s3tests.conf.SAMPLE. This is not cosmetic: with a single shared credential every "other user" collapses into the caller, so ownership shows the wrong account and cross-user access is never denied — about forty ACL and access-grant tests fail as a block, and the shared cleanup fixture then cascades that into hundreds of downstream errors. If you write node configs by hand, give them the same three users.

Run the suite in parallel

TZ=UTC S3TEST_CONF=s3tests.conf .venv/bin/pytest s3tests/functional/test_s3.py \
    -m "not fails_on_aws" -v -rf --tb=line \
    -n 4 -p no:cacheprovider --timeout=120 --timeout-method=signal

-n 4 is the required flag (needs pip install pytest-xdist). Keep every other flag from the standalone run — the TZ=UTC, --timeout and --timeout-method=signal rationale is unchanged and still load-bearing.

Watch it live

-q only emits dots; use -v (as above) so each test name streams, and run the dashboard against the pytest log:

.venv/bin/pytest ... > run.log 2>&1 &
python3 core/distributed/testing/dashboard.py run.log 8300
# open http://localhost:8300

It shows a progress bar, live pass/fail counters, the accumulating FAILED list, and a feed of the most recent results (green/red). It parses both plain and xdist ([gw0] PASSED …) output.

Separate real failures from load noise

A parallel run can surface a few spurious failures from CPU starvation rather than a real bug. Two cheap filters tell them apart:

1. Look at the shape. Real gaps fail as a coherent block (a whole ACL group, all the bucket-gone tests); load noise is one scattered basic test (test_bucket_create_delete passing on the next run).

2. Re-run the failures serially. Feed the FAILED list back with -n0:

   .venv/bin/pytest s3tests/functional/test_s3.py -n0 --tb=line \
       $(sed 's/^/-k /' failed.txt | paste -sd' ' -)

   Anything that passes serially was load-induced; anything that still fails is ours to fix.

Beyond test_s3.py

The suite has more than the object API. Measured with -m "not fails_on_aws and not fails_on_rgw":

File	Status
test_headers.py	28 / 28
test_s3select.py	28 passed, 6 failed — JSON input, error reporting, progress frames
test_iam.py, test_sts.py, test_sns.py, test_s3control.py	unimplemented services, not S3

Filter fails_on_rgw as well when reading these: many of those tests cannot do what they claim — one notes "remove once we have learned how to remove the authorization header", which botocore re-adds — so they fail against every implementation.

See Feature status for the full sweep across every suite and remaining gaps.

Current status

test_s3.py -m "not fails_on_aws" against the 4-node cluster (distributed/testing/cluster-run.sh, -n 4). The current pass/fail/skip baseline and the per-failure triage live in docs/conformance-status.md, the single source of truth for that number.

Measure against a fresh storage directory. Reusing one between runs inflates the result: some tests pass only because an earlier run left state behind, and the number quietly drifts up. Two runs are comparable only if both started clean.

All 11 remaining failures are ones we do not intend to pass: eight are the tenant group (which fails CreateBucket with InvalidBucketName — tenants are a Ceph concept AWS has no equivalent for), test_list_buckets_paginated, and two bucket-policy edge cases (below).

Interpreting the result

Some failures are expected and should not be "fixed":

• test_bucket_policy_set_condition_operator_end_with_IfExists — expects a policy's implicit deny to lock out the bucket owner. AWS does not do this; the test is marked fails_on_rgw with Ceph's own TODO against it.
• test_bucket_policy_put_obj_request_obj_tag — a broken test. It registers the tagging header on client then calls alt_client.put_object, so no tags are ever sent.
• test_get_object_torrent — AWS retired BitTorrent support.
• Two SSE-C wrong-key tests — they expect Ceph's 400; AWS returns 403.
• test_multipart_resend_first_finishes_last — lists PartNumber 1 twice in the completion and expects it to succeed. AWS requires parts in ascending order and rejects duplicates with InvalidPartOrder, which is what we now return. Passing this test would mean being more lenient than AWS. The behaviour that does matter — a re-uploaded part replacing the previous one — works.

Everything else that fails is ours.

Known gaps

Absent, and so responsible for whole blocks of failures: SSE-KMS, replication, notification transports other than webhooks (the SNS/SQS-shaped tests), storage classes beyond STANDARD, lifecycle transitions, inventory and analytics, batch operations, and IAM/STS.

One caveat worth stating plainly: SSE-S3 and SSE-KMS do not encrypt at rest. The headers are accepted and AES256 is reported back, but the object is written in cleartext. Only SSE-C actually encrypts. Treat those as unimplemented rather than working.

Performance benchmarking

Throughput/latency benchmarks live in the benchmarks/ module (perfBenchmark, concurrentPutProfile, maxThroughputBenchmark, …). They fork a local server by default or run against an external endpoint with -Pbench.endpoint, which is also how you do a fair, same-client comparison against another S3 implementation. Full runbook — tasks, parameters, and the one-server-at-a-time / start-empty / matched-durability rules for a fair comparison — is in benchmarks/README.md.