# S3 **The transport that is not a mount: serve a driver as an S3-compatible HTTP endpoint.** `mountx/s3` speaks enough of the S3 REST API — path-style, SigV4-signed — for `rclone`, the AWS CLI, an AWS SDK, or a presigned URL in a browser to talk to any `FsDriver` as if it were a bucket. There is no kernel involved and nothing here produces a mountpoint, which is why it is a separate subpath rather than another choice for [`mountx/auto`](/transports/auto): auto's whole contract is "hand back a mounted directory", and a gateway never has one to hand back. ```ts import { createS3Server } from "mountx/s3"; import { createMemoryDriver } from "mountx/drivers/memory"; await using server = await createS3Server(createMemoryDriver()).listen(); server.url; // http://127.0.0.1: // aws --endpoint-url $server.url s3 ls s3://mountx ``` ## Quick start ### One driver, or several ```ts // One driver, one bucket (default name "mountx"): const one = createS3Server(driver, { bucket: "photos" }); // Several drivers, several buckets: const many = createS3Server({ buckets: { photos: photosDriver, notes: notesDriver } }); ``` The two shapes are told apart by the **driver** side: a value with `stat`, `readdir` and `open` is a driver whatever else it carries; only then is a `buckets` property looked at. `createS3Server()` returns immediately — nothing is bound until [`listen()`](#s3server) — but it throws right away for a bucket name that cannot be a URL path segment, or for a `host` [it will not bind](#who-may-connect). ### rclone The two flags below are not stylistic — this gateway implements `ListObjectsV2` only and has no `CreateBucket`, and rclone needs to be told both things or its first request fails: ```ini # rclone.conf, or the equivalent RCLONE_CONFIG_MX_* environment variables [mx] type = s3 provider = Other access_key_id = AKIAIOSFODNN7EXAMPLE secret_access_key = wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY endpoint = http://127.0.0.1:PORT region = us-east-1 force_path_style = true list_version = 2 no_check_bucket = true ``` ```sh rclone sync ./photos mx:mountx/photos rclone lsjson -R mx:mountx ``` - **`list_version=2`** — rclone picks the list API from the `provider`: V2 for `AWS`, V1 for everything else, including `Other`. This gateway only implements `ListObjectsV2`, so an unconfigured remote gets `501 NotImplemented: ListObjects (V1) is not implemented by this gateway` on its first listing. - **`no_check_bucket=true`** — rclone's S3 backend sends `CreateBucket` before an upload unless a listing has already marked the bucket as existing; this gateway's buckets are the drivers it was constructed with; there is nothing to create, so it answers `NotImplemented` and rclone reports `failed to prepare upload`. - **`force_path_style=true`** — this gateway speaks **path-style URLs only** (`/bucket/key`), never virtual-hosted (`bucket.host/key`). An empty directory needs **both** `--create-empty-src-dirs` (rclone's local backend otherwise drops it before the sync sees it) and `--s3-directory-markers` (its S3 backend otherwise has no key to represent one) — `rclone sync --create-empty-src-dirs --s3-directory-markers ./tree mx:mountx/tree` round-trips one, both directions. ### Presigned URLs `mountx/s3` exports the same SigV4 codec the session verifies against, so a presigned URL needs no server: ```ts import { presignRequest, formatAmzDate, uriEncode } from "mountx/s3"; const path = "/mountx/notes/hello.txt"; const signed = presignRequest({ method: "GET", path, headers: [{ name: "host", value: "127.0.0.1:PORT" }], credentials: { accessKeyId, secretAccessKey }, region: "us-east-1", timestamp: formatAmzDate(Date.now()), expiresIn: 300, // seconds }); const query = signed.query.map((q) => `${uriEncode(q.name)}=${uriEncode(q.value)}`).join("&"); const url = `http://127.0.0.1:PORT${path}?${query}`; ``` ```sh curl "$url" # no Authorization header anywhere — the signature lives in the query ``` ## Who may connect SigV4 is only authentication when there is a secret to verify against, so the rule is split on whether `credentials` was passed: - **No `credentials`** — every signature is parsed but never checked, so the bind is **loopback-only**: a non-loopback `host` — including the unspecified `0.0.0.0`/`::`, which bind every interface — is refused outright, before a socket opens, with a named `S3BindError`. - **With `{ accessKeyId, secretAccessKey }`** — every request is verified with strict SigV4 (both the header form and the presigned-query form), clock skew bounded to ±15 minutes, and any `host` is allowed. The loopback check is **literal**: `localhost`, `::1`, and any `127.x.y.z` (plus the `::ffff:` IPv4-mapped spelling a dual-stack listener uses). Nothing resolves a name, so a hostname that happens to point at `127.0.0.1` is still refused — the check is a fact about the address handed in, not about where it currently resolves. ## Semantics ### ETags are derived, not MD5 An object's ETag is the first 32 hex characters of `sha256("dev:ino:size:mtimeMs")`, suffixed `-1`. The `-1` is the shape a multipart ETag has, which is deliberate: it is the signal to a client that this is **not** a content hash. rclone reads it that way — `rclone check` announces it is using MD5, finds none it can verify, and falls back to comparing size and modification time instead, reporting the files as matching and the hashes as "could not be checked" rather than treating the mismatch as a difference. `--size-only` gets the same result with no hash step logged at all. The ETag is otherwise stable across repeated `GET`s of the same object. ### mtime, not much else `x-amz-meta-mtime` (epoch seconds — a bare integer when whole, three decimals when not) is rclone's own metadata convention, and the only `x-amz-meta-*` header this gateway keeps. On `PUT` (and on `CompleteMultipartUpload`, from the value given to `CreateMultipartUpload`) it maps to `driver.utimes()` when the driver declares `times`; on `GET`/`HEAD` it is emitted from `stat.mtime`. Every other `x-amz-meta-*` header is dropped. Resolution stops at the **millisecond** — the header has no room for more — so a file with sub-millisecond mtime precision re-transfers on the next `rclone sync` unless it is run with `--modify-window 1ms`. `Content-Type` is not stored at all: every `GET`/`HEAD` answers `application/octet-stream`. ### Directories are prefixes There is no directory object except for an **empty** one. `PUT key/` (empty body) is a recursive `mkdir`; `DELETE key/` is an `rmdir` (`ENOTEMPTY` → `409 BucketNotEmpty`). A directory with children needs no marker — its children are the listing — but an empty one would otherwise vanish from every listing, so it lists as a zero-byte object with key `key/`. That is exactly what lets `sync` recreate it, and exactly what rclone's `--s3-directory-markers` flag asks for. ### The `NotImplemented` boundary The capabilities philosophy applies to the wire as much as to a driver: an operation this gateway does not implement answers a well-formed S3 `NotImplemented` (501) error document, never a fall-through to a neighbouring one — `GET /bucket/key?acl` is refused, and must never quietly answer the object's bytes because the sub-resource went unrecognized. `CreateBucket`/`DeleteBucket` answer the same way; bucket existence here is "was a driver given to `createS3Server()`", not something a request can create. `?versioning` is refused too, which is what makes `rclone purge` log `Failed to read versioning status, assuming unversioned` and then carry on correctly. Supported: `ListBuckets`, `HeadBucket`, `ListObjectsV2`, `GetObject`, `HeadObject`, `PutObject`, `DeleteObject`, `DeleteObjects`, `CopyObject`, and the five multipart operations below. Conditionals (`If-Match`/`If-None-Match`/`If-Modified-Since`/`If-Unmodified-Since`, `304`/`412`) and a single `Range` (`416` when unsatisfiable) are implemented on `GET`/`HEAD`. ### Multipart `CreateMultipartUpload`, `UploadPart`, `CompleteMultipartUpload`, `AbortMultipartUpload` and `ListParts` stage their parts **through the driver**, under a reserved bucket-root prefix, `.mountx-multipart//part-` — invisible to every other operation (a direct `GET`/`HEAD`/`PUT` of a staging key answers as though it did not exist, and it is skipped in every listing). `CompleteMultipartUpload` streams the staged parts together in the order the client listed them, because parts arrive out of order and out of size, so the offset of part _N_ is unknowable until every part before it exists. Aborting an upload, and closing the server, both sweep the staging area; an interrupted upload leaves no debris behind once either happens. ### The write-in-place caveat A `PUT` (and multipart completion) writes the object in place: there is no temporary file and no rename, because the driver interface has no atomic-create primitive to build one on. What is guaranteed is the _first_ byte — the destination is not opened, so an existing object is not truncated and a new one is not created, until the first payload byte has arrived (and, for a signed `aws-chunked` body, been verified). An upload **rejected at or before its first byte** therefore leaves the bucket exactly as it was; one that fails **after** writing has begun leaves whatever had been written by then — a partial object where a whole one used to be, not the old object restored. ## Serving without mounting, on purpose There is nothing platform-specific here: `createS3Server` is `node:http`, and Node's HTTP server runs wherever Node runs. Nothing in this transport has been exercised on Windows, but nothing in it depends on anything Linux- or macOS-specific either — it is an HTTP server serving an already-portable driver interface. ## `createS3Server(source, options?)` ```ts function createS3Server( source: FsDriver | { buckets: Record }, options?: S3ServerOptions, ): S3Server; ``` ### `S3ServerOptions` Extends [`S3SessionOptions`](#s3session), so everything the session takes is settable here too. | option | default | | | ------------------ | ------------- | ------------------------------------------------------------------------------------------- | | `bucket` | `"mountx"` | bucket name for the single-driver call shape; ignored for a bucket map | | `host` | `"127.0.0.1"` | address to bind — see [Who may connect](#who-may-connect) | | `port` | `0` | an ephemeral port, which `S3Server.port` then reports; never a conventional S3-gateway port | | `credentials` | none | `{ accessKeyId, secretAccessKey }` — present enables verified SigV4 and any bind | | `drainTimeout` | `5000` | ms `close()` lets in-flight responses finish before dropping connections | | `onTransportError` | none | `(error, peer)` — a socket error, or a reply that could not be written | ### `S3Server` ```ts interface S3Server extends AsyncDisposable { readonly session: S3Session; readonly host: string; readonly port: number; readonly url: string; // e.g. "http://127.0.0.1:54321"; IPv6 bracketed readonly buckets: string[]; readonly connections: number; listen(): Promise; // idempotent; resolves once bound close(): Promise; // stop accepting, drain, drop, sweep multipart staging — idempotent } ``` `close()` drains in a deliberate order: stop accepting, let in-flight responses finish (bounded by `drainTimeout`), drop what's left, and only then sweep the multipart staging area — sweeping first could race a part still being written by a request that was allowed to finish. ### `S3BindError` / `isS3BindError()` / `isLoopbackHost()` ```ts class S3BindError extends Error { readonly code: "ERR_S3_BIND"; readonly host: string; } function isS3BindError(error: unknown): error is S3BindError; function isLoopbackHost(host: string): boolean; ``` The one error type `createS3Server()` throws for an address, named so it can be caught rather than pattern-matched on a message. `isLoopbackHost()` is the literal check described above, exported on its own because it is worth asking without constructing a server first. ## `S3Session` ```ts new S3Session(buckets: Record, options?: S3SessionOptions) ``` One HTTP request in, one S3 reply out, with no socket anywhere — which is what makes the whole protocol testable with no listener and no client, the same posture `FuseSession` and `NfsSession` keep. ```ts session.buckets; // Map session.bucketNames; // string[], in listing order session.stats; // { requests, replies, errors, operations: Map, assertions } await session.handleRequest(head, body?); // → S3StreamResponse; never rejects await session.close(); // idempotent; sweeps multipart staging ``` `handleRequest`'s boundary is **streaming**, unlike `NfsSession.handleCall(bytes)`: the body in and the body out may each be an `AsyncIterable`, because buffering a multi-gigabyte `PUT` to parse it — or a multi-gigabyte `GET` to answer it — is not something a gateway may do. ### `S3SessionOptions` | option | default | | | ---------------- | --------------------------- | ------------------------------------------------------------------------------ | | `credentials` | none | present verifies every request; absent parses signatures without checking them | | `region` | any | region the credential scope must name | | `now` | `Date.now` | the clock — SigV4 skew and reply timestamps are facts about now | | `requestId` | random hex | the `x-amz-request-id` minter | | `maxBodyBytes` | unlimited | cap on a decoded `PUT` body; over it is `EntityTooLarge` | | `maxXmlBytes` | the XML parser's own budget | cap on a request document (`DeleteObjects`, `CompleteMultipartUpload`) | | `readChunkBytes` | 128 KiB | bytes per positional read while streaming a `GET` | | `debug` | on outside production | run the reply-exactly-once assertions | | `onError` | none | called for every request that ends in an error reply | | `onAssertion` | collect | called when a dev-mode assertion fails | ## The layers below Everything except `server.ts` touches no socket and no listener, which is what makes the protocol testable by a signing JS client built out of the same codecs (`test/s3/client.ts`, the conformance column's transport): | module | | | -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `constants.ts` | the errno → S3 error table (`s3ErrorOf`), and the protocol's limits (`MAX_KEYS`, `MIN_PART_SIZE`/`MAX_PART_SIZE`/`MAX_PARTS`, `MAX_KEY_BYTES`, `MULTIPART_PREFIX`) | | `sigv4.ts` | AWS Signature Version 4, signed _and_ verified: `signRequest`/`verifyRequest`/`presignRequest`, the canonical request and signing-key derivation, `uriEncode` | | `xml.ts` | the bounded XML encoder and parser for the list/error/multipart documents | | `chunked.ts` | the `aws-chunked` / `STREAMING-AWS4-HMAC-SHA256-PAYLOAD` decoder (`AwsChunkedDecoder`), per-chunk signature verification | | `protocol.ts` | pure request parsing and routing — path-style URL to `(bucket, key)`, the `S3_OPS` table, `Range` and the conditional headers, the error documents | | `session.ts` | `S3Session` — the operation semantics, over one or more drivers | | `server.ts` | the socket, and the only file that imports `node:http` | All of it is re-exported from `mountx/s3` by name, except the generic XML layer `xml.ts` is itself built on (`XmlNode`, `xmlDocument`, `parseXml`, and the rest) — primitives rather than anything a consumer of an S3 gateway composes with, the same treatment `mountx/nfs` gives its own sub-struct helpers. ::note There is no RFC for S3. Every constant and error code here is transcribed from Amazon's own documentation and named where it is used — the S3 API Reference, its "Error responses" page, and the AWS SigV4 specification (including the official `aws-sig-v4-test-suite` goldens). Real clients (rclone, curl, the AWS SDKs) are oracles for this transport, never sources: what they accept is evidence, what Amazon documents is the fact. :: ## Not available - **`CreateBucket`/`DeleteBucket`, bucket ACLs and policies, object ACLs, versioning, and every `list-type=1` request.** All refused with a well-formed `NotImplemented` rather than a fake answer — see [the boundary](#the-notimplemented-boundary) above. - **Symlinks, hardlinks, permissions and access time.** An S3 object has none of these, so there is nothing for the gateway to carry even where the driver underneath has one. - **Windows**, not because anything here is platform-specific, but because it has not been run there. ## Next - [FUSE](/transports/fuse) and [NFSv3](/transports/nfs) — the two transports that produce a mountpoint. - [`mountx/auto` reference](/transports/auto) — the chooser that picks among the three mount transports, and why S3 is not an option there.