celld

Use mountx/s3 as the bucket for a celld fleet.

celld lets you run Cloudflare Workers and Durable Objects on your own machines. Each Durable Object uses its own SQLite database. celld copies the database data to an S3 bucket. Its nodes also use the bucket to coordinate with each other. It does not need a separate control service.

celld uses only a small part of the S3 API, but conditional writes must work correctly. mountx/s3 supports these requests and can serve a celld bucket from an FsDriver.

#What celld needs from a bucket

celld's fencing document lists three requirements. Only one node may own a cell at a time. A record in the bucket controls that ownership.

celld also sends ListObjectsV2, GET, ranged GET, PUT, HEAD, and DELETE requests. It uses path-style URLs and SigV4 signatures. Its list requests use prefix and start-after. It does not use multipart uploads.

celld adds x-amz-meta-litestream-timestamp to replication segments. The gateway stores only x-amz-meta-mtime, but celld does not read the Litestream header back.

#Setup

First, serve a directory as an S3 bucket:

import { createS3Server } from "mountx/s3";
import { createNodeFsDriver } from "mountx/drivers/node-fs";

await using server = await createS3Server(createNodeFsDriver("./bucket"), {
  bucket: "cells",
  port: 9100,
  credentials: { accessKeyId: "celld", secretAccessKey: "celld-secret" },
  region: "us-east-1",
}).listen();

console.log(server.url); // http://127.0.0.1:9100

Pass the same credentials to celld through the standard AWS environment variables. celld deploy also needs esbuild on PATH:

export AWS_ACCESS_KEY_ID=celld AWS_SECRET_ACCESS_KEY=celld-secret AWS_REGION=us-east-1

celld deploy ./my-worker \
  --bucket s3://cells \
  --endpoint http://127.0.0.1:9100 \
  --region us-east-1

CELLD_WATCH=./state celld \
  --bucket s3://cells \
  --endpoint http://127.0.0.1:9100 \
  --region us-east-1 \
  --listen 127.0.0.1:8080 \
  --internal-listen 127.0.0.1:8081 \
  --advertise 127.0.0.1:8081

You must set both --endpoint and --region. The endpoint is not AWS, so celld cannot find the region from it. CELLD_WATCH is the node's local working directory. It is not the bucket directory.

#What works

• Deploy: celld deploy bundles the Worker. It writes deploy/current.json and the versioned objects.
• Activation: When no node owns a requested cell, one node gets the ownership record and starts the isolate.
• Replication: celld copies the cell's SQLite data to the bucket as LTX segments under an epoch prefix.
• Restore: You can stop a node, delete its CELLD_WATCH directory, and start it again. It restores the cell from the bucket, and the Durable Object keeps its previous state.
• Two nodes: The nodes find each other through leases in the bucket. If a node loses the race for a cell, it sends the request to the owner.
• Fencing: When nodes race for a new cell, only one becomes the owner. The others receive 412 and follow the winning owner.

#Known problems

#A reader may see an incomplete PUT

The gateway writes directly to the target object. It does not write a temporary file and rename it later. See the write-in-place caveat. Real S3 does not let readers see a partly written PUT, but this gateway can.

celld can hit this problem when a node loses a conditional write and immediately reads the winning record:

celld ownership read failed: decode s3://cells/cells/Counter:5a8beec…/own.json:
  EOF while parsing a value at line 1 column 0
→ Worker failed: rejected: Error: route failed: ResolveFailed

In one test, two nodes requested 100 different new cells at the same time. This error happened four times. The request failed, but the nodes did not create two owners or different copies of the data. See issue #20.

#celld diagnose only checks access

The bucket test in celld diagnose sends one list request. It can print ok bucket even when a bucket does not support conditional writes. Use this result only to confirm that celld can reach the bucket. It does not confirm that the bucket is safe for a fleet.

#Not tested

• More than one gateway process. Do not use this setup because conditional writes are not protected across processes.
• Long runs, many cells, or low-memory conditions.
• Crash safety for the directory used by the driver. The gateway passes writes to the driver and does not call fsync itself.
• Drivers other than node-fs. The memory driver keeps the whole fleet in RAM and loses it when the process exits, so it is not suitable for a fleet bucket.

#Next

• S3: details about conditional writes and who may connect.
• Built-in drivers: choose the storage behind the bucket.
• celld documentation: learn how to use celld.