S3 storage

A container on Cloud Run, Fargate, Kubernetes or Fly.io loses its disk on every redeploy, and two replicas never see each other's files. daravel_s3 puts the Storage facade on a bucket instead: AWS S3 or any S3-compatible store (Cloudflare R2, MinIO, DigitalOcean Spaces, Backblaze B2, Wasabi, Google Cloud Storage through its interoperability API). It signs requests with AWS Signature Version 4 in pure Dart, so it adds no AWS SDK and compiles into the same single AOT binary.

dart pub add daravel_s3

// bootstrap/app.dart
import 'package:daravel_s3/daravel_s3.dart';

Application createApp() => Application.configure(basePath: basePath)
    .withS3()
    // ...
    .create();

The variables are the ones Laravel's config/filesystems.php reads, so a Laravel and a Daravel application can share one .env and one bucket:

FILESYSTEM_DISK=s3            # every Storage call goes to the bucket

AWS_ACCESS_KEY_ID=...
AWS_SECRET_ACCESS_KEY=...
AWS_DEFAULT_REGION=ap-southeast-1
AWS_BUCKET=my-app-files
AWS_URL=https://cdn.example.com   # optional public base URL (a CDN)
AWS_VISIBILITY=private            # or public: objects get x-amz-acl public-read
# AWS_SESSION_TOKEN=...           # temporary credentials

With FILESYSTEM_DISK=s3 boot fails while the key, the secret or the bucket is missing. Without it the bucket is still there as Storage.disk('s3') next to local and public. withS3(disk: 'media') registers it under another name.

Other providers#

Path-style addressing (endpoint/bucket/key) is also the way to reach a bucket whose name contains dots over HTTPS.

Using the disk#

Every Disk method works on the bucket, with the same results as on a local disk (the package runs one contract test suite against LocalDisk, MemoryDisk and S3Disk):

await Storage.disk('s3').put('avatars/${user.id}.png', bytes);
final png = await Storage.disk('s3').getBytes('avatars/${user.id}.png');

await Storage.disk('s3').copy('a.pdf', 'archive/a.pdf'); // copied inside S3
await Storage.disk('s3').delete(['old/1.txt', 'old/2.txt']); // one request per 1000
final files = await Storage.disk('s3').allFiles('reports');

Objects are stored with a Content-Type from their extension, so a browser opening their URL renders them. Directories are key prefixes; makeDirectory writes the empty dir/ object Laravel writes, and both frameworks list the same tree. append reads the object and writes it back, since S3 cannot append.

url(path) is AWS_URL/key when AWS_URL is set and the bucket's own URL otherwise. It is only reachable when the object is public (bucket policy or AWS_VISIBILITY=public); for private objects use a temporary URL.

Streaming large files#

putStream keeps at most one part (8 MiB) in memory. A stream that fits in one part is sent as a single PUT; a longer one becomes a multipart upload, which is aborted if the stream fails, so a 2 GB recording never sits in RAM. Uploaded files are stored this way, from their temporary file:

router.post('/recordings', (request) async {
  final video = await request.file('video');
  final path = await video!.store('recordings', disk: 's3');
  return {'path': path};
});

readStream hands the download to the response as it arrives:

return Response.stream(
  Storage.disk('s3').readStream('recordings/$id.mp4'),
  contentType: 'video/mp4',
);

Temporary URLs#

A temporary (presigned) URL lets a client download a private object straight from the bucket until it expires (at most seven days), without the bytes passing through your server:

final url = Storage.disk('s3').temporaryUrl(
  'invoices/1029.pdf',
  expiration: 30.minutes.fromNow,
  responseContentDisposition: 'attachment; filename="invoice-1029.pdf"',
);

The expiry follows Daravel's clock, so Clock.fake() in a test fixes it.

Direct uploads#

For large uploads, a client asks your server for a presigned PUT and sends the file straight to the bucket:

router.post('/api/uploads', (request) async {
  final path = 'materials/${Str.uuid()}.mp4';
  final upload = Storage.disk('s3').temporaryUploadUrl(
    path,
    expiration: 15.minutes.fromNow,
    contentType: 'video/mp4',
  );
  return {'path': path, ...upload.toJson()}; // method, url, headers
});

The client sends the file with exactly the returned headers:

// Flutter, package:http
await http.put(Uri.parse(json['url']), headers: json['headers'], body: bytes);

contentType and contentLength are part of the signature: S3 rejects an upload with another type, or another size when contentLength is given. A presigned PUT cannot cap the size, only fix it.

Capping the size: a POST form#

To let a client upload up to a size, hand out a presigned POST form instead (S3's POST policy). The policy is signed, and S3 checks its conditions on upload: a file over maxSize, a content type that does not start with contentTypePrefix, or any changed field is refused.

router.post('/api/uploads', (request) async {
  final path = 'materials/${Str.uuid()}.mp4';
  final form = Storage.disk('s3').temporaryUploadForm(
    path,
    expiration: 15.minutes.fromNow,
    maxSize: 500 * 1024 * 1024, // 500 MB
    contentTypePrefix: 'video/', // or contentType: 'video/mp4', exactly
  );
  return {'path': path, ...form.toJson()}; // method, url, fields
});

The client sends fields unchanged as multipart/form-data to url, adds Content-Type when a prefix was given, and puts the file last, in a field named file:

// Flutter, package:http
final upload = http.MultipartRequest('POST', Uri.parse(json['url']))
  ..fields.addAll({...json['fields'], 'Content-Type': 'video/mp4'})
  ..files.add(await http.MultipartFile.fromPath('file', video.path));
final response = await upload.send(); // 204 when S3 stored it

A browser does the same with a <form method="post" enctype="multipart/form-data"> or a FormData.

Testing#

Storage.fake('s3') replaces the bucket with an in-memory disk as for any disk. temporaryUrl, temporaryUploadUrl and temporaryUploadForm work on a fake too and return a placeholder (/storage/s3/<path>?expiration=<unix time>), so routes that hand out URLs can be tested without a network:

final disk = Storage.fake('s3');
final response = (await client.post('/api/uploads')).assertOk();
final body = response.json! as Map<String, Object?>;
expect(body['url'], startsWith('/storage/s3/materials/'));
disk.assertMissing(body['path']! as String); // the client uploads it

Against a real server, the package's integration suite runs on RustFS in CI, or on any S3-compatible server: DARAVEL_TEST_S3=http://127.0.0.1:9000 dart test --tags integration in packages/daravel_s3.

On the wire#

A request may take timeout (5 minutes) to send its body and get its response headers, and a response body may go quiet for that long; past it the request is aborted or the body cut off with a ConnectionException (isTimeout), and the socket is not returned to the pool. A call that S3 answers 500, 502, 503 (SlowDown) or 504, or whose connection fails before any response, is sent again up to retries (3) more times after retryDelay (100 ms) doubled each time with jitter, as the AWS SDKs do; every call carries its whole body, so a resend is exact.

Limits#

• copy and move use S3's server-side copy, which takes objects up to 5 GB.
• Credentials come from .env (or AWS_SESSION_TOKEN for temporary ones); instance roles and web identity tokens are not read yet.
• Every part but the last of a multipart upload must be at least 5 MiB; S3Disk(..., partSize: ...) changes the 8 MiB default.

Laravel equivalents#

More of Laravel that this works with, and the API to look up on pub.dev.