S3Worm SDK

Turn any S3-compatible storage into a TypeScript file system.

S3Worm (v0.3.3) provides a high-level TypeScript API for S3 operations with JSON schema management, metadata queries, and content-addressed storage.

Installation

npm install s3worm

Quick Start

import { S3Worm } from 's3worm';

const worm = new S3Worm({
  provider: 'storj',
  endpoint: 'https://gateway.storjshare.io',
  accessKeyId: 'your-access-key',
  secretAccessKey: 'your-secret-key',
  bucket: 'my-bucket'
});

// Write a file
await worm.writeFile('/data/config.json', {
  version: '1.0',
  features: ['sync', 'encrypt']
});

// Read a file
const config = await worm.readFile('/data/config.json');

// List files with glob patterns
const files = await worm.listFiles('/data/**/*.json');

// Delete a file
await worm.deleteFile('/data/config.json');

Configuration

interface S3WormConfig {
  provider: 'storj' | 'aws' | 'backblaze' | 'minio' | 'custom';
  endpoint: string;
  accessKeyId: string;
  secretAccessKey: string;
  bucket: string;
  region?: string;
  schema?: string;  // Path to JSON schema file
}

File Operations

Write

// Write text
await worm.writeFile('/docs/readme.md', '# Hello World');

// Write JSON (auto-serialized)
await worm.writeFile('/data/users.json', [
  { id: 1, name: 'Alice' },
  { id: 2, name: 'Bob' }
]);

// Write with metadata
await worm.writeFile('/data/model.bin', buffer, {
  contentType: 'application/octet-stream',
  metadata: { version: '2.0', author: 'Alice' }
});

Read

// Read as string
const content = await worm.readFile('/docs/readme.md');

// Read as JSON
const users = await worm.readFile<User[]>('/data/users.json');

// Read metadata only
const meta = await worm.getMetadata('/data/model.bin');

List

// List all files in a directory
const files = await worm.listFiles('/data/');

// Glob patterns
const jsonFiles = await worm.listFiles('/data/**/*.json');
const images = await worm.listFiles('/assets/*.{png,jpg}');

// With metadata
const detailed = await worm.listFiles('/data/', { includeMetadata: true });

Delete

// Delete a single file
await worm.deleteFile('/data/old-config.json');

// Delete a directory (all files with prefix)
await worm.deleteDirectory('/data/old/');

Metadata Queries

S3Worm supports querying files by their metadata:

const results = await worm.query({
  type: 'training-data',
  created: { $gte: '2024-01-01' },
  size: { $lt: 1000000 }
});

JSON Schema Management

Define schemas for your S3 data:

{
  "models": {
    "User": {
      "path": "/users/(id:uuid)/profile.json",
      "schema": {
        "type": "object",
        "properties": {
          "name": { "type": "string" },
          "email": { "type": "string", "format": "email" }
        }
      }
    }
  }
}

Supported Providers

Provider	Endpoint
Storj	https://gateway.storjshare.io
AWS S3	https://s3.{region}.amazonaws.com
Backblaze B2	https://s3.{region}.backblazeb2.com
MinIO	Your MinIO endpoint
DigitalOcean	https://{region}.digitaloceanspaces.com
Custom	Any S3-compatible endpoint

Architecture

S3Worm sits between your application and the S3 API:

Your App → S3Worm → AWS SDK → S3 Provider
              │
              ├── Schema validation
              ├── Metadata management
              ├── Content addressing (SHA-256)
              └── Query engine