EmailVault - Email Archiving System

A modern, fast, and secure email archiving solution with powerful search capabilities and mobile-optimized interface.

Features

• 🚀 Lightning Fast - Optimized performance with background processing
• 🔒 Secure - Enterprise-grade encryption and security
• 🔍 Powerful Search - Full-text search across all email content
• 📱 Mobile Optimized - Beautiful responsive design
• 📧 Universal Support - Works with any SMTP/IMAP email provider
• 🔄 Background Processing - Non-blocking email archiving via BullMQ workers
• 📁 Folder Preservation - Maintains original email folder structure
• 🔁 Incremental Sync - Checkpoint-based resume for interrupted archives
• 🔑 Multi-Provider OAuth2 - Azure, Google, Keycloak (OIDC) with automatic token refresh
• 📊 Structured Logging - JSON-formatted logs with context and operation tags
• ⚡ Graceful Shutdown - Configurable timeouts for clean worker termination

Tech Stack

Frontend

• Next.js 15 with App Router
• React 19
• Tailwind CSS + shadcn/ui
• TypeScript
• React Query (TanStack Query)

Backend

• Node.js with Fastify
• Prisma ORM
• BullMQ (job queue framework)
• TypeScript
• Structured logging (JSON)

Database & Storage

• PostgreSQL (with full-text search)
• Valkey (Redis-compatible store for queues & caching)
• RustFS (S3-compatible attachment storage)

Quick Start

Prerequisites

• Docker and Docker Compose
• Node.js 20+ (for local development)

Setup

1. Clone and navigate to the project:

   cd email-vault
   

2. Start all services:

   docker-compose up -d
   

3. Initialize the database:

   # Run database migrations
   docker-compose exec backend npm run db:push
   

4. Access the application:

   • Frontend: http://localhost:3000
   • Backend API: http://localhost:3001
   • RustFS Console: http://localhost:9001 (emailvault / emailvault_password)
   • Nginx (Production): http://localhost:80

Object Storage (RustFS)

EmailVault uses RustFS, a high-performance S3-compatible object storage system written in Rust, for attachment and export storage. The backend communicates via the standard S3 API, so any S3-compatible provider can be used.

• Automated (default with Docker Compose):

  • Both docker-compose.yml and docker-compose.prod.yml include a rustfs-setup one-shot service that:
    • Creates a private bucket named emailvault on first start.
  • Credentials (dev): RUSTFS_ACCESS_KEY=emailvault, RUSTFS_SECRET_KEY=emailvault_password.
  • Endpoint (dev): http://localhost:9000 (console at http://localhost:9001).

• What the backend expects:

  • The attachment worker stores files in the emailvault bucket (see backend/src/workers/attachment-processing.worker.ts).
  • It will also auto-create the bucket at runtime if missing, via ensureBucketExists(). The compose-based setup makes this explicit and deterministic.

• Manual fallback (if you start RustFS without Compose setup):

  1. Use the RustFS Console (http://localhost:9001) or any S3-compatible CLI (e.g., mc, aws s3):

     mc alias set local http://localhost:9000 emailvault emailvault_password
     mc mb -p local/emailvault
     mc anonymous set none local/emailvault
     

Production notes

• In docker-compose.prod.yml set:
  • RUSTFS_ACCESS_KEY, RUSTFS_SECRET_KEY (for the rustfs container).
  • S3_ENDPOINT for the backend (defaults to rustfs:9000).
  • S3_ACCESS_KEY, S3_SECRET_KEY for the backend (should match the RustFS credentials unless you provision separate keys).
  • Optionally override S3_BUCKET used by rustfs-setup (defaults to emailvault).
  • The RustFS console is not exposed by default in prod compose; expose ports if you need console access.
  • Legacy MINIO_ENDPOINT, MINIO_ACCESS_KEY, MINIO_SECRET_KEY env vars are accepted as fallbacks for backward compatibility.

Development

For local development without Docker:

1. Start infrastructure services:

   docker-compose up postgres valkey rustfs -d
   

2. Install dependencies:

   # Backend
   cd backend && npm install
   
   # Frontend
   cd ../frontend && npm install
   

3. Set up environment variables:

   # Backend
   cp backend/.env.example backend/.env
   
   # Frontend  
   cp frontend/.env.example frontend/.env
   

4. Run development servers:

   # Backend (terminal 1)
   cd backend && npm run dev
   
   # Frontend (terminal 2)
   cd frontend && npm run dev
   

Environment Variables

See example.env for a full annotated reference. Key variables:

Backend (.env)

# Core
DATABASE_URL=postgresql://emailvault:emailvault_password@localhost:5432/emailvault
REDIS_URL=redis://valkey:6379
S3_ENDPOINT=rustfs:9000
S3_ACCESS_KEY=emailvault
S3_SECRET_KEY=emailvault_password
JWT_SECRET=your-super-secret-jwt-key-change-in-production
ENCRYPTION_KEY=your-encryption-key

# Worker Concurrency
EMAIL_WORKER_CONCURRENCY=1        # Concurrent archive jobs
ATTACHMENT_WORKER_CONCURRENCY=4   # Concurrent attachment uploads
DELETION_WORKER_CONCURRENCY=2     # Concurrent deletion jobs

# IMAP Timeouts
IMAP_OPERATION_TIMEOUT_MS=30000    # Per-operation IMAP timeout
BATCH_BODY_FETCH_TIMEOUT_MS=60000  # Batch body fetch timeout
PIPELINE_TIMEOUT_MS=300000         # Per-email pipeline timeout

# IMAP TLS (set to "false" for self-signed certs)
IMAP_REJECT_UNAUTHORIZED=true

# OIDC Providers (Azure example)
OIDC_AZURE_CLIENT_ID=...
OIDC_AZURE_CLIENT_SECRET=...
OIDC_AZURE_DISCOVERY_URL=https://login.microsoftonline.com/common/v2.0/.well-known/openid-configuration
OIDC_AZURE_REDIRECT_URI=https://your-domain/api/auth/oidc/azure/callback
OIDC_AZURE_SCOPES=openid email profile offline_access https://outlook.office.com/IMAP.AccessAsUser.All

Frontend (.env.local)

NEXT_PUBLIC_API_URL=http://localhost:3001

Project Structure

email-vault/
├── docker-compose.yml          # Docker services (dev)
├── docker-compose.prod.yml     # Production overrides
├── example.env                 # Annotated env var reference
├── Makefile                    # Dev & prod workflow shortcuts
├── LICENSE                     # MIT license
├── frontend/                   # Next.js 15 frontend
│   ├── src/
│   │   ├── app/               # App Router pages
│   │   ├── components/        # React components
│   │   ├── contexts/          # Auth context (OAuth, token refresh)
│   │   ├── services/          # API service layer
│   │   └── utils/             # Utilities (api-client, etc.)
│   ├── package.json
│   └── Dockerfile
├── backend/                    # Fastify backend
│   ├── src/
│   │   ├── index.ts           # Server entry + graceful shutdown
│   │   ├── app.ts             # Fastify app + health checks
│   │   ├── config/            # Queue & ingestion config
│   │   ├── routes/            # API routes
│   │   ├── services/          # Business logic + storage service
│   │   ├── workers/           # BullMQ workers + manager
│   │   ├── ingestion/         # Email ingestion pipeline
│   │   ├── middleware/        # Auth & security middleware
│   │   └── utils/             # Shared utilities
│   ├── prisma/
│   │   └── schema.prisma      # Database schema
│   ├── package.json
│   └── Dockerfile
├── scripts/
│   └── rustfs-setup.sh        # One-shot bucket creation for RustFS
├── nginx/
│   └── nginx.conf             # HTTPS reverse proxy
└── docs/                      # Documentation

API Endpoints

Authentication

• POST /auth/login - User login
• POST /auth/register - User registration
• POST /auth/refresh - Refresh access token
• DELETE /auth/logout - Logout
• GET /auth/oidc/providers - List available OIDC providers
• GET /auth/oidc/:provider - Initiate OIDC login flow

Email Accounts

• GET /email-accounts - List email accounts
• POST /email-accounts - Add email account
• PUT /email-accounts/:id - Update email account
• DELETE /email-accounts/:id - Remove email account

Archives

• GET /archives - List archives
• POST /archives - Start new archive
• GET /archives/:id - Get archive details
• DELETE /archives/:id - Delete archive

Emails

• GET /emails - Search emails
• GET /emails/:id - Get email details
• GET /emails/:id/attachments - Get email attachments

Architecture

Workers

Worker	Queue	Default Concurrency	Purpose
Email Processing	email-processing	1	IMAP connect, fetch, parse, store, queue attachments
Attachment Processing	attachment-processing	4	Upload attachments to object storage, store metadata
Email Deletion	email-deletion	2	Delete source emails post-archive

All workers use BullMQ with Valkey as the backing store. Concurrency is configurable via environment variables.

Resilience Features

• Per-operation IMAP timeouts prevent indefinite hangs on dropped connections
• Checkpoint-based resume allows interrupted archives to continue from the last processed email
• OAuth2 refresh mutex (Redis lock) prevents concurrent token refresh races
• Proactive token refresh checks expiry before each folder, not just on auth errors
• Failed email tracking records per-UID failures with automatic retry scheduling
• Graceful shutdown with configurable timeout drains workers before exit
• Health check endpoint (/api/health) reports database, storage, queue, and worker status

Frontend Auth Resilience

• Token refresh with exponential backoff retry (max 3 attempts)
• Visibility-aware refresh triggers token check when backgrounded tab regains focus
• createAuthenticatedFetch utility auto-retries on 401 with token refresh

Testing

# Backend tests
cd backend && npm test

# Frontend tests  
cd frontend && npm test

# E2E tests
cd frontend && npm run test:e2e

Security & Safe Email Viewer

• See docs/safe-email-viewer.md for details on the frontend Safe Email Viewer, URL rewriting, and backend proxy endpoints (/api/proxy/image, /api/proxy/link, and attachment proxy routes). It covers sanitization, SSRF protection, allowed content types/size limits, caching, and response security headers.

Production Deployment

1. Copy and configure environment:

   cp example.env .env.prod
   # Edit .env.prod with production secrets and endpoints
   

2. Build and deploy:

   docker compose -f docker-compose.yml -f docker-compose.prod.yml --env-file .env.prod up -d
   

3. Verify health:

   curl -k https://your-domain/api/health
   

Contributing

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Add tests
5. Submit a pull request

License

MIT License - see LICENSE file for details