Adblock Compiler — System Architecture

A comprehensive breakdown of the adblock-compiler system: modules, sub-modules, services, data flow, and deployment targets.

Table of Contents

1. High-Level Overview
2. System Context Diagram
3. Core Compilation Pipeline
4. Module Map
5. Detailed Module Breakdown
   • Compiler (src/compiler/)
   • Platform Abstraction (src/platform/)
   • Transformations (src/transformations/)
   • Downloader (src/downloader/)
   • Configuration & Validation (src/configuration/, src/config/)
   • Storage (src/storage/)
   • Services (src/services/)
   • Queue (src/queue/)
   • Diagnostics & Tracing (src/diagnostics/)
   • Filters (src/filters/)
   • Formatters (src/formatters/)
   • Diff (src/diff/)
   • Plugins (src/plugins/)
   • Utilities (src/utils/)
   • CLI (src/cli/)
   • Deployment (src/deployment/)
6. Cloudflare Worker (worker/)
7. Web UI (public/)
8. Cross-Cutting Concerns
9. Data Flow Diagrams
10. Deployment Architecture
11. Technology Stack

High-Level Overview

The adblock-compiler is a compiler-as-a-service for adblock filter lists. It downloads filter list sources from remote URLs or local files, applies a configurable pipeline of transformations, and produces optimized, deduplicated output. It runs in three modes:

System Context Diagram

graph TD
    subgraph EW["External World"]
        FLS["Filter List Sources<br/>(URLs/Files)"]
        WB["Web Browser<br/>(Web UI)"]
        AC["API Consumers<br/>(CI/CD, scripts)"]
    end

    subgraph ACS["adblock-compiler System"]
        CLI["CLI App<br/>(Deno)"]
        WUI["Web UI<br/>(Static)"]
        CFW["Cloudflare Worker<br/>(Edge API)"]
        CORE["Core Library<br/>(FilterCompiler / WorkerCompiler)"]
        DL["Download & Fetch"]
        TP["Transform Pipeline"]
        VS["Validate & Schema"]
        ST["Storage & Cache"]
        DG["Diagnostics & Tracing"]
    end

    KV["Cloudflare KV<br/>(Cache, Rate Limit, Metrics)"]
    D1["Cloudflare D1<br/>(SQLite, Metadata)"]

    FLS --> CLI
    WB --> WUI
    AC --> CFW
    CLI --> CORE
    WUI --> CORE
    CFW --> CORE
    CORE --> DL
    CORE --> TP
    CORE --> VS
    CORE --> ST
    CORE --> DG
    ST --> KV
    ST --> D1

Core Compilation Pipeline

Every compilation—CLI, library, or API—follows this pipeline:

flowchart LR
    A["1. Config<br/>Loading"] --> B["2. Validate<br/>(Zod)"]
    B --> C["3. Download<br/>Sources"]
    C --> D["4. Per-Source<br/>Transforms"]
    D --> E["5. Merge<br/>All Sources"]
    E --> F["6. Global<br/>Transforms"]
    F --> G["7. Checksum<br/>& Header"]
    G --> H["8. Output<br/>(Rules)"]

Step-by-Step

Module Map

src/
├── index.ts                    # Library entry point (all public exports)
├── version.ts                  # Canonical VERSION constant
├── cli.ts / cli.deno.ts        # CLI entry points
│
├── compiler/                   # 🔧 Core compilation orchestration
│   ├── FilterCompiler.ts       #    Main compiler (file system access)
│   ├── SourceCompiler.ts       #    Per-source compilation
│   ├── IncrementalCompiler.ts  #    Incremental (delta) compilation
│   ├── HeaderGenerator.ts      #    Filter list header generation
│   └── index.ts
│
├── platform/                   # 🌐 Platform abstraction layer
│   ├── WorkerCompiler.ts       #    Edge/Worker compiler (no FS)
│   ├── HttpFetcher.ts          #    HTTP content fetcher
│   ├── PreFetchedContentFetcher.ts  # In-memory content provider
│   ├── CompositeFetcher.ts     #    Chain-of-responsibility fetcher
│   ├── PlatformDownloader.ts   #    Platform-agnostic downloader
│   ├── types.ts                #    IContentFetcher interface
│   └── index.ts
│
├── transformations/            # ⚙️ Rule transformation pipeline
│   ├── base/Transformation.ts  #    Abstract base classes
│   ├── TransformationRegistry.ts  # Registry + Pipeline
│   ├── CompressTransformation.ts
│   ├── DeduplicateTransformation.ts
│   ├── ValidateTransformation.ts
│   ├── RemoveCommentsTransformation.ts
│   ├── RemoveModifiersTransformation.ts
│   ├── ConvertToAsciiTransformation.ts
│   ├── InvertAllowTransformation.ts
│   ├── TrimLinesTransformation.ts
│   ├── RemoveEmptyLinesTransformation.ts
│   ├── InsertFinalNewLineTransformation.ts
│   ├── ExcludeTransformation.ts
│   ├── IncludeTransformation.ts
│   ├── ConflictDetectionTransformation.ts
│   ├── RuleOptimizerTransformation.ts
│   ├── TransformationHooks.ts
│   └── index.ts
│
├── downloader/                 # 📥 Filter list downloading
│   ├── FilterDownloader.ts     #    Deno-native downloader with retries
│   ├── ContentFetcher.ts       #    File system + HTTP abstraction
│   ├── PreprocessorEvaluator.ts  # !#if / !#include directives
│   ├── ConditionalEvaluator.ts #    Boolean expression evaluator
│   └── index.ts
│
├── configuration/              # ✅ Configuration validation
│   ├── ConfigurationValidator.ts  # Zod-based validator
│   ├── schemas.ts              #    Zod schemas for all request types
│   └── index.ts
│
├── config/                     # ⚡ Centralized constants & defaults
│   └── defaults.ts             #    NETWORK, WORKER, STORAGE defaults
│
├── storage/                    # 💾 Persistence & caching
│   ├── IStorageAdapter.ts      #    Abstract storage interface
│   ├── PrismaStorageAdapter.ts #    Prisma ORM adapter (SQLite default)
│   ├── D1StorageAdapter.ts     #    Cloudflare D1 adapter
│   ├── CachingDownloader.ts    #    Intelligent caching downloader
│   ├── ChangeDetector.ts       #    Content change detection
│   ├── SourceHealthMonitor.ts  #    Source health tracking
│   └── types.ts                #    StorageEntry, CacheEntry, etc.
│
├── services/                   # 🛠️ Business logic services
│   ├── FilterService.ts        #    Filter wildcard preparation
│   ├── ASTViewerService.ts     #    Rule AST parsing & display
│   ├── AnalyticsService.ts     #    Cloudflare Analytics Engine
│   └── index.ts
│
├── queue/                      # 📬 Async job queue
│   ├── IQueueProvider.ts       #    Abstract queue interface
│   ├── CloudflareQueueProvider.ts  # Cloudflare Queues impl
│   └── index.ts
│
├── diagnostics/                # 🔍 Observability & tracing
│   ├── DiagnosticsCollector.ts #    Event aggregation
│   ├── TracingContext.ts       #    Correlation & span management
│   ├── OpenTelemetryExporter.ts  # OTel bridge
│   ├── types.ts                #    DiagnosticEvent, TraceSeverity
│   └── index.ts
│
├── filters/                    # 🔍 Rule filtering
│   ├── RuleFilter.ts           #    Exclusion/inclusion pattern matching
│   └── index.ts
│
├── formatters/                 # 📄 Output formatting
│   ├── OutputFormatter.ts      #    Adblock, hosts, dnsmasq, etc.
│   └── index.ts
│
├── diff/                       # 📊 Diff reporting
│   ├── DiffReport.ts           #    Compilation diff generation
│   └── index.ts
│
├── plugins/                    # 🔌 Plugin system
│   ├── PluginSystem.ts         #    Plugin registry & loading
│   └── index.ts
│
├── deployment/                 # 🚀 Deployment tracking
│   └── version.ts              #    Deployment history & records
│
├── schemas/                    # 📋 JSON schemas
│   └── configuration.schema.json
│
├── types/                      # 📐 Core type definitions
│   ├── index.ts                #    IConfiguration, ISource, enums
│   ├── validation.ts           #    Validation-specific types
│   └── websocket.ts            #    WebSocket message types
│
├── utils/                      # 🧰 Shared utilities
│   ├── RuleUtils.ts            #    Rule parsing & classification
│   ├── StringUtils.ts          #    String manipulation
│   ├── TldUtils.ts             #    Top-level domain utilities
│   ├── Wildcard.ts             #    Glob/wildcard pattern matching
│   ├── CircuitBreaker.ts       #    Circuit breaker pattern
│   ├── AsyncRetry.ts           #    Retry with exponential backoff
│   ├── ErrorUtils.ts           #    Typed error hierarchy
│   ├── EventEmitter.ts         #    CompilerEventEmitter
│   ├── Benchmark.ts            #    Performance benchmarking
│   ├── BooleanExpressionParser.ts  # Boolean expression evaluation
│   ├── AGTreeParser.ts         #    AdGuard rule AST parser
│   ├── ErrorReporter.ts        #    Multi-target error reporting
│   ├── logger.ts               #    Logger, StructuredLogger
│   ├── checksum.ts             #    Filter list checksums
│   ├── headerFilter.ts         #    Header stripping utilities
│   └── PathUtils.ts            #    Safe path resolution
│
└── cli/                        # 💻 CLI application
    ├── CliApp.deno.ts          #    Main CLI app (Deno-specific)
    ├── ArgumentParser.ts       #    CLI argument parsing
    ├── ConfigurationLoader.ts  #    Config file loading
    ├── OutputWriter.ts         #    File output writing
    └── index.ts

worker/                         # ☁️ Cloudflare Worker
├── worker.ts                   #    Worker entry point
├── router.ts                   #    Modular request router
├── websocket.ts                #    WebSocket handler
├── html.ts                     #    Static HTML serving
├── schemas.ts                  #    API request validation
├── types.ts                    #    Env bindings, request/response types
├── tail.ts                     #    Tail worker (log consumer)
├── handlers/                   #    Route handlers
│   ├── compile.ts              #    Compilation endpoints
│   ├── metrics.ts              #    Metrics endpoints
│   ├── queue.ts                #    Queue management
│   └── admin.ts                #    Admin/D1 endpoints
├── middleware/                  #    Request middleware
│   └── index.ts                #    Rate limit, auth, size validation
├── workflows/                  #    Durable execution workflows
│   ├── CompilationWorkflow.ts
│   ├── BatchCompilationWorkflow.ts
│   ├── CacheWarmingWorkflow.ts
│   ├── HealthMonitoringWorkflow.ts
│   ├── WorkflowEvents.ts
│   └── types.ts
└── utils/                      #    Worker utilities
    ├── response.ts             #    JsonResponse helper
    └── errorReporter.ts        #    Worker error reporter

Detailed Module Breakdown

Compiler (src/compiler/)

The orchestration layer that drives the entire compilation process.

flowchart TD
    FC["FilterCompiler\n← Main entry point (has FS access)"]
    FC -->|uses| SC["SourceCompiler"]
    FC -->|uses| HG["HeaderGenerator"]
    FC -->|uses| TP["TransformationPipeline"]
    SC -->|uses| FD["FilterDownloader"]

Platform Abstraction (src/platform/)

Enables the compiler to run in environments without file system access (browsers, Cloudflare Workers, Deno Deploy).

flowchart TD
    WC["WorkerCompiler\n← No FS access"]
    WC -->|uses| CF["CompositeFetcher\n← Chain of Responsibility"]
    CF --> PFCF["PreFetchedContentFetcher"]
    CF --> HF["HttpFetcher\n(Fetch API)"]

Transformations (src/transformations/)

The transformation pipeline uses the Strategy and Registry patterns.

flowchart TD
    TP["TransformationPipeline\n← Applies ordered transforms"]
    TP -->|delegates to| TR["TransformationRegistry\n← Maps type → instance"]
    TR -->|contains| ST1["SyncTransformation\n(Deduplicate)"]
    TR -->|contains| ST2["SyncTransformation\n(Compress)"]
    TR -->|contains| AT["AsyncTransformation\n(future async)"]

Base Classes:

Built-in Transformations:

Downloader (src/downloader/)

Handles fetching filter list content with preprocessor directive support.

flowchart TD
    FD["FilterDownloader\n← Static download() method"]
    FD -->|uses| CF["ContentFetcher\n(FS + HTTP)"]
    FD -->|uses| PE["PreprocessorEvaluator\n(!#if, !#include)"]
    PE -->|uses| CE["ConditionalEvaluator\n(boolean expr)"]

Configuration & Validation

src/configuration/ — Runtime validation:

src/config/ — Centralized constants:

Storage (src/storage/)

Pluggable persistence layer with multiple backends.

flowchart TD
    ISA["IStorageAdapter\n← Abstract interface"]
    ISA --> PSA["PrismaStorageAdapter\n(SQLite, PostgreSQL, MySQL, etc.)"]
    ISA --> D1A["D1StorageAdapter\n(Edge)"]
    ISA --> MEM["(Memory) — Future"]
    CD["CachingDownloader"] -->|uses| ISA
    SHM["SourceHealthMonitor"] -->|uses| ISA
    CD -->|uses| CHD["ChangeDetector"]

Services (src/services/)

Higher-level business services.

Queue (src/queue/)

Asynchronous job processing abstraction.

flowchart TD
    IQP["IQueueProvider\n← Abstract interface"]
    IQP --> CQP["CloudflareQueueProvider\n← Cloudflare Workers Queue binding"]
    CQP --> CM["CompileMessage\n(single compilation)"]
    CQP --> BCM["BatchCompileMessage\n(batch compilation)"]
    CQP --> CWM["CacheWarmMessage\n(cache warming)"]
    CQP --> HCM["HealthCheckMessage\n(source health checks)"]

Diagnostics & Tracing (src/diagnostics/)

End-to-end observability through the compilation pipeline.

flowchart LR
    TC["TracingContext\n(correlation ID, parent spans)"]
    DC["DiagnosticsCollector\n(event aggregation)"]
    OTE["OpenTelemetryExporter\n(Datadog, Honeycomb, Jaeger, etc.)"]
    TC --> DC
    DC -->|can export to| OTE

Filters (src/filters/)

Formatters (src/formatters/)

Diff (src/diff/)

Plugins (src/plugins/)

Extensibility system for custom transformations and downloaders.

flowchart TD
    PR["PluginRegistry\n← Global singleton"]
    PR -->|registers| P["Plugin\n{manifest, transforms, downloaders}"]
    P --> TPLG["TransformationPlugin"]
    P --> DPLG["DownloaderPlugin"]

Utilities (src/utils/)

Shared, reusable components used across all modules.

CLI (src/cli/)

Command-line interface for local compilation.

See the CLI Reference for the full flag list and examples.

Deployment (src/deployment/)

Cloudflare Worker (worker/)

The edge deployment target that exposes the compiler as an HTTP/WebSocket API.

flowchart TD
    REQ["Incoming Request"]
    REQ --> W["worker.ts\n← Entry point (fetch, queue, scheduled)"]
    W --> R["router.ts\n(HTTP API)"]
    W --> WS["websocket.ts (WS)"]
    W --> QH["queue handler\n(async jobs)"]
    R --> HC["handlers/compile.ts"]
    R --> HM["handlers/metrics.ts"]
    R --> HQ["handlers/queue"]
    R --> HA["handlers/admin"]

API Endpoints

Admin Endpoints (require X-Admin-Key)

Middleware Stack

flowchart LR
    REQ["Request"] --> RL["Rate Limit"]
    RL --> TS["Turnstile"]
    TS --> BS["Body Size"]
    BS --> AUTH["Auth"]
    AUTH --> H["Handler"]
    H --> RESP["Response"]

Durable Workflows

Long-running, crash-resistant compilation pipelines using Cloudflare Workflows:

Environment Bindings

Web UI (public/)

Static HTML/JS/CSS frontend served from Cloudflare Workers or Pages.

Cross-Cutting Concerns

Error Handling

flowchart TD
    BE["BaseError (abstract)"]
    BE --> CE["CompilationError\n— Compilation pipeline failures"]
    BE --> NE["NetworkError\n— HTTP/connection failures"]
    BE --> SE["SourceError\n— Source download/parse failures"]
    BE --> VE["ValidationError\n— Configuration/rule validation failures"]
    BE --> CFE["ConfigurationError\n— Invalid configuration"]
    BE --> FSE["FileSystemError\n— File system operation failures"]

Each error carries: code (ErrorCode enum), cause (original error), timestamp (ISO string).

Event System

The ICompilerEvents interface provides lifecycle hooks:

flowchart TD
    CS["Compilation Start"]
    CS --> OSS["onSourceStart\n(per source)"]
    CS --> OSC["onSourceComplete\n(per source, with rule count & duration)"]
    CS --> OSE["onSourceError\n(per source, with error)"]
    CS --> OTS["onTransformationStart\n(per transformation)"]
    CS --> OTC["onTransformationComplete\n(per transformation, with counts)"]
    CS --> OP["onProgress\n(phase, current/total, message)"]
    CS --> OCC["onCompilationComplete\n(total rules, duration, counts)"]

Logging

Two logger implementations:

Both implement ILogger (extends IDetailedLogger): info(), warn(), error(), debug(), trace().

Resilience Patterns

Data Flow Diagrams

CLI Compilation Flow

flowchart LR
    CFG["config.json"] --> CL["ConfigurationLoader"]
    FS["Filter Sources\n(HTTP/FS)"] --> FC
    CL --> FC["FilterCompiler"]
    FC --> SC["SourceCompiler\n(per src)"]
    FC --> TP["TransformationPipeline"]
    FC --> OUT["output.txt"]

Worker API Flow (SSE Streaming)

sequenceDiagram
    participant Client
    participant Worker
    participant Sources

    Client->>Worker: POST /api/compile/stream
    Worker->>Sources: Pre-fetch content
    Sources-->>Worker: content
    Note over Worker: WorkerCompiler.compile()
    Worker-->>Client: SSE: event: log
    Worker-->>Client: SSE: event: source-start
    Worker-->>Client: SSE: event: source-complete
    Worker-->>Client: SSE: event: progress
    Note over Worker: Cache result in KV
    Worker-->>Client: SSE: event: complete

Async Queue Flow

sequenceDiagram
    participant Client
    participant Worker
    participant Queue
    participant Consumer

    Client->>Worker: POST /compile/async
    Worker->>Queue: enqueue message
    Worker-->>Client: 202 {requestId}
    Queue->>Consumer: dequeue
    Consumer->>Consumer: compile
    Consumer->>Queue: store result
    Client->>Worker: GET /queue/results/:id
    Worker->>Queue: fetch result
    Worker-->>Client: 200 {rules}

Deployment Architecture

graph TD
    subgraph CFN["Cloudflare Edge Network"]
        subgraph CW["Cloudflare Worker (worker.ts)"]
            HAPI["HTTP API Router"]
            WSH["WebSocket Handler"]
            QC["Queue Consumer\n(async compile)"]
            DWF["Durable Workflows"]
            TW["Tail Worker"]
            SA["Static Assets\n(Pages/ASSETS)"]
        end
        KV["KV Store\n- Cache\n- Rates\n- Metrics"]
        D1["D1 (SQL)\n- Storage\n- Deploy\n- History"]
        QQ["Queues\n- Std\n- High"]
        AE["Analytics Engine"]
    end

    CLIENTS["Clients\n(Browser, CI/CD, CLI)"] -->|HTTP/SSE/WS| HAPI
    HAPI -->|HTTP fetch sources| FLS["Filter List Sources\n(EasyList, etc.)"]

Technology Stack