Cloudflare Services Integration

This document describes all Cloudflare services integrated into the adblock-compiler project, their current status, and configuration guidance.

Service Status Overview

ServiceStatusBindingPurpose
KV Namespaces✅ ActiveCOMPILATION_CACHE, RATE_LIMIT, METRICSCaching, rate limiting, metrics aggregation
R2 Storage✅ ActiveFILTER_STORAGEFilter list storage and artifact persistence
D1 Database✅ ActiveDBCompilation history, deployment records
Queues✅ ActiveADBLOCK_COMPILER_QUEUE, ADBLOCK_COMPILER_QUEUE_HIGH_PRIORITYAsync compilation, batch processing
Analytics Engine✅ ActiveANALYTICS_ENGINERequest metrics, cache analytics, workflow tracking
Workflows✅ ActiveCOMPILATION_WORKFLOW, BATCH_COMPILATION_WORKFLOW, CACHE_WARMING_WORKFLOW, HEALTH_MONITORING_WORKFLOWDurable async execution
Hyperdrive✅ ActiveHYPERDRIVEAccelerated PostgreSQL (PlanetScale) connectivity
Tail Worker✅ Activeadblock-compiler-tailLog collection, error forwarding
SSE Streaming✅ Active—Real-time compilation progress via /compile/stream
WebSocket✅ Active—Real-time bidirectional compile via /ws/compile
Observability✅ Active—Built-in logs and traces via [observability]
Cron Triggers✅ Active—Cache warming (every 6h), health monitoring (every 1h)
Pipelines✅ ConfiguredMETRICS_PIPELINEMetrics/audit event ingestion → R2
Log Sink (HTTP)✅ ConfiguredLOG_SINK_URL (env var)Tail worker forwards to external log service
API Shield📋 Dashboard—OpenAPI schema validation at edge (see below)
Containers🔧 ConfiguredADBLOCK_COMPILERDurable Object container (production only)

Cloudflare Pipelines

Pipelines provide scalable, batched HTTP event ingestion — ideal for routing metrics and audit events to R2 or downstream analytics.

Setup

# Create the pipeline (routes to R2)
wrangler pipelines create adblock-compiler-metrics-pipeline \
  --r2-bucket adblock-compiler-r2-storage \
  --batch-max-mb 10 \
  --batch-timeout-secs 30

Usage

The PipelineService (src/services/PipelineService.ts) provides a type-safe wrapper:

import { PipelineService } from '../src/services/PipelineService.ts';

const pipeline = new PipelineService(env.METRICS_PIPELINE, logger);

await pipeline.send({
    type: 'compilation_success',
    requestId: 'req-123',
    durationMs: 250,
    ruleCount: 12000,
    sourceCount: 5,
});

Configuration

The binding is defined in wrangler.toml:

[[pipelines]]
binding = "METRICS_PIPELINE"
pipeline = "adblock-compiler-metrics-pipeline"

Log Sinks (Tail Worker)

The tail worker (worker/tail.ts) can forward structured logs to any HTTP log ingestion endpoint (Better Stack, Grafana Loki, Logtail, etc.).

Configuration

Set these secrets/environment variables:

wrangler secret put LOG_SINK_URL       # e.g. https://in.logs.betterstack.com
wrangler secret put LOG_SINK_TOKEN     # Bearer token for the log sink

Optional env var (defaults to warn):

wrangler secret put LOG_SINK_MIN_LEVEL  # debug | info | warn | error

Supported Log Sinks

ServiceLOG_SINK_URLAuth
Better Stackhttps://in.logs.betterstack.comBearer token
Logtailhttps://in.logtail.comBearer token
Grafana Lokihttps://<host>/loki/api/v1/pushBearer token
Custom HTTPAny HTTPS endpointBearer token (optional)

API Shield

Cloudflare API Shield enforces OpenAPI schema validation at the edge for all requests to /compile, /compile/stream, and /compile/batch. This is configured in the Cloudflare dashboard — no code changes are required.

Setup

  1. Go to Cloudflare Dashboard → Security → API Shield
  2. Click Add Schema and upload docs/api/cloudflare-schema.yaml
  3. Set Mitigation action to Block for schema violations
  4. Enable for endpoints:
    • POST /compile
    • POST /compile/stream
    • POST /compile/batch

Schema Location

The OpenAPI schema is at docs/api/cloudflare-schema.yaml (auto-generated by deno task schema:cloudflare).

Analytics Engine

The Analytics Engine tracks all key events through src/services/AnalyticsService.ts. Data is queryable via the Cloudflare Workers Analytics API.

Tracked Events

EventDescription
compilation_requestEvery incoming compile request
compilation_successSuccessful compilation with timing and rule count
compilation_errorFailed compilation with error type
cache_hit / cache_missKV cache effectiveness
rate_limit_exceededRate limit hits by IP
workflow_started / completed / failedWorkflow lifecycle
batch_compilationBatch compile job metrics
api_requestAll API endpoint calls

Querying

-- Average compilation time over last 24h
SELECT
  avg(double1) as avg_duration_ms,
  sum(double2) as total_rules
FROM adguard-compiler-analytics-engine
WHERE timestamp > NOW() - INTERVAL '1' DAY
  AND blob1 = 'compilation_success'

D1 Database

D1 stores compilation history and deployment records, enabling the admin dashboard to show historical data.

Schema

Migrations are in migrations/. Apply with:

wrangler d1 execute adblock-compiler-d1-database --file=migrations/0001_init.sql --remote
wrangler d1 execute adblock-compiler-d1-database --file=migrations/0002_deployment_history.sql --remote

Workflows

Four durable workflows handle crash-resistant async operations:

WorkflowTriggerPurpose
CompilationWorkflow/compile/asyncSingle async compilation with retry
BatchCompilationWorkflow/compile/batchPer-item recovery for batch jobs
CacheWarmingWorkflowCron (every 6h)Pre-populate KV cache
HealthMonitoringWorkflowCron (every 1h)Check source URL health

References