Light
Dark
All diagrams ← Technical architecture Sync lifecycle →
Candid Leap · WF-Algolia

Deep Dive — Backend

Sync pipeline
API & data access
Ingress & auth Express 5.1 · Node ≥20 · Railway

Rate limits

/auth 10/min · /api 60/min · /internal 100 req/s · /api/events 120/min per-site

JWT session

Designer idToken exchanged at POST /auth/resolve-id-token → HS256 JWT, 60-min TTL; assertSiteMatch blocks cross-site (IDOR) access

Internal ingress

/internal/sync-trigger gated by verifySharedSecret (x-shared-secret header) — only the CF queue consumer calls it

Route groups

/auth

resolve-id-token (token exchange)

/api/sites

site CRUD + sync control: sync-now, force-reindex, sync-cost-preview

/api/mappings

collection-mapping CRUD

/api/events

analytics proxy (extension events → PostHog)

/internal/sync-trigger

queue consumer ingress

Services

syncWorker

1.5 s poll, atomic claim (findOneAndUpdate), MAX_ATTEMPTS 5 → dead_letter, reaper re-queues jobs with heartbeat older than 10 min

reconcileService

3-phase sync: fetch → transform → save, resumable via ReconcileCursor checkpoints

webflowService

items/live pagination, 100 per page

transformService

applies field mappings; oversize records ~10 KB truncated/skipped, hard-skip >100 KB

encryption

AES-256-GCM, v1:iv:tag:ciphertext; boot guard assertCredentialsEncrypted

algoliaService

batched writes ≤1,000 ops / 900 KB per batch, backoff on 429

cronSyncService

hourly tick (per-mapping schedules) · daily 03:00 ET stale-site + webhook-health sweeps · DLQ sweep every 5 min

dlqSweepService

pulls DLQ via Cloudflare Queues REST API (pull/ack), re-posts to /internal/sync-trigger

posthogService

classifyError → 7 ErrorClass values; typed capture()

postmarkService

welcome · stale-site · stale-Algolia-key · sync-failure alert emails (7-day per-alert cooldown)

Data — MongoDB Atlas

Site

SiteConfig

CollectionMapping

SyncJob

lockKey partial-unique on pending/in_progress collapses duplicate triggers

ReconcileCursor

Algolia objectID = itemId_cmsLocaleId
External calls

Webflow Data API

OAuth token, encrypted at rest

Algolia Admin API

admin key server-side only

PostHog

Postmark

Guarantees

Idempotent

duplicate webhook triggers collapse on lockKey

Fail-fast env

Zod schema: JWT_SECRET ≥32 chars, ENCRYPTION_KEY 64 hex