WF-Algolia — Technical Architecture

Webflow — customer environmentDesigner, published site, platform APIs

Designer Extension

React 18 panel inside the Designer — onboarding, field mapping, sync health, Build Mode.

JWT HS256React Query v5

Data + CDN API

CMS collections, live-item pagination and webhook registration.

OAuth 2.0REST v2

Webhooks

site_publish, item published / unpublished / deleted.

HMAC-SHA256

Published Site

Customer's live site. Loads the search script from the CDN via a script tag.

visitors

Search Script

Attribute-driven search, browse and filter UI. No customer JavaScript required.

search-only keyno telemetry

Cloudflare — edge ingressDecouples Webflow from the backend, absorbs bursts

Worker — ingress

Verifies the webhook signature (±5 min replay window), then reduces events to ID-only SyncTrigger messages — content never enters the queue.

HMAC verifyIDs only

Queue — webflow-sync-production

Durable, at-least-once delivery to the backend consumer.

5 retries30-min delay14-day retention

Dead-letter queue

Events that exhausted queue retries; retained and replayed automatically.

swept every 5 min

Application — RailwaySingle always-on Node 20 service, in-process workers

Express API

OAuth + JWT auth, site and mapping CRUD, sync control, analytics proxy. Internal ingress gated by a shared secret.

rate-limitedZod-validatedshared secret60-min JWT

Sync Worker

Claims jobs atomically from MongoDB (1.5 s poll). Three-phase reconcile: fetch → transform → save, with per-page checkpoints.

5 attempts → dead_letterstuck jobs auto-recover in 10 min

Cron Scheduler

Hourly per-collection scheduled syncs; drives the always-on health monitors below.

in-processhourly + 5-min + daily ticks

Data & searchOperational store and customer-owned indices

MongoDB Atlas

Site · SiteConfig · CollectionMapping · SyncJob · ReconcileCursor.

AES-256-GCM credentialsUS region

Algolia

Customer-owned search indices. Writes batched at ≤1,000 ops / 900 KB with rate-limit backoff.

sync key server-side only

Always-on health monitorsEvery failure mode detected, self-healed, or escalated to the customer — 24/7

Pipeline Heartbeat

A liveness beat every 5 minutes carrying dead-letter counts. Thirty minutes of silence raises an alert — a dead-man's switch on the whole pipeline.

every 5 minalert on silence

Scheduled-Sync Retry

A failed scheduled sync retries automatically after 5 minutes; a second consecutive failure pauses the schedule and emails the customer.

auto-retry +5 minemail on repeat

Dead-Letter Sweep

Pulls dead-lettered events every 5 minutes and replays them into the pipeline; sites that keep failing are flagged for attention.

every 5 minauto-replay

Webhook Health

Daily reconcile of every site's webhook registrations against Webflow — dropped subscriptions are re-created, orphans removed.

dailyself-healing

Webflow Token Monitor

Daily probe of every site's access — detects revoked tokens and app uninstalls, flags the site, and notifies the customer by email.

dailycustomer email

Algolia Key Monitor

A revoked or under-scoped key is caught instantly on sync (401/403) and by a daily sweep — schedules pause safely and the customer is emailed; recovery un-pauses exactly what was paused.

instant + dailypause + email

Technical Architecture