This guide covers a reference architecture, concrete SLOs, and on-call playbooks for three core pieces in operator terms: identity resolution as a service, the event pipeline as the backbone, and reverse ETL as activation.
Core Requirements
Real-time should mean a number, not a vibe:
- Collection-to-activation under 2 seconds for priority paths
- Under 30 seconds for bulk or low-priority events
- Sub-second to a few seconds for onsite personalization
- Within minutes for sales alerts
- Within hours or next day for audience builds and modeling
Aim for 80–95% coverage by channel and reserve hard latency targets for 5–10 key journeys — not everything.
Identity Resolution Backbone
Three identifier layers: device IDs, web cookies, person/account IDs. Make deterministic vs probabilistic match rules explicit in code, not slides. Architecture:
- Real-time ID service consuming events and updating the graph in milliseconds
- Offline batch jobs for backfills, merges, and replays
- Priority order: user_id over email over cookie_id over IP+UA
Target match rates: 95–99% person-level for logged-in events; 85–95% for email-based with verification; 40–70% for cookie or device-only. Track identity incident metrics like sudden profile-per-user spikes and merge reversals — when those move fast, you want alerts and a playbook.
Event Pipeline You Can Debug
SDKs and server collectors → message bus (Kafka/Kinesis) → stream processors → storage → routers. Segment by priority — checkout and key B2B intent ride the fast lane; content views and email opens accept more latency.
Use versioned schemas with required fields and clear producer/consumer contracts. Design for 3–5x peak. SLOs by priority: P1 under 500ms through the pipeline, lower-priority under 5 minutes end-to-end, error budget under 0.1% for P1.
Reverse ETL Without Surprises
Source of truth (warehouse, lakehouse, or CDP profiles) → transformation → sync connectors → CRM, MAP, ad platforms, offline. Mix streaming syncs with microbatches:
- Sales alerts, fraud, cart abandonment: near-real-time API events
- LTV segments, churn models: every 15–60 minutes
Freshness targets: CRM behavior under 15 minutes (firmographics daily), hot ad audiences under 60 minutes. Protect with idempotency keys, consistent customer_id mapping across destinations, and field-level contracts.
SLOs, SLAs, and On-Call Playbooks
Draw a hard line between internal SLOs (99.5% of checkout events under 1s) and external SLAs (audiences update under 30 minutes 99% of the time). Cover identity (resolve under 100ms P95, merge error 1–2% or lower), events (end-to-end latency per priority, ingestion success), and reverse ETL (sync latency by destination, retry behavior).
Write playbooks for event spikes, identity meltdowns, and sync failures. Each should cover detection, 5–10 minute triage, containment (throttling, feature flags), and who in marketing/sales/product to notify.
Turn This Into a Q3 Plan
- 30 days: inventory identifiers, events, syncs; document current latencies and failure modes; write a basic SLO sheet
- 60 days: tighten schemas and contracts; stand up identity and pipeline dashboards; add the top three alerts
- 90 days: pilot real-time paths for one or two high-value journeys; run a planned chaos exercise before peak
Next Step
If you can't name your latency targets, match rates, and error budgets, start there before buying another tool. See our real-time customer data platform, or book a demo.