Real-Time Streaming Pipeline

The Challenge

The business needed near-instant visibility into user events, but the existing batch ETL only refreshed every six hours. By the time anomalies or fraud signals surfaced in dashboards, the window to act on them had already closed.

The Approach

The redesign focused on streaming data continuously rather than batching it, without sacrificing correctness at scale.

• Stood up Apache Kafka topics partitioned by event type to handle 50M+ events a day with room to scale further.
• Wrote Python producers and consumers with schema validation to keep malformed events out of downstream tables.
• Used Spark Structured Streaming on Databricks to continuously ingest into Delta Lake with exactly-once semantics.
• Tuned micro-batch intervals and partitioning to consistently hit sub-second end-to-end latency.

The Results

Teams that used to wait for the next batch window could now see — and react to — events as they happened.

Key Takeaways

Exactly-once semantics and schema validation aren't optional at this scale — they're what let the business actually trust a real-time number enough to act on it. Sub-second latency means nothing if the data underneath it is wrong.