Case Study: Real-Time Event Streaming Platform for Market Risk

Capital Markets

Real-Time Event Streaming Platform for Market Risk

The Challenge: Batch-Based Risk in a Real-Time Market

A mid-sized investment bank operating across equities, fixed income, and derivatives had a market risk infrastructure that was fundamentally incompatible with the demands of modern trading. End-of-day batch processing produced risk positions 12 to 16 hours after markets closed. Daily VaR calculations took over six hours to complete, and intraday limit breaches were invisible until the following morning's risk report. As Basel IV requirements tightened intraday reporting obligations and regulators began scrutinizing the bank's ability to monitor real-time exposures, the gap became a compliance liability. Simultaneously, the front office was losing business to competitors who could provide traders with live P&L attribution and real-time risk analytics — capabilities the existing architecture could not deliver without a fundamental rebuild. Three prior attempts to extend the batch system incrementally had failed to solve the underlying latency problem.

Our Solution: Apache Kafka and Flink Streaming Platform

Lydatum designed and delivered a greenfield event streaming platform built on Apache Kafka and Apache Flink, replacing the batch risk pipeline with a continuous, event-driven architecture capable of processing market data and position updates in real time:

• Event Streaming Backbone: We deployed a 24-broker Apache Kafka cluster with cross-datacenter replication for resilience, handling inbound market data from Bloomberg B-PIPE and Reuters Elektron alongside internal trade event streams from six trading systems. Kafka Schema Registry enforced message contracts across all producers, eliminating schema drift issues that had plagued previous integration efforts.
• Real-Time Risk Computation: Apache Flink stateful streaming jobs computed rolling position aggregations, P&L attribution, sensitivities (delta, gamma, vega), and VaR estimates continuously as trade and market events arrived. Computation latency from trade event to updated risk position averaged under 200 milliseconds across the full book.
• Risk Analytics and Alerting: ClickHouse served as the analytical store for time-series risk data, enabling sub-second queries across billions of risk snapshots. Grafana dashboards provided real-time risk visibility for traders, risk managers, and the CRO. Automated limit breach alerting notified risk managers within seconds of a threshold being crossed — replacing the next-morning discovery process.
• Observability and Reliability: Full pipeline observability was implemented using Prometheus and Grafana, with consumer lag monitoring, throughput dashboards, and automated failover procedures. The platform was designed for 99.99% availability during market hours, with chaos engineering tests validating recovery behavior under broker and datacenter failure scenarios.

The Impact: Real-Time Risk Visibility Across the Full Book

The platform went live in production for equities and rates desks after a four-month build, with derivatives coverage added in a subsequent phase. Risk managers described the transition from daily batch reports to live dashboards as transformational.

The bank passed its Basel IV intraday risk monitoring assessment without remediation requirements — the first time in three examination cycles. The front office cited real-time P&L attribution as a key factor in retaining two institutional trading relationships that had been at risk. The platform's throughput headroom supported a 40% increase in trading volume in the 12 months following launch without infrastructure changes.

Services Delivered: Real-time Streaming & Events, Data Pipeline & Integration, Data Quality & Observability, Data Architecture & Modeling

Technologies Used: Apache Kafka, Apache Flink, ClickHouse, Grafana, Prometheus, Bloomberg B-PIPE, Reuters Elektron