Real-Time Stream Processing Architectures for High-Frequency Financial Data: Latency-Throughput Trade-offs in Apache Flink, Apache Spark Streaming, and Apache Storm
High-frequency financial data processing -- encompassing market tick data, order book events, and payment transaction streams -- imposes latency and throughput requirements at the boundary of what commodity stream processing frameworks can sustain, making architectural choices consequential for both business outcomes and infrastructure cost. This paper presents a rigorous comparative evaluation of three leading distributed stream processing frameworks -- Apache Flink, Apache Spark Structured Streaming, and Apache Storm -- under financial workload conditions. We design a benchmark suite comprising three representative financial workloads: sub-millisecond tick data aggregation, real-time fraud detection over payment event streams, and order book reconstruction with market microstructure analytics. Benchmarks are executed on standardized 24-node clusters across AWS, simulating peak trading session loads of up to 8 million events per second. Apache Flink achieves the lowest median end-to-end latency at 3.2ms for tick aggregation, compared to 12.1ms for Spark Structured Streaming and 8.7ms for Storm. Spark achieves the highest sustained throughput at 11.2M events/second before degradation. We introduce the Stream Processing Fitness Score (SPFS) that aggregates latency percentiles, throughput ceiling, fault recovery time, and operational complexity. We also characterize watermarking strategies, state backend selection, and checkpointing frequency as the three most impactful configuration decisions affecting latency under production conditions.