Real-Time Data Processing: Kafka, Flink, and the Modern Streaming Stack

Course Description

Real-time data processing has become a foundational capability across virtually every sector of the software industry. From fraud detection and personalized recommendations to operational dashboards and machine learning feature pipelines, organizations that can act on data in motion - rather than data at rest - gain decisive advantages in speed, relevance, and reliability.

This course provides a comprehensive, hands-on introduction to real-time data processing using the industry-standard open source stack: Apache Kafka for event streaming, Apache Flink for stream processing, and Apache Iceberg for data lakehouse integration. Students will build working pipelines entirely on their laptops using Docker-based local environments, eliminating the need for cloud accounts or paid infrastructure. The course progresses from foundational concepts, including the differences between batch processing and RPC-style computing, and progresses to production-grade patterns including stateful processing, event-time semantics, real-time OLAP analytics, and reliable delivery to open-format data lakehouses. The final module introduces AI agents as operational tools, giving students hands-on experience using AI assistants to monitor, debug, and operate streaming applications - a skill increasingly expected in modern data engineering roles.

Learning Outcomes

At the conclusion of the course, you should be able to

• Explain the architectural differences among batch processing, RPC-style compute, and stream processing, and identify appropriate use cases for each paradigm in real-world data system design.
• Design and implement event-driven data pipelines using Apache Kafka for event transport and Apache Flink for stateful stream processing, applying concepts including partitioning, consumer groups, windowing, and event-time semantics.
• Build end-to-end streaming pipelines that ingest, process, and deliver data to downstream systems including real-time OLAP stores and Apache Iceberg-based data lakehouses.
• Use browser developer tools to inspect, debug, and optimize layouts.
• Monitor and operate streaming applications using observability tools and AI-assisted workflows, including diagnosing common failure modes such as consumer lag, checkpoint failures, and late data.
• Evaluate architectural trade-offs, including throughput versus latency, exactly-once versus at-least-once delivery, and stream versus micro-batch and apply them to design decisions appropriate to specific business requirements.

Topics Include

• The streaming paradigm: how real-time processing differs from batch and RPC
• Event streaming with Apache Kafka: producers, consumers, topics, partitions, consumer groups
• Stream processing with Apache Flink: stateful computation, windowing, event-time processing
• FlinkSQL: SQL interface for expressing Flink streaming jobs - table API, continuous queries, temporal joins
• Downstream serving: real-time OLAP engines and Apache Iceberg data lakehouse integration
• End-to-end pipeline design: schema evolution, Kafka Connect, advanced monitoring and observability, path to production
• AI and Flink: AI agents for Flink operations, and Flink as infrastructure for AI agent systems

Skills Needed

• Programming proficiency in Python or Java
• Basic SQL (SELECT, WHERE, GROUP BY)
• Familiarity with command-line / terminal usage
• No prior experience with Kafka, Flink, or distributed systems required