We're sorry but this page doesn't work properly without JavaScript enabled. Please enable it to continue.
Feedback

Fixin the Hard Bits of Event Processing with Restate & Kafka

Formal Metadata

Title
Fixin the Hard Bits of Event Processing with Restate & Kafka
Title of Series
Number of Parts
64
Author
License
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Identifiers
Publisher
Release Date
Language

Content Metadata

Subject Area
Genre
Abstract
Event-driven applications are still hard in 2024, despite all their strong and desirable properties (scalability, resilience, ability to replay, ...). While streaming analytics are supported by a wave of powerful tools (like Flink, KStreams, Materialize, RisingWave, ...), the transactional side of event-driven applications is not quite as far. Complex per-event control flow, complicated state semantics, and interactions with many external dependencies (APIs/services/...) make those applications challenging. Observability and debuggability are still ways behind what RPC-based architectures offer with today's tools. In this talk we present 'Restate' (https://restate.dev/), a new open source tool we have been working on. Restate complements systems like Kafka when building event-driven workflows, async tasks, distributed signaling, or even full microservices architectures. While Restate internally is an event-driven system (sharing some similarities with a Kafka/Flink/KStreams crossover), it has a very simple programming model: Durable Execution (for event-handlers and RPC), Durable Promises (for signaling), and Virtual Objects. Those three constructs are the resilient encapsulations of the core ingredients of every distributed application: compute, communication, and state. In the talk, we will first we'll look practically at common challenging patterns: Where do those tend to break current architectures? How can we address those issues with Kafka/Restate? - burst-y loads / rapid scaling - head-of-the-line waiting on long event processing times - combining Kafka events with webhooks and other signals - Transactionally updating databases - high-fan-out routing to consumer APIs - state machines with long-running transitions - delaying/postponing certain events Second, we take a look at similarities, synergies, and differences between Restate, Kafka, and Stream-Processing. We look at the way applications are expressed, how the systems organize event logs and event processing, and how this mirrors the differences between analytical- and transactional applications respectively.