Beyond Kafka: Cutting Costs and Complexity with WarpStream and S3

Apache Kafka powers many event and ETL pipelines, but it is notoriously difficult and expensive to operate in the cloud. Whether you’re dealing with a high-throughput workload that costs an arm and a leg in cross-AZ data transfer, a long retention duration that explodes your EBS bill, or a cluster that pages you every time a node fails because you need to rebalance partitions: Kafka leaves you without many options. This workshop will explain how building streaming storage on top of object storage like Amazon S3 mitigates all three of these pain points, and how we built WarpStream to package these ideas into an Apache Kafka protocol-compatible interface. We’ll benchmark a WarpStream cluster running live with a workload of hundreds of MiB/s of ingestion and compare the TCO to running open-source Apache Kafka yourself.