Flink Forward 2019 系列文章--k8s 相关(2)--Lyft公司分享管理k8s上的Flink job

Managing Flink on Kubernetes--Anand Swaminathan & Ketan Umare

The goal of Lyft is to “Improve people’s lives with the world’s best transportation”. Our product is fundamentally real-time and building a reliable platform that consumes and processes massive amounts of streaming data empowers us to achieve our mission. The advent of containers and Kubernetes has completely changed how we deploy and manage stateless services. At Lyft we have doubled down on Docker containers and Kubernetes for all the services in production. To achieve a homogenous infrastructure we decided to extend Kubernetes to manage stateful streaming services like Flink. We developed the FlinkK8sOperator which leverages Kubernetes CustomResourceDefinition to enable native management of Flink applications on Kubernetes. FlinkK8sOperator employs a state machine that transitions the application through a series of states, until a stable state is attained. Each Flink application on Kubernetes spins up a separate Flink Cluster, with its own UI, providing clear isolation for monitoring and debugging. This talk provides an overview of running Flink applications on Kubernetes using FlinkK8sOperator, showcasing the entire lifecycle of the application from creation to execution, with focus of transitions during deployments and stateful updates, concluding with a demo.

Lyft的目标是“用世界上最好的交通工具改善人们的生活”。我们的产品基本上是实时的，并且构建了一个可靠的平台，它消耗和处理大量的流数据，使我们能够实现我们的使命。容器和kubernetes的出现完全改变了我们部署和管理无状态服务的方式。在Lyft，我们已经加倍购买码头集装箱和Kubernetes，以提供生产中的所有服务。为了实现同质的基础设施，我们决定扩展kubernetes来管理像Flink这样的有状态的流服务。我们开发了Flink k8Soperator，它利用Kubernetes自定义资源定义来实现对Kubernetes上的Flink应用程序的本地管理。Flink k8Soperator使用一个状态机，通过一系列状态转换应用程序，直到达到稳定状态。Kubernetes上的每个Flink应用程序都会旋转一个单独的Flink集群，并有自己的UI，为监视和调试提供了清晰的隔离。本文概述了使用Flink k8Soperator在Kubernetes上运行Flink应用程序，展示了应用程序从创建到执行的整个生命周期，重点介绍了部署期间的转换和状态更新，最后是演示。