Hey everyone! We’re evaluating different vendor solutions to run stateful IoT data processing workloads on Flink in our GCP environment. We’re a small team with limited ops experience, so a managed or low-overhead option would be ideal. Does anyone have insights or experiences with the following (or other) solutions? • Google Dataproc for Flink (and any tips on managing stateful workloads) • Ververica BYOC (with upcoming GCP support) • Cloudera Stream Processing • Confluent Cloud - Flink How well do these options integrate with GCP services, and how much operational overhead do they typically require? Any guidance on the best fit (preferably a managed service that could integrate with GCP) would be greatly appreciated! Thanks

Hello there! Just to understand some constraints I'll ask some questions and we move forward 😄 What is the stateful size we are talking about? It would be nice to also know throughput and resource utilization. I've got experience with: • Ververica • decodable • OnPrem (gcp k8s) It would also be good to understand what are your pipelines like. Are you using DatastreamAPI, TableAPI, or Flink SQL?

disclaimer: I work for Confluent, and I used to work for Decodable. • Apache Flink on Confluent Cloud is available on GCP https://docs.confluent.io/cloud/current/flink/reference/cloud-regions.html#gc-supported-regions • Decodable is only available on AWS: https://docs.decodable.co/reference/regions-and-ip-addresses.html Where is your IoT data coming from in GCP; is your processing consuming it directly from the devices, or via something like a Kafka topic?

@Pedro Mázala Is there a limit on the size of the state? In our situation, we require state management for a couple of thousand keys, with each key holding several integer values. This would roughly amount to a few gigabytes of data, and we anticipate this requirement to increase over time. I am also interested in knowing if there are any absolute limits on the state size. Given the strong need for stateful computations in our use case, the core application logic must have direct access to low-level process APIs and datastream APIs. @rmoff The data ingestion service is responsible for writing the streaming data from our camera devices into a Pub/Sub topic on GCP. Our team has limited experience with infrastructure management, so we want to evaluate the operational overhead associated with managing Flink clusters, resource allocation, and overseeing the execution lifecycle in this scenario. After considering several options, both the Ververica Platform and Google’s GKE combined with the Flink Kubernetes Operator appears a close match since we need some integration with GCP services. Below are some of the key questions we need to address for both Ververica and the GKE with Flink Kubernetes Operator orchestration, which can be grouped into two main categories: Cluster Operational Complexity and Pricing. Ververica Platform • Developer Perspective: ◦ What are its main advantages and drawbacks from an application developer’s point of view? ◦ Which components are fully managed versus those that require user intervention? ▪︎ Key Areas of Focus: • State management and tuning • Storage management • Application deployment lifecycle • Failure and recovery strategies • Cluster capacity scaling • Upgrades • Security (SSL/TLS) • IAM • Operational Complexity: ◦ What is the overall cluster operational complexity? ◦ Which cluster-specific tasks are handled by the team? ◦ Can you provide insights into their pricing for cloud offerings? GKE with Flink Kubernetes Operator • Developer Perspective: ◦ What are the main challenges, such as scaling, state management, and checkpointing? ◦ What do you see as the main pros and cons in terms of cluster operational complexity? ◦ Which parts are fully managed and which parts require user management? ▪︎ Key Areas of Focus: • State management and tuning • Storage management • Application deployment lifecycle • Failure and recovery strategies • Cluster capacity scaling • Upgrades • Security (SSL/TLS) • IAM • Operational Complexity: ◦ How steep is the learning curve for setting up and maintaining Kubernetes-based Flink deployments? ◦ Beyond application development, what cluster administration activities are necessary? ◦ How do you handle upgrades, cluster scaling, application failure recovery ? ◦ Is a dedicated team needed to handle these administrative and operational tasks? Or is this setup manageable by an application developer ? The overall goal is to compare each product’s operational and administrative overhead, understand their pricing models, and clarify how responsibilities are divided between application development and cluster management.

FWIW If you wanted to do this on Confluent Cloud, you'd create a connector to get the data from Pub/Sub into a topic on Confluent Cloud, and then you'd have access to it for your Flink processing.

Hiya! I'm from Ververica :) VVP and VVC are similar but different due to deployment & management options. Happy to walk you through the differences & advantages

Is there a limit on the size of the state? In our situation, we require state management for a couple of thousand keys, with each key holding several integer values. This would roughly amount to a few gigabytes of data, and we anticipate this requirement to increase over time. I am also interested in knowing if there are any absolute limits on the state size.

There is no limit. As bigger it gets, as time consuming it is to load everything in the case of a restart. Some companies have terabytes of data on Flink. Since @Ben Gamble is here, I'll take the

Operational Complexity

questions.

• How steep is the learning curve for setting up and maintaining Kubernetes-based Flink deployments?

Doing it for the first time may bring you some issues here and there. The community has quite some extended materials around how to find your way. But it may be a long way depending on how much knowledge the team has around distributed systems deployments.

• Beyond application development, what cluster administration activities are necessary?

I'd say upgrades; consistency and issues when the usage grows. It's almost the same effort as keeping kafka in your infra.

• How do you handle upgrades, cluster scaling, application failure recovery ?

It all depends on the taste of your organization. You may have different versions running; single version; single cluster; multiple clusters. Anything here is achievable with the right effort. On the failure recovery side it depends on the failure. There are platform failures and pipeline failures (IMO).

• Is a dedicated team needed to handle these administrative and operational tasks? Or is this setup manageable by an application developer ?

Companies either rely on a specific team to do so or hire companies/services to provide such service. Imagine you will have on duty rotation to be sure the cluster is up and healthy; upgrades that could jeopardize the regular flow of development. Many undiscovered shenanigans related to the data being processed. Disclaimer: I'm from evoura. We offer help from starting the first Flink pipeline to productionizing the workload. And we offer help to either have it within your infra (k8s operator) or throughout any provider.

I'm running Flink on GKE to handle streaming data pipeline to process 100k events/second

Very cool!

@Pedro Mázala @Ben Gamble I hope you’ve been doing well. Thanks for your messages. Please accept my sincere apologies for the delayed response. I’ve been away on vacation for the past two weeks. My team is currently assessing various vendor solutions (GKE self-managed vs. vendor-managed) against several criteria to determine which best meets our needs. As part of this process, we’re exploring each option in detail to understand its complexities, so we’ll need a bit more time to gain a comprehensive understanding before deciding whether we require support from your company. Rest assured, if we do need your assistance, I’ll be in touch. Thank you! 🙌