guys, how do you manage flink environment variables. it seems the flink environment has a file that it copies into /opt/flink/conf in the current available containers. need to have a look at the startup script, have a plan how to create a flink-conf.yaml on container start.

The Open Source Analytics Conference - free and online Explore the latest in data ingestion, orchestration, databases, infrastructure, governance, visualization, and AI. Date: Nov 19-21 (if you cannot make it, recordings get sent to registrants) Time: 8am - 1pm Pacific Price: Free 🙌 Register: osacon.io

Hi there, is anyone here using

EXPLAIN ESTIMATED_COST

to get an estimate for resource usage of a given Flink SQL query? If so how do you interpret the values given for cpu, memory, io etc? Results look like this.

== Optimized Physical Plan ==
TableSourceScan(..., cumulative cost ={1.0E8 rows, 1.0E8 cpu, 2.4E9 io, 0.0 network, 0.0 memory})

We have run some experiments, comparing the explained costs to actually observed resource consumption and comparing explained costs for different query modification but so far the values make little sense to us 😄

Is there a good way to remove "old" files from the Flink file source state? We are reading from S3 and move data out periodically to another archive bucket.

Hi, we're looking into upgrading our jobs to flink 1.19, we run on k8s and looking at the compatibility matrix for flink-k8s-operator I see that 1.9.0 only supports flink 1.18, is there a target release date for flink-k8s-operator 1.10 yet? Thanks!

I know queryable state has been marked as deprecated - do we yet know what will take it's place, I can't seem to find anything with any details online?

You are 9 hours earlier so at 9am there its basically around midnight here. I am usually up until 2am (I know its not a great schedule)

Hi, Does anyone know when Flink Forward Asia Jakarta 2024 will be in person or online only?

CFP for Current Bengaluru 2025: The Data Streaming Event - is officially OPEN If you’re deep in the world of Flink, building real-time data processing magic, or have a cool story to share, Current is a great opportunity to get on stage and inspire the community. It can be cutting-edge tech, lessons learned, exciting FLIP or creative solutions; your voice matters! 🔗 Submit your proposal: https://lnkd.in/eVDVtQqm 📅 Deadline: December 19, 2024 Hope to hear from you soon!

Hi all, I was wondering about how task managers store intermediate outputs in batch jobs (especially before keyed streams) are they stored in memory, or in local file storage? I have not been able to find where this data is stored?

Hey everyone, we have a couple openings within Stripe's Flink team. If you are currently in the USA/Canada, and are interested on pushing the boundaries of Flink to power innovative use cases, please apply. More information here.

Hi. I wonder is there any plan to support Iceberg hidden partition on Flink?

Anyone know when we can expect s3 tables support with flink. Seems like its own catalog implementation

Hi everyone, we have a few openings in Autodesk for the Data Streaming team in Dublin, Ireland. If you are interested in learning more about it, you can check out this post. I would be happy to answer any questions as well so feel free to reach out.

CFP for Current Bengaluru 2025: The Data Streaming Event - is officially OPEN If you’re deep in the world of Flink, building real-time data processing magic, or have a cool story to share, Current is a great opportunity to get on stage and inspire the community. It can be cutting-edge tech, lessons learned, exciting FLIP or creative solutions; your voice matters! 🔗 Submit your proposal: https://current.confluent.io/bengaluru 📅 Deadline: December 19, 2024 Hope to hear from you soon!

Slightly off topic question, let me know if this belongs in some other channel. I am a newcomer to Flink, and would like to contribute to the code. I found a recently accepted FLIP that I was interested in. What is the best way to try and get involved in the development process?

Has anyone ever seen a way to work with Broadcast State that's larger than working memory? Would it be terrible to use rocks directly if I'm not too concerned about data redundancy/durability?

Hi, does Flink 1.19.2 is out? I can find mvn repo of it, but not release docs

Hi all 👋! Do you have an open source project that you think could turn into a business? You might like this talk that is talking place tomorrow, Wednesday 5 March. It takes examples from three companies (Percona, DBeaver, and Altinity) that built profitable businesses selling, supporting, and running open source software. Register here: https://altinity.com/events/build-a-great-business-on-open-source-without-selling-your-soul

is anyone using Zeppelin and Flink? Love the idea but seems it doesn't support recent versions, e.g. https://github.com/apache/zeppelin/pull/4864

Hey everyone! We’re evaluating different vendor solutions to run stateful IoT data processing workloads on Flink in our GCP environment. We’re a small team with limited ops experience, so a managed or low-overhead option would be ideal. Does anyone have insights or experiences with the following (or other) solutions? • Google Dataproc for Flink (and any tips on managing stateful workloads) • Ververica BYOC (with upcoming GCP support) • Cloudera Stream Processing • Confluent Cloud - Flink How well do these options integrate with GCP services, and how much operational overhead do they typically require? Any guidance on the best fit (preferably a managed service that could integrate with GCP) would be greatly appreciated! Thanks

Hi is it a bad idea to have a large accumulator when doing a windowed aggregation with an AggregateFunction? Is the accumulator serialized/deserialized on every event that comes in or only on checkpoints?

hey 👋 is JDBC connector supported for Flink 1.20.1? if not - is this planned? if not yet supported - is it ok to use Flink

1.20.1

and flink-connector-jdbc eg.

3.2.0-1.19

? are they compatible? thanks!

Hi Everyone Does anyone have some examples of managing Secrets in Flink Sql. The Connectors like Kafka generally require secrets for connecting and I want to avoid placing it in Sql Scripts? For example can you reference an Env Var inside a SQL script? Thanks

Hi everyone, I wonder if anyone bought early-bird ticket for Flink Forward in Barcelona? I made the purchase successfully to Ververica, but I don’t receive any confirmation/invoice nor ticket from Ververica. Is it normal? 🤔

hi, any one know about Arroyo https://www.arroyo.dev/ ? Look like another stream processing engine

And this is why OLake (Open Source) is fast! Here's something for your weekend read: Exploring OLake's Architecture. If you're diving into real-time data replication or building modern data lakehouse architectures with Apache Iceberg, we've just shared an in-depth look at how OLake actually works behind the scenes. Whether your stack includes MongoDB, PostgreSQL, or MySQL, and you're targeting formats like Apache Iceberg or Parquet, this article has practical insights on designing scalable, efficient data pipelines. OLake is an open-source tool specifically built for high-speed data ingestion. Key Highlights: ⚡ Speed: Load data 4x to 10x faster compared to traditional ETL tools. 🕒 Real-Time CDC: Minimal-lag Change Data Capture from MongoDB, PostgreSQL, and MySQL. 🧩 Plug-and-Play Architecture: Cleanly separated core, drivers, and writers make extending OLake straightforward. 📊 Schema Flexibility: Seamlessly handles schema evolution and type changes compatible with Apache Iceberg. 🔄 Reliable Syncs: Built-in state management means your sync operations can resume effortlessly if interrupted. https://olake.io/blog/olake-architecture-deep-dive

hi hi all, anyone use the flink/prometheus connector that can assist. need the required jar file and then how to package data via flink using flink sql. got json payloads inbound that I need to reshape into the correct format and send out to the sink connector.

<https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/prometheus/>

there's a lot of talk of ai stuff in flink but the only pytorch or tensorflow connector i can find is the old alibaba dl-on-flink repo, which is not maintained / out of date. hasn't been updated in 3 years, wants you to use a similarly old pytorch version, flink version, etc flink's own ml library doesn't have these features and also isn't on flink 2.0 is there something out there i don't know about or what?

🚨 Benchmark Alert — OLake is rewriting the rules. 🚨 We ran head-to-head sync benchmarks and here’s what shook out: ✅ 100× faster than Airbyte ✅ 99× cheaper than Fivetran ✅ 3× faster than Debezium ✅ 11× faster and far cheaper than Estuary OLake synced 4 billion rows for only $75. Competitors? Either took hours… or cost thousands. 😳 You seriously need to take a closer look at OLake. Happy to share details or set up a deeper dive — just ping me. More details here: https://olake.io/docs/connectors/postgres/benchmarks