https://flink.apache.org/ logo
Join Slack
Powered by
# dev
  • r
    does anyone know the status of support for Flink 2.0 in the Iceberg connector ? There's only 1.20 on Maven, but I've seen activity here: https://github.com/apache/iceberg/tree/main/flink/v2.0
    m
    • 2
    • 2
  • a
    hi all, i have raised a PR : https://github.com/apache/flink-shaded/pull/147 to update the curator library. Would appreciate review from the flink community members. Thank you 🙂
  • t
    Hey folks, I have a PR for updating the Flink Connector Kafka to the 3.9.1 Kafka client version. Would someone with the power be able to kick off the CI? Maybe @Martijn Visser or @Arvid Heise?
    a
    f
    • 3
    • 52
  • v
    Does flink 1.18 have basic support for Java 21 ? I am trying to place which version of flink I need to upgrade to use Java 21. From the docs it looks like flink 2.0 has full support, but I might not need the full support. Please clarify. Thanks
  • r
    👋 We're looking for Flink Connector Kafka maintainers to look at an internal change we've merged for handling Projection Pushdown for Kafka https://github.com/apache/flink-connector-kafka/pull/174 This helps us handle non-uniform streams (CDC), particularly with Flink SQL and the Flink Catalog.
    👍 2
  • f
    hi all, RM for Flink 1.19.3 and 1.20.2 here, both release requires binding votes, if any PMC has some time, pls vote: • 1.19 thread: https://lists.apache.org/thread/bw2s9j76k3glt16365ls4hbqbh820qd8 • 1.20 thread: https://lists.apache.org/thread/xwsmpvj6t0jnhkyfhpqd4m79hnyssjyy
  • c
    Does anyone know what the release schedule looks like for the 1.19.3 and 1.20.2 releases? When will the official releases be available?
    f
    • 2
    • 1
  • j
    Hey, opened a ticket and made a pr a few months ago was hoping to get some eyes on to merge this one soon.
    r
    • 2
    • 2
  • t
    Hey Folks, I have opened PR to update the Flink Connector Parent pom in-line with the Flink Parent pom. I realise this could effect quite a few connector projects so I wanted to get some 👀 on it.
    • 1
    • 1
  • t
    Hey Devs, I have opened a PR to bump a lot of the dependencies in the flink-connector-kafka project. Many were out of date and/or didn't match the what the main Flink project was using. Would someone with the power be able to kick the CI (it passes locally and in CI on my fork)?
    ✅ 1
    • 1
    • 3
  • t
    Me again, cross posting this email I sent to the mailing list about Flink Kafka connector releases. TL;DR: I think we should do a 4.1.0 release with the Kafka 3.9.1 client (as it fixes a high severity CVE) and this dep bump PR. Then move to the Kafka 4.0.0 client (with this PR) and, as this would drop support for Kafka clusters running 2.0.0 or lower, do a 5.0 release of the connector. It would also be good to move to Java 17 and drop java 8 by updating the connector parent pom.
    a
    • 2
    • 2
  • b
    👋 團隊成員們好！我是新加入的成員，我正在使用flink operator ，我發現原生的flink似乎可以處理locall:///路徑，但是flink operator在處理submitsessionjob時缺少了相關的判斷，我只瀏覽過原始碼，還沒有對原生flink進行相關的測試。我正在考慮對flink operator的項目申請pr，我想先進行相關的討論了解其可行性。
  • m
    Hello everyone, I hope you're doing well. This is Mahesh. I’m reaching out to let you know that I’ve been assigned the following JIRA task: FLINK-38058, which involves writing end-to-end tests for the Flink CDC Fluss connector. As this is my first contribution to the project, I’m not entirely sure where to begin or which test cases would be most valuable to implement. I would really appreciate your guidance on how to get started and any suggestions you might have on the structure or scope of the tests. Thank you for your time and support.
    🙌 1
  • m
    Hi All, 👋 We are using Flink with GCP PubSub and have recently worked on migrating to Flink v2 with this branch of the GCP PubSub connector. The integration seems to work fine. We would like to contribute our changes back to the connector, but it seems that the repository has been dormant for over a year. A few questions: • who is working on/maintaining the code in the 
    flink-connector-gcp-pubsub
    repository? Who can review/approve/merge any MRs? • we migrated the code base to Flink v2 and have adopted the streaming pull approach from this branch, implementing FLIP-27. Should we create multiple tickets and branches to get this accepted? Thanks for any help on this!
    ✅ 1
  • m
    Hiya. Some call FLINK-7129 the most watched issue of all time. The FLIP discussion seems to have stalled a couple of years back. I came to this issue through searching for a tool that allows pattern processing in a setup consisting of a control plane (adding rules to a table) and a data plane (processing those rules), i.e. Dynamic CEP. It should be a pretty solid case for Flink, but somehow developing the functionality has stalled. There’s some existing work; a relatively newer one that I don’t think is open-source: https://www.alibabacloud.com/blog/application-of-dynamic-flink-cep-at-bank-of-hangzhou_602041 They’re building on FLIP-200 and Ververica’s Dynamic Flink CEP demo code.
  • a
    I think I have found a bug (or at least oddity) in some Flink-managed k8s pods where the 
    ownerReference
    on Pods are being set to an unrelated 
    Deployment
    (as opposed to the 
    FlinkDeployment
    who is the real owner controlling the pods' creation). This ends up creating unexpected results in downstream metrics (investigation here). I think ArgoCD does a good job of illistrating the 
    ownerReference
    relationships for visual learners.
    Copy code
    apiVersion: v1
kind: Pod
metadata:
  labels:
    app: apache-feed-job-notifications
    <http://app.kubernetes.io/instance|app.kubernetes.io/instance>: nis-apache-feed-app
    <http://app.kubernetes.io/name|app.kubernetes.io/name>: apache-flink-feed-job
    component: taskmanager
    <http://helm.sh/chart|helm.sh/chart>: apache-flink-feed-job-0.0.3
    type: flink-native-kubernetes
  name: apache-feed-job-notifications-taskmanager-5-39
  namespace: nis
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: Deployment
    name: apache-feed-job-notifications
spec:
  {...}
    I am waiting on a JIRA account to file an issue but thought I would at least start a discussion here to get the community's appetite for aligning these pods' 
    ownerRefernce
    to be more consistent with k8s best practices(?). I am a bit new to all the intrecicies of 
    ownerReference
    so maybe there is some part I am missing here but at minimum the current behavior seems strange that a 3rd party controller is setting an 
    ownerReference
    to a different k8s kind outside of it's domain, no?
    ➕ 1
    • 1
    • 2
  • d
    Hi devs, The vote thread for flink-connector-aws v5.1.0 RC1 is open at https://lists.apache.org/thread/5zpjy2s1gxjlm9q8n5w52lg3gxsgfgdp A few devs and I have tested it and shared the results on the thread, and currently looking for more reviewers, testers and voters. If you are interested to review, vote or simply to find out more about the connector, please do not hesitate to reach out to me on Slack or email! Thank you!
  • v
    Hey All, why is there no 
    java21
    docker image for flink 
    1.19.X or 1.20.X
    ?
    p
    a
    • 3
    • 5
  • r
    Hi, masters, While conducting preliminary POC validation for FLIP-495, I noticed several spelling issues. Therefore, I’ve submitted a PR https://github.com/apache/flink/pull/26820 attempting to fix it. If anyone has time, could you please help review it? Many thanks!
    • 1
    • 1
  • v
    Does anyone know why many connectors got abandoned after 1.18? Looks like even when there are PRs open to update them, there is no one to review them and release updates.
    • 1
    • 4
  • p
    Hello devs, As a part of this initiative : SourceFunction API removal i am working on this task https://issues.apache.org/jira/browse/FLINK-32695 . Since many test cases still use the legacy Source API, I’ve compiled a spreadsheet that lists the SourceFunctions involved and their corresponding PRs updating them to Source V2. The first set of PRs is already up, and I plan to complete most of the remaining ones over the next 1-2 weeks. (Code is almost done, just need to push the PR's) To keep things manageable and reviewable, I’ve kept the changes isolated per test case. I’d really appreciate the support of committee members for reviewing and merging these PRs. If you're available to help, please let me know — I’ll be happy to tag you on the current and upcoming PRs next week. Thank you so much for your support!
    • 1
    • 1
  • r
    👋 Hello, looking for Flink Runtime maintainers to look at this change https://github.com/apache/flink/pull/26811 It corrects the hostname resolution between the BlobServer and BlobClient in Flink clusters when a Jobmanager bind-host is set.
  • r
    👋 Hello, Devs. Kindly Requesting votes/discussion on FLIP-495. I have done the POC[1] for FLIP-495[2] to provide you with a concrete sample for discussion[3], voting[4] review. Currently, - FLIP-495 [2] is waiting for a final review and is lacking a required binding vote at least (get 2 binding votes and 2 non-binding votes now). - FLIP-487 [5] is pending the completion of FLIP-495[1].(The UI design is completed.) It would be greatly appreciated if you could share some feedback on discussion[3] or help with the voting[4]. [1]https://github.com/RocMarshal/flink/pull/7 [2]https://cwiki.apache.org/confluence/display/FLINK/FLIP-495%3A+Support+AdaptiveScheduler+record+and+query+the+rescale+history [3]https://lists.apache.org/thread/t3r9wdd5gpbqnvzw35kb3wb3d9brpnon [4]https://lists.apache.org/thread/7v10k44kgkfmn3dftmb7loslom4hndy5 [5]https://cwiki.apache.org/confluence/display/FLINK/FLIP-487%3A+Show+history+of+rescales+in+Web+UI+for+AdaptiveScheduler CC @Matthias @rmetzger
    m
    • 2
    • 2
  • v
    Hi All, I want to consume messages from RabbitMQ (super streams) using Flink connectors. I explored RabbitMQ connectors for Flink at https://nightlies.apache.org/flink/flink-docs-release-2.1/docs/connectors/datastream/rabbitmq/ but this page only explains on consuming messages from RabbitMQ queue and not from the RabbitMQ super streams. More details on RabbitMQ streams can be found here https://www.rabbitmq.com/docs/streams I'm looking out for an out of the box Flink connector which provides fault tolerance and scalability to connect to RabbitMQ streams and consume messages using Flink?
  • r
  • j
  • r
  • p
  • m
    Hello devs, is it in the roadmap to introduce a HyperLogLog Operator for DataStream API? I have seen that it has just been introduced for SQL API
  • j
    hello folks, just a kindly ping @Gyula Fóra @Máté Czagány about https://issues.apache.org/jira/browse/FLINK-38253 https://github.com/apache/flink/pull/26908 , could you please take a look once you will have some spare time? Thanks a lot