Heya I believe I’ve found a bug with the Airflow p...
# ingestionq
quiet-television-68466
05/16/2023, 4:13 PMHeya I believe I’ve found a bug with the Airflow plugin: https://github.com/datahub-project/datahub/issues/8058.
In summary: when
datahub.capture_ownership_info = false✅ 1
quiet-television-68466
05/16/2023, 4:23 PMI believe this can be fixed by adding a property:
capture_ownership_infoquiet-television-68466
05/16/2023, 4:23 PMbasically it shouldn’t be creating a metadata change proposal at all (for owners) if capture_ownership_info is false.
quiet-television-68466
05/16/2023, 4:24 PMbear in mind i haven’t tested this, this is just my first guess on how to fix the issue
a
astonishing-answer-96712
05/16/2023, 8:14 PM@famous-waitress-64616 might have some insight here!
f
famous-waitress-64616
05/16/2023, 9:15 PMHi Matthew, our teams is looking into this but it'll be a day or so before I can get back to you. Thanks for your patience
q
quiet-television-68466
05/17/2023, 1:50 PMThat’s great thank you!
l
limited-forest-73733
05/18/2023, 11:47 AM@quiet-television-68466 i am ingesting airflow metadata to datahub using datahub_kafka_default . I am not getting any error its showing emission is happening but not ingesting to datahub. Can u pls help me here
g
gray-shoe-75895
05/19/2023, 1:00 AM@quiet-television-68466 looks like you’re right - would you mind opening a PR to make that tweak?
l
limited-forest-73733
05/19/2023, 5:53 AM@gray-shoe-75895 i am doing via datahub kafka so in that case capture_ownership_info is true so it shouldn’t impact anything
q
quiet-television-68466
05/19/2023, 9:32 AMYes I can hopefully get to that early next week!
quiet-television-68466
05/19/2023, 9:33 AM@limited-forest-73733 is your gms pod writing any logs when that occurs?
l
limited-forest-73733
05/19/2023, 9:34 AMI am able to integrate airflow with datahub using datahub rest but facing issue with datahub kafka
limited-forest-73733
05/19/2023, 9:34 AMWith gms its introspecting airflow metadata to datahub
limited-forest-73733
05/19/2023, 12:47 PM@quiet-television-68466 can u pls clear me here. I am integrating airflow with datahub using datahub kafka. Its showing emission is happening in my airflow dag log but nothing is ingesting to datahub nor added to database as well. Whats the problem here.
I installed the plugin, set lazy load plugin to false and added the corresponding configuration. Thanks
g
gray-shoe-75895
05/20/2023, 12:58 AM@limited-forest-73733 i suspect that’s because we’re not calling flush in the airflow code. That should be fixed by https://github.com/datahub-project/datahub/pull/8093
l
limited-forest-73733
05/20/2023, 6:23 AM@gray-shoe-75895 using datahub-rest we can introspect airflow metadata to datahub, facing emission issue via datahub-kafka
a
aloof-gpu-11378
05/20/2023, 6:34 AMYup - the flush call is a no op for the rest sink, but fixes the kakfa sink
l
limited-forest-73733
05/24/2023, 1:15 PM@gray-shoe-75895 i saw you merged your PR for flush changes, can you please tell me any ETA of new release? Thanks
a
aloof-gpu-11378
05/25/2023, 6:07 AMWe should be releasing 0.10.3 this week
4 Views