https://datahubproject.io logo
Join Slack
Powered by
Hi everyone, does DataHub provide ingestion librar...
# ingestion
o
Hi everyone, does DataHub provide ingestion libraries in Java? Currently I only see Python modules for that. Thanks.
b
at the moment, it is python based library for ingestion only.
thank you 1
o
Thanks.
m
Hey @gray-table-56299: we are working on this through the recently added 
spark-lineage
PR. Once we get those jars published, we should be able to come back and release the underlying library that allows you to emit the raw events.
We're getting the spark-lineage jars published later this week.
g
So we’d be able to have java apps emit MCEs directly? Do we have an ETA on when you can release the underlying libraries?
@mammoth-bear-12532 do you also mind linking the pr for that spark lineage? curious how they are related to what i’m requesting
m
@gray-table-56299: I'll be able to get you a more accurate estimate for releasing the underlying libraries after we successfully release the spark libraries this week.
g
Can I expect these libraries to have the same degree of support as the current Python client?
m
yes!
one question -> are you using the REST pathway or the Kafka pathway for metadata events into DataHub at your company?
g
@mammoth-bear-12532 Kafka pathway. i am hoping to develop a kafkastreams app that generates MCEs off change events across our data ecosystem. Does that make sense?
m
sounds good @gray-table-56299 yes that makes sense.
g
Based on my understanding, I believe we would need the underlying libraries similar to this python emitter… Or am I missing something? 😅 Is this a workflow thats being done elsewhere that you know of?
@mammoth-bear-12532 I was able to use the avro schemas and create an MCE from a local kafkastreams app to update a dataset property in our dev instance of datahub. It was a bit more involved than I expected. What is the timeline for releasing Java libraries?
m
@gray-table-56299: perfect timing -> https://search.maven.org/search?q=a:datahub-client 😄
caveat: this contains the REST emitter currently.
we need to fix a couple of links in these docs for the artifact itself, but the rest of the guide works.
g
Nice! What is the timeline for kafka emitter? I’ve been using the MCE Builder class directly.
m
the MCP is what we are supporting going forward, MCE is frozen in time and does not get new additions to the model.
kafka emitter timeline -> will be driven by demand from the community. @little-megabyte-1074: maybe we can create a feature request task for this?
l
I’m not sure I have enough context to capture the request effectively. @mammoth-bear-12532 or @gray-table-56299, would you mind adding a feature request here under “Developer Experience”? https://feature-requests.datahubproject.io/
m
g
That link is 404-ing for me 🤔
m
try now?
I needed to approve it 😄
g
👍 I can see it! Upvoted too. Thank you!
5 Views
Open in Slack
PreviousNext
Powered by