https://datahubproject.io logo
Docs
Join the conversationJoin Slack
Channels
acryl-omnisend
advice-data-governance
advice-metadata-modeling
all-things-datahub-in-windows
all-things-deployment
announcements
authentication-authorization
chatter
column-level-lineage
contribute
contribute-datahub-blog
data-council-workshop-2023
datahub-soda-test
demo-slack-notifications
design-business-glossary
design-data-product-entity
design-data-quality
design-datahub-documentation
design-dataset-access-requests
design-dataset-joins
feature-requests
flyte-datahub-integration
getting-started
github-activities
help-
i18n-community-contribution
ingestion
integration-alteryx-datahub
integration-azure-datahub
integration-dagster-datahub
integration-databricks-datahub
integration-datastudio-datahub
integration-iceberg-datahub
integration-powerbi-datahub
integration-prefect-datahub
integration-protobuf
integration-tableau-datahub
integration-vertica-datahub
introduce-yourself
jobs
metadata-day22-hackathon
muti-tenant-deployment
office-hours
openapi
plugins
show-and-tell
talk-data-product-management
troubleshoot
ui
Powered by Linen
integration-alteryx-datahub
  • a

    astonishing-answer-96712

    01/05/2023, 8:37 PM
    https://datahubspace.slack.com/archives/C02R2NBJXD1/p1671022547495179
  • l

    lemon-lock-89160

    01/17/2023, 9:48 PM
    Hi @astonishing-answer-96712 and @gray-shoe-75895. @bitter-park-52601 and I are trying to understand how we get started on this custom Alteryx integration. However, we keep banging our heads into the wall. We both come from an analytics background and not CS, so how would you recommend we get started on building this connector? Alteryx, which we are trying to hook Datahub up to, will in many ways look and feel like Airflow does in Datahub. An Alteryx workflow is a task that will have data inputs, do some transformations, and then have data outputs. So it resembles an Airflow task in many ways. We have however, not been able to understand how we should build the connector, even after look at some structures for how Airflow is setup. Any input on how to get move forward would be highly appreciated 🙏
    g
    • 2
    • 2
Powered by Linen
Title
l

lemon-lock-89160

01/17/2023, 9:48 PM
Hi @astonishing-answer-96712 and @gray-shoe-75895. @bitter-park-52601 and I are trying to understand how we get started on this custom Alteryx integration. However, we keep banging our heads into the wall. We both come from an analytics background and not CS, so how would you recommend we get started on building this connector? Alteryx, which we are trying to hook Datahub up to, will in many ways look and feel like Airflow does in Datahub. An Alteryx workflow is a task that will have data inputs, do some transformations, and then have data outputs. So it resembles an Airflow task in many ways. We have however, not been able to understand how we should build the connector, even after look at some structures for how Airflow is setup. Any input on how to get move forward would be highly appreciated 🙏
g

gray-shoe-75895

01/18/2023, 6:45 AM
Hi @lemon-lock-89160! We have this basic guide around building a custom datahub source: https://datahubproject.io/docs/metadata-ingestion/adding-source. Much of the airflow logic for creating “dataflows” and “datajobs” (think airflow dags and tasks) lives here https://github.com/datahub-project/datahub/blob/b796db1cafacd2d0f2dc7c78df89be2110[…]data-ingestion/src/datahub_provider/client/airflow_generator.py
l

lemon-lock-89160

01/18/2023, 9:00 AM
Thank you @gray-shoe-75895 we will look into it. We may need to go back a step even further in regards to how to work with the datahub environment on a more general/basic level.
View count: 5