https://linen.dev logo
Join Slack
Powered by
:wave: Hello <!everyone>! If you utilize dbt with...
# announcements
r
👋 Hello <!everyone>! If you utilize dbt within your stack for data transformation, we’d love to learn more. • Extra context: related github discussion on custom normalization (dbt) in Airbyte OSS. Our question to you: As a user of Airbyte OSS, what method or tools are you using to trigger dbt jobs on your data? Please let us know in the thread! 🙌
airbyte 1
D 10
✅ 6
c
airflow
n
airflow
plus1 3
👍 1
c
dagster!
plus1 12
n
airflow
e
Prefect
o
Perfect
d
prefect
d
Airflow
✅ 1
c
We use dbt core with Astronomer Cosmos (a free OSS package for Airflow)
👀 2
a
Airflow's DBT Operators
s
KubernetesPodOperator
of Airflow. It fetches the recent image of dbt code from Amazon ECR and executes the dbt commands within this new pod
👍 1
k
dagster
✅ 1
m
Airflow
👀 1
p
Dagster
a
prefect
b
Kubernetes/Argo
h
kestra
🔥 6
l
airflow
y
Airflow
j
Airflow with 
KubernetesPodOperator
, similar to what @Salman Jamil mentioned but in GCP.
m
Airflow
r
Dagster
j
Airbyte POSTs to Prefect, and Prefect parses the payload for the connection UUID in order to determine deployments that should be kicked off.
🤔 1
c
Dagster
g
dbt cloud
m
Airflow
r
Dagster
t
Dagster
t
Prefect
a
dagster
m
Kestra
🔥 3
q
Kestra
🔥 3
f
Prefect
n
dagster
m
Airflow
m
dbt operator from astronomer cosmos
👍 1
m
dbt cloud
d
Airflow
d
Airflow calling dbt CLI.
n
Airflow
n
Airflow
j
Airflow
j
Airflow, but looking into Temporal
s
Prefect flow running in Fargate on a schedule driven by Cloudwatch Events (no need for Prefect server in this simple deployment)
t
Similar to Salman and Juliano. We have a custom operator, with 
KubernetesPodOperator
as base, which gives us flexibility in running DBT with different configurations
l
prefect
i
Airflow that triggers GH workflow in dbt repo
v
prefect
j
dbt cloud
f
Airflow triggering dbt job running on Fargate
a
Kestra
🦾 2
🔥 3
l
Dagster
t
Dagster
a
Airflow
d
dbt Cloud
d
prefect
j
nooooo dont get rid of custom transforms! 🙂 We write our own normalization in lieu of airbytes that triggers after each airbyte run
n
Airflow
w
airflow
g
Airflow
d
Dagster
c
Airflow (MWAA)
m
Temporal
g
dbt Cloud
p
dbt cloud
a
dbt cloud but with their shift to consumption-based pricing will probably switch to dagster when the time is right.
s
dbt Cloud
d
Kestra and basic cron job from operating system.
i
Running dbt Core models using Airflow OSS. About to add Astronomer's Cosmos to it. Also, moving away from Airbyte's built in normalization of JSON ("Normalized tabular data" to be absolutely clear) and planning to do all transformations only in dbt since this is more predictable for us.
👍 2
n
Prefect
j
dagster (you can use the Dagster D to answer instead, easier to count that way 😄)
m
KubernetesPodOperator
using the DBT Core image.
l
Prefect here, and also indirectly with custom normalization image for databricks normalization step. Also triggers some on github actions for CI.
a
m
Airflow
d
dagster
q
Airflow itself, repo on github, Docker image on Artefacts registry. So, when the connection jobs run in Air8, it trigger the dbt transformation sql code
j
Dagster
u
Gitlab pipelines and we recently adopted Snowflake's dynamic tables
c
dbt cloud
m
Prefect
r
Kestra
t
Airflow
j
Dagster
m
Dagster
m
Google Cloud Workflows, triggers airbyte first, then transformations in dbt core running on cloud run.
👍 1
m
airflow!
j
We use the Airbyte post run custom DBT transforms heavily. Also dbt cloud for other orchestrations. The running of DBT immediately post Airbyte run is infinitely handy and avoids requirement to introduce other tooling
same 1
f
Airflow with Cosmos
a
Google workflows w dataform transformations
a
surely late for this, but airbyte dbt run is super useful. Normalisation sure, but just to trigger dbt run is priceless
same 1
fivetran has this feature so as other replication tools, will be a real damage to loose it
a
Prefect starts the airbyte sync using api And then once that is done dbt job is run
Open in Slack
PreviousNext
Powered by