https://linen.dev logo
Join Slack
Powered by
Hi… how can I extract noramlization files like exp...
# replication-troubleshooting
m
Hi… how can I extract noramlization files like explained in https://docs.airbyte.com/operator-guides/transformation-and-normalization/transformations-with-dbt in a Kubernetes deployment? Thank you!
m
In Kubrrnetes you need to execute other process which is harder 😅
m
Is this explained somewhere? or could you please give me some pointers? Thank you!
From https://docs.airbyte.com/deploying-airbyte/on-kubernetes/ “Reading Files”… it seams i can just access the pod and read it, but I can’t find the tmp/workspace folder in airbyte-server pod for the normalization files…are they somewhere else?
a
+1 . same problem here
f
same problem help!
m
@Sunny Hashmi (Airbyte) can you check this question here? would be nice to update the docs to show people how to do it
🙏 1
👀 1
f
you can helpme ?
s
Hey all, this is currently not very easy on Kubernetes however we do have an open issue to address it, please add your thoughts and a thumbs up to this github issue to help with prioritization: https://github.com/airbytehq/airbyte/issues/8090 Also this issue: https://github.com/airbytehq/airbyte/issues/3365 On kubernetes the workspace folder actually lives in the normalization pod, which is only alive for a very short time and shuts down after its job completes. It may be possible to grab the files from there, but usually it isn't because the job is done before you can extract them. Currently the recommendation is to spin up the same configs in Docker and export the dbt project as per the docs.
f
oh my god, is horrible, my computer is small to process a sync to db with 200gb
and after get dbt files
i belive that should enabled from iu
ui
m
you can get the dbt from a failed sync
f
ah ok ok
thanks for info
@Fabiano Pena this is a option
✅ 1
@Jared Rhizor (Airbyte) "you can get the dbt from a failed sync" I am current try active of sync after cancel but can't get dbt file
if or if this should fail?
m
well, it must run the normalization process and fail in that step
f
oh
if get error for a database with big size of data, 4000tables this process fail and not run the process normalization
this process leads me to have a computer with great capacity to execute the loads and then obtain my dbt and finally bring all this to my gke, this process is very long
@Marcos Marx (Airbyte) i have question, if i have selected a table of my scheme, in the same way it generates the dbt for all the tables or only the selected one?
s
@Francisco Viera my understanding is that the transformations are generated based on the data airbyte has loaded into the destination's raw tables as a json blob. So if only some tables are synced then the dbt files would only act on that data https://docs.airbyte.com/understanding-airbyte/basic-normalization#high-level-overview
If you have Basic Normalization enabled, Airbyte automatically uses this JSON blob to create a schema and tables with your data in mind, converting it to the format of your destination. This runs after your sync and may take a long time if you have a large amount of data synced.
Can you create a request in github to add the desired functionality? Suggestions in slack get lost very easily
f
I feel that it is reworked a lot, it should be feasible to download it from the interface or store it in storage
Open in Slack
PreviousNext
Powered by