Hi All, I am new to Airbyte and looking for some h...
# replication-troubleshooting
a
Hi All, I am new to Airbyte and looking for some help. I have set an opensource Airbyte in my local machine by cloning the github repository and running the instance using docker-compose up. I have added source and destinations and also created a connection as well inside the transformation tab I have selected the radio button for Normalized tabular data, inside replication tab I have selected the Destination Namespace* as Mirror Source Structure and prefix is blank and Finally clicked the sync button to Run. I noticed that we have the data tables created in the Destination from Source but also it has created some new tables there that stores the data into RAW JSON format. Since I have selected only 2 tables as of now so there is not an issue, going forward if I select more tables then this could lead to space issues. The additional tables start with this prefix "_airbyte_raw_" and I have not mentioned this prefix any where in the process. For more info I have attached the screenshot below that shows the actual tables and JSON tables.
cc: @Amruta Ranade
cc: @Adam Knobel (Airbyte)
m
Those tables are used internally by Airbyte to generate the final ones
a
Hi, @Marcos Marx (Airbyte) Thanks for your reply. As per my understanding from the documentation airbyte creates these JSON blobs to move the data through E and L stages without any transformation. But this is causing duplicate data in Destination (One in JSON Blob and other in the Tabular format) where size of the JSON Blob table is much greater than the actual table. How do we handle this memory issues.
u
Airbyte works with ELT paradigm which extract all your data to your datawarehouse because storage is cheap. You can delete raw tables if you're using incremental syncs of after runing the sync with an external process.
a
Thanks @Marcos Marx (Airbyte) this was really helpful. One another thing after creating connection I tried replicating two tables from Source (Postgres) to Destination (Postgres) with Destination Namespace* as "*Mirror source structure*" and selected two streams to sync One stream got loaded in destination successfully and other stream is giving this error.
Copy code
When searching for a relation, dbt found an approximate match. Instead of guessing 
which relation to use, dbt will move on. Please delete "postgres"."CEMD_RAW"."raw_edreports_ratings", or rename it to be less ambiguous.
Searched for: "postgres"."cemd_raw"."raw_edreports_ratings"
Found: "postgres"."CEMD_RAW"."raw_edreports_ratings"
we tried deleting and creating that schema as well but it didnt work
m
if you use custom destination namespace it work?
a
yeah it works