Hi Guys, When Airyite creates a table into snowfla...
# announcements
h
Hi Guys, When Airyite creates a table into snowflake, the table name has the format :`dest_schema_name.source_schema_name`.
source_name
. For example source table in Postgres :
public.student
destination table name into Snowflake:
db_name.airbyte_raw.public.student
What causes some annoying issues, because the table name has a dot inside name. A simple way to fix it should be use the
Cleaned name
that already replace the
.
for
_
c
Hi Hudson! We can definitely look into something like this. What's the issue you're running into due to the dots?
h
Hi Charles! In snowflake if your table has dot in the name you need to use double quotes like this:
Copy code
select * from airbyte_raw."public.student"
My current problem is for example when I try to use the table inside DBT - Sources DBT removes the double quotes when compile the query what cause an issue.
👍 1
It's not a big deal but I think is a small fix also, I'm not sure if the solution should be implemented only in the snowflake connector or in all connectors. Because in the general it's not usual to use dots in the table name.
c
@Hudson, i agree with you, it would be nice in Airbyte to have a better control on the output schema/table names (we have some issues created about this already) In the meantime, you can also configure your DBT project to adapt to this specificity and make sure the source produced by airbyte with the weird
.
character in the table name is handled properly: https://docs.getdbt.com/faqs/source-quotes/ So DBT will indeed generate the query:
Copy code
select * from airbyte_raw."public.student"
h
Hey @Chris (deprecated profile) Nice catch, thanks for this link.
j
Hi @Chris (deprecated profile) can you elaborate on which issue(s) we can watch to see customized destination table names? Right now this is the only blocker from our team adopting airbyte. Thanks!
c
Right now this is the only blocker from our team adopting airbyte. Thanks!
What is the exact blocker for your team? I think the
.
character should be resolved in the latest version already
j
maybe i need to do more testing, but we have a postgres instance that stores raw data from our clients. we're using dbt to transform it to be operational, and then trying to replicate to a separate operational postgres cluster our naming in the dbt transformed views was client specific initially, however our operational store is a 1 db per client approach, so ideally we could just set the table names by removing the client prefix
i'm new to dbt so might be able to customize on that end too
c
our operational store is a 1 db per client approach, so ideally we could just set the table names by removing the client prefix
So if i understand, you have : • one DB per client (
client1.users
,
client2.users
, etc) • and you want only one table for all clients in the destination? (
public.users
)
j
one table in the source and one per client in destination, but as i type this it probably makes sense to have it be namespaced separately somehow
but i suspect I'll still need some level of customization, i'll keep testing. thanks for the quick response!
is it possible to insert/append into an existing table? django manages my operational datastore so ideally i could just insert/append into an existing table (which wouldn't have
public_
as a prefix to the table)
c
this sounds pretty tricky (in cases where you have table schema changing between source/destination) maybe if you could write a custom SQL query to execute at the end of airbyte pipeline, you could decide to merge some airbyte output table into an existing table of your own?
j
certainly an option, though i'd prefer to not to have the extra table created in my operational store