I'm not 100% sure this is the right channel. Curre...
# troubleshooting
c
I'm not 100% sure this is the right channel. Currently my source is a Postgres database with many schemas. Using WAL replication I'm listening to all these schemas and dumping them to the same destination. I'd like to include source schema name as an additional column in the destination table. For example I might have schemas:
moe, larry and curly.
All three of these source schemas have the same table called
stooges
. My destination would only have a single schema called
public
and I would like all three sources to dump into the same
stooges
table in this destination schema; however, I would like to add an additional text column in the destination table called source_schema which would take on the value of
moe, larry and curly
.
m
One suggestion is use views and add those columns in your source tables, but you’re using CDC and probably want to get updates/deletes from tables. Unfortunately this is not possible today to add new information in the destination.
c
Thanks for clarifying! I know the schema name is included in Postgres WAL replication, do you know if there is any plan to make it available for people adding custom transformations with DBT
For anyone else creating a similar pipeline, I’ve got a workaround I'm currently testing. I’m creating unique sources for each schema. And then a unique connector for each source. All connectors are using the same destination. Each connector leverages the same custom defined DBT transformation. The connectors declare the source schema as a dbt var in the run command. This source schema is then appended as a column in the transformation.
m
Awesme @Connor Francis that you found a workaorund… not the best way but at least you can overcome it