Hi, I’m giving Airbyte a go and am facing some iss...
# advice-data-transformation
s
Hi, I’m giving Airbyte a go and am facing some issues. My destination is Greenplum DB, which is not supported on its own, so I’m using Postgres connector as they are quite similar. In the future I may even build my own connector to get more out of it. However, when using default normalization, models fail. Because transformations include indexing (which I’m somewhat against on GP like dbs) and syntax for creating indices is different between PG and GP due to version differences. So it fails and, therefore, the whole sync fails. I’ve tried accessing the folder that shows up in my log, but it is empty. Maybe I’m not doing it right. Regardless, my actual question is this: how does dbt default transformations build its syntax? just by looking at the destination and matching syntax to it? If I build a new connector, will it automatically resolve the issue? Or do I also have to build my own transformations? I’m not against that, but are there any templates available? I mean, exactly what Airbyte tries to do. And then just work it out from there.
j
FYI support is provided on Discourse: discuss.Airbyte.io
This channel is more about exchanging tips, experiences, etc.
s
Can you maybe at least point me to the basic normalization Airbyte does with dbt? I’d like to see what is done by default to be able to create my own dbt that does it right for Greenplum
s
I’ve seen it, I’ve been over the documentation multiple times. But it’s no help, unfortunately. The best advice on there is to look inside the directory where models are stored during dbt run, but I wasn’t able to find actual models. Maybe, because transformation step fails, so it gets rolled back. I just want to see what Airbyte asks dbt to do in case of basic normalization: how it handles meta-columns Airbyte generates and how it handles json unpacking. With that I would be able to customize it to my use-case and then use that as custom transformation basis.
j
The support team can definitely help on Discourse, don’t hesitate to post there
h
@Saymon_says1985 , the custom-transformation doc linked above should be relevant, even if normalization failed (and got rolled back). The models (dbt sql files) should exist in that workspace directory. I've created many custom transofrmations for postgres now, based on these, though often using native postgres syntax for things like jsonb extraction since I don't also need to support other databases.