https://linen.dev logo
Join Slack
Powered by
Hey everyone, hope you all are doing well. I neede...
# ask-community-for-troubleshooting
r
Hey everyone, hope you all are doing well. I needed some help in locating the code wherein column names are changed during the normalization step by Airbyte. Can anyone please point me to that portion of the airbyte repo on Github, was looking to do some custom changes myself - any help would be really appreciated! 🙂
✅ 1
c
what changes to column names are you talking about? example?
r
So, for example, if 1 of the column names in a table in my source is airbyteSync, when I sync that data using Airbyte into my Bigquery warehouse, it remains as it is, I was looking to maybe change it to airbyte_sync
I’ve actually created an issue for the same
c
are you planning to recompile and run your airbyte server with your code change?
r
I was hoping if I could induce some hacks in my local airbyte code so that it’d help me in generating custom column-names
Yes actually, I’m planning to do that, hence I’m looking out for the location of the column-name generation during normalization code 😅
c
I’m working on the custom dbt feature: https://github.com/airbytehq/airbyte/issues/2959 it only needs the last issue on UI to be implemented: https://github.com/airbytehq/airbyte/issues/3235 But with this functionality, you wouldn’t have to rebuild your own airbyte server and you can tweak the normalized models to run as your custom transformation at the end of sync
Otherwise, this might be more involved if you wanted to tweak the normalization/code generation process yourself, it would probably be here in this function where we build a map of the JSON column name to the actual SQL column names in two flavors (one to use outside and inside jinja macros): https://github.com/airbytehq/airbyte/blob/0c11b27cf67461f175a04fb62018af3a0c276bd7[…]rmalization/normalization/transform_catalog/stream_processor.py
r
Oh okay, if I may ask, when are you guys planning to complete this? 😅
Okay thanks, I’ll be exploring this for a bit now
c
Oh okay, if I may ask, when are you guys planning to complete this
I hope this week or next week, it should be ready
Happy to read through your PR if you manage to come up with the right python code for implementing the fivetran “adapter” or naming conventions though 🙂
3 Views
Open in Slack
PreviousNext
Powered by