Heya, Ive setup Salesforce as a Source and Bigquer...
# announcements
t
Heya, Ive setup Salesforce as a Source and Bigquery As a destination with normalization enabled. But it seems that all Date fields are omitted
c
All date fields are omitted
Do you see these date fields in the catalog screen (when selecting which tables and columns are going to be replicated?)
t
Hey Chris,
Yes for example: Omzetmaand_Date__c object Omzetmaand_Date__c
its noted as an object
c
Ok then I think because it is detected and noted as an object, it is not picked up by the basic normalization process since it doesn't know the data type that it should use and is omitted
Are you able to run your own transformation tool? maybe you already use DBT after loading data in your warehouse?
t
Im running datform
dataform
but that would require me to write a SQL query to parse json and thats not what i would luke
like
c
1. Yes you are right, at the moment you could write your own SQL query to parse the json and extract those columns (which is not ideal) We have a tutorial on how to retrieve the SQL airbyte generates to do this instead of going from scratch if you'd like... 2. In the meantime, we have to change in airbyte how we deal with columns we don't know the datatypes so they can still be exposed and you don't have to do step 1: It could either by: • supporting more data types • supporting custom types provided by user • unknown types exposed as simple strings that you can cast yourself afterward
t
could you link me the tutorial?
It is generating DBT files but you could export and use the SQL files to edit and customize in dataform instead
t
Hmm
c
Would that work out for you as a quick fix for the moment?
t
I think ill put in a feature request to provide better control over normalization in Airbyte
👍 1
and ill do the normalization myself in Dataform.
c
Yes definitely, we'll work on that in the near future so it should be helpful to drive towards the right direction