https://linen.dev logo
Join Slack
Powered by
<@U04SE163WC8> is there a way to find what tables ...
# ask-ai
c
@kapa.ai is there a way to find what tables from a particular source will get normalised by Airbyte in snowflake
k
Yes, you can find out what tables from a particular source will get normalized by Airbyte in Snowflake. When you set up a connection in Airbyte and choose Snowflake as the destination, Airbyte will create raw tables and normalized tables based on the data from the source. To view the names of the tables in the destination, you can use the following command in the Postgres shell (assuming you are using Postgres as an intermediary step):
Copy code
sql \dt;
This command will list all the tables in the destination. For example, you might see something like this:
Copy code
sql                  List of relations  Schema |          Name          | Type  |  Owner    --------+------------------------+-------+----------  public | _airbyte_raw_table_one | table | postgres  public | _airbyte_raw_table_two | table | postgres  public | table_one              | table | postgres  public | table_two              | table | postgres  public | table_two_scd          | table | postgres (5 rows)
In this example, 
_airbyte_raw_table_one
and 
_airbyte_raw_table_two
are the raw tables, while 
table_one
, 
table_two
, and 
table_two_scd
are the normalized tables. The 
table_two_scd
is a historical table, and 
table_two
is a deduplicated table. For more detailed information, you can refer to the [Airbyte tutorial on incremental data synchronization](https://airbyte.com/tutorials/incremental-data-synchronization#step-7). If you encounter issues with normalization, such as missing columns or failed connections, you might want to check the specific configurations and ensure that the Snowflake destination is set up correctly. For example, using SSO authentication in Snowflake might cause issues with DBT connections, as mentioned in the [forum post](https://discuss.airbyte.io/t/1328). For further assistance, you can also refer to the [Airbyte documentation on basic normalization](https://docs.airbyte.com/using-airbyte/core-concepts/basic-normalization#example). If you need more specific help, feel free to ask!
c
Is there a way to find it before establishing a connection
@kapa.ai
k
Sorry @Chidambara Ganapathy, I'm not sure what you mean. Please try again. 👋
c
Is there a way to find what tables will be normalised by Airbyte for a particular source before a connection with snowflake is established
@kapa.ai
k
Sorry @Chidambara Ganapathy, I'm not sure what you mean. Please try again. 👋
c
Is there a way to find what tables will be normalised by Airbyte for a particular source before a connection with snowflake is established @kapa.ai
k
A message from kapa.ai
Hmm, I don't know enough to give you a confident answer yet. However, you can refer to the [Airbyte documentation on basic normalization](https://docs.airbyte.com/using-airbyte/core-concepts/basic-normalization#example) for more details on how normalization works and what to expect. This documentation provides insights into how Airbyte handles normalization for different destinations, including Snowflake. If you have specific questions or need further assistance, you might also consider reaching out to the Airbyte community on their [forum](https://discuss.airbyte.io/) for more detailed guidance.
c
Is there a way to prevent normalisation in Airbyte by using api
Is there a way to prevent normalisation in Airbyte by using api @kapa.ai
k
A message from kapa.ai
Yes, you can prevent normalization in Airbyte by using the API. When creating a connection via the API, you can omit the 
operations
field or set it to an empty array to ensure that normalization is not applied. Here is an example of how you can create a connection without normalization using the Airbyte API:
Copy code
json {   "sourceId": "YOUR_SOURCE_ID",   "destinationId": "YOUR_DESTINATION_ID",   "syncCatalog": {     "streams": [       {         "stream": {           "name": "YOUR_STREAM_NAME",           "namespace": "YOUR_NAMESPACE",           "jsonSchema": {}         },         "config": {           "syncMode": "full_refresh",           "destinationSyncMode": "overwrite"         }       }     ]   },   "status": "active",   "name": "YOUR_CONNECTION_NAME",   "namespaceDefinition": "source",   "namespaceFormat": "${SOURCE_NAMESPACE}",   "prefix": "",   "operations": [] }
In this example, the 
operations
field is set to an empty array, which means no normalization will be applied to the data. For more details on how to use the Airbyte API, you can refer to the [Airbyte API documentation](https://reference.airbyte.com/reference/start). If you need further assistance or have more specific questions, feel free to ask!
Open in Slack
PreviousNext
Powered by