https://linen.dev logo
Join Slack
Powered by
why are my records being landed in databricks with...
# replication-ask-ai
d
why are my records being landed in databricks with all columns in a single column json object called _airbyte_data instead of individual columns for each field?
k
The behavior you're observing is actually the expected behavior of Airbyte when syncing data to Databricks. Airbyte typically creates three columns in the destination: 
_airbyte_ab_id
, 
_airbyte_emitted_at
, and 
_airbyte_data
. The 
_airbyte_data
column contains the data from your source as a JSON string. This is because Airbyte first converts the incoming data stream in JSON schema to an Avro schema, then the JSON object is converted to an Avro record, and finally, the Avro record is outputted to the Parquet format. This process has arbitrary rules and limitations due to the fact that the data stream can come from any data source. If you want to flatten the JSON blob into multiple columns, you might want to consider using a different destination connector that supports normalization (flattening), such as the CSV output in Azure Blob Storage or R2 destinations. With these connectors, when root level normalization is selected, the root level fields are expanded into separate columns. Please refer to the following sources for more information: - [Databricks Destination Output Schema](https://docs.airbyte.com/integrations/destinations/databricks#output-schema) - [Azure Blob Storage Destination CSV Output](https://docs.airbyte.com/integrations/destinations/azure-blob-storage#csv) - [R2 Destination CSV Output](https://docs.airbyte.com/integrations/destinations/r2#csv)
d
@kapa.ai why doesn't databricks destination support normalization?
k
Normalization is a feature that has been deprecated in Airbyte and will be removed in future releases. This is why it is not supported in the Databricks destination. The deprecation is due to the shift towards Destination v2, which no longer uses dbt for normalization. Instead, the normalization option should be set to Raw in Destination v2. For more details, you can refer to these discussions: - [Github Issue 8090](https://github.com/airbytehq/airbyte/issues/8090) - [Github Issue 22197](https://github.com/airbytehq/airbyte/issues/22197) - [Github Issue 28826](https://github.com/airbytehq/airbyte/issues/28826)
d
@kapa.ai Why is normalization being deprecated?
k
A message from kapa.ai
Normalization is being deprecated in Airbyte for several reasons: 1. Focus on reliability: Airbyte is focusing on improving the reliability of its core features. The team has decided to invest their time in improving features that work as intended or investing that time elsewhere where it's more needed. 2. Low usage: The Custom Normalization feature, which triggers dbt jobs, is only used by a small minority of Airbyte OSS users. The time spent maintaining this feature could be better spent on developing other features that benefit a larger portion of the user community. 3. Simplification: As the use cases for Airbyte OSS expand into mature production deployments on massive scales, the team is focusing on executing the fundamentals of Extract & Load very well, rather than trying to do too many things at once. The deprecation of the Custom Normalization feature is part of a strategy to invest less in one area to invest more heavily in other platform areas that are critical to ensuring a fantastic user experience. For more details, you can refer to this [discussion](https://github.com/airbytehq/airbyte/discussions/34860) on the Airbyte Github page.
Open in Slack
PreviousNext
Powered by