Hi @kapa.ai, Could you please provide a summary of all reported issues related to upgrading open source Airbyte to latest version 1.0 (https://airbyte.com/v1)? I'm particularly interested in:Common upgrade challenges 1. Any breaking changes that users have encountered 2. Compatibility issues with existing connectors or configurations 3. Performance or stability problems after the upgrade 4. Any workarounds or solutions that have been suggested for these issues This information will help us prepare for our own upgrade process

Sure, here's a summary of the issues related to upgrading Airbyte to version 1.0 based on the provided knowledge sources: 1. Common Upgrade Challenges: - Users have reported issues when upgrading Airbyte in a Docker Compose environment. The upgrade process is described as brittle and can often result in Airbyte not restarting correctly ([source](https://github.com/airbytehq/airbyte/discussions/40599)). - Some users have encountered errors while upgrading Airbyte with existing connections in an air gap environment ([source](https://github.com/airbytehq/airbyte/issues/44441)). 2. Breaking Changes: - The upgrade to version 1.0 removes the option to use "normalization" with Microsoft SQL Server, Clickhouse, and Oracle. This change will likely require updates to downstream dbt / SQL models ([source](https://docs.airbyte.com/integrations/destinations/mssql-migrations#ms-sql-server-migration-guide), [source](https://docs.airbyte.com/integrations/destinations/clickhouse-migrations#clickhouse-migration-guide), [source](https://docs.airbyte.com/integrations/destinations/oracle-migrations#oracle-migration-guide)). 3. Compatibility Issues: - The upgrade changes the schema and database of Airbyte's "raw" tables to be compatible with the new Destinations V2 format. This change removes the ability to do deduplicated syncs with Microsoft SQL Server, Clickhouse, and Oracle ([source](https://docs.airbyte.com/integrations/destinations/mssql-migrations#ms-sql-server-migration-guide), [source](https://docs.airbyte.com/integrations/destinations/clickhouse-migrations#clickhouse-migration-guide), [source](https://docs.airbyte.com/integrations/destinations/oracle-migrations#oracle-migration-guide)). 4. Performance or Stability Problems: - Some users have reported that Airbyte ingestion became slower after a connector upgrade to 1.X. In one case, the ingestion time increased to more than double the original time ([source](https://discuss.airbyte.io/t/2416)). 5. Workarounds or Solutions: - For the Docker Compose upgrade issue, Airbyte has prioritized building a new local deployment tool,

abctl

, that deploys Airbyte locally using Kubernetes with minimal onboarding friction ([source](https://github.com/airbytehq/airbyte/discussions/40599)). - For the breaking changes related to "normalization", users are provided with SQL queries to migrate their data from the old raw table to the new raw table ([source](https://docs.airbyte.com/integrations/destinations/mssql-migrations#ms-sql-server-migration-guide), [source](https://docs.airbyte.com/integrations/destinations/clickhouse-migrations#clickhouse-migration-guide), [source](https://docs.airbyte.com/integrations/destinations/oracle-migrations#oracle-migration-guide)). - For the performance issue, the user's concern was escalated to GitHub for further input from the engineering team ([source](https://discuss.airbyte.io/t/2416)).

Sure, here's a summary of the issues related to upgrading Airbyte to version 1.0 based on the provided knowledge sources: 1. Common Upgrade Challenges: - Users have reported issues when upgrading Airbyte in a Docker Compose environment. The upgrade process is described as brittle and can often result in Airbyte not restarting correctly ([source](https://github.com/airbytehq/airbyte/discussions/40599)). - Some users have encountered errors while upgrading Airbyte with existing connections in an air gap environment ([source](https://github.com/airbytehq/airbyte/issues/44441)). 2. Breaking Changes: - The upgrade to version 1.0 removes the option to use "normalization" with Microsoft SQL Server, Clickhouse, and Oracle. This change will likely require updates to downstream dbt / SQL models ([source](https://docs.airbyte.com/integrations/destinations/mssql-migrations#ms-sql-server-migration-guide), [source](https://docs.airbyte.com/integrations/destinations/clickhouse-migrations#clickhouse-migration-guide), [source](https://docs.airbyte.com/integrations/destinations/oracle-migrations#oracle-migration-guide)). 3. Compatibility Issues: - The upgrade changes the schema and database of Airbyte's "raw" tables to be compatible with the new Destinations V2 format. This change removes the ability to do deduplicated syncs with Microsoft SQL Server, Clickhouse, and Oracle ([source](https://docs.airbyte.com/integrations/destinations/mssql-migrations#ms-sql-server-migration-guide), [source](https://docs.airbyte.com/integrations/destinations/clickhouse-migrations#clickhouse-migration-guide), [source](https://docs.airbyte.com/integrations/destinations/oracle-migrations#oracle-migration-guide)). 4. Performance or Stability Problems: - Some users have reported that Airbyte ingestion became slower after a connector upgrade to 1.X. In one case, the ingestion time increased to more than double the original time ([source](https://discuss.airbyte.io/t/2416)). 5. Workarounds or Solutions: - For the Docker Compose upgrade issue, Airbyte has prioritized building a new local deployment tool,

abctl

, that deploys Airbyte locally using Kubernetes with minimal onboarding friction ([source](https://github.com/airbytehq/airbyte/discussions/40599)). - For the breaking changes related to "normalization", users are provided with SQL queries to migrate their data from the old raw table to the new raw table ([source](https://docs.airbyte.com/integrations/destinations/mssql-migrations#ms-sql-server-migration-guide), [source](https://docs.airbyte.com/integrations/destinations/clickhouse-migrations#clickhouse-migration-guide), [source](https://docs.airbyte.com/integrations/destinations/oracle-migrations#oracle-migration-guide)). - For the performance issue, the user's concern was escalated to GitHub for further input from the engineering team ([source](https://discuss.airbyte.io/t/2416)). Please note that these are the issues and solutions identified from the provided knowledge sources. There might be other issues or solutions not covered in these sources. For a comprehensive understanding, it's recommended to review the official [Airbyte GitHub issues](https://github.com/airbytehq/airbyte/issues) and [Airbyte discussions](https://discuss.airbyte.io/).

@kapa.ai how to create nested field as individual field on destination table in low-code connector

A message from kapa.ai

In the low-code connectors, nested fields are not directly supported. However, you can use transformations to add the nested field to the root of the record. Here's how you can do it: 1. Use the "Add field" transformation. This transformation allows you to add a new field to the record. You can find more details about this in the [Airbyte documentation](https://docs.airbyte.com/connector-development/connector-builder-ui/record-processing#adding-fields). 2. Use the

{{ stream_partition.<identifier> }}

interpolation placeholder to attach the partition a record belongs to the record itself. This can be useful when you want to use the partition value during analysis in the destination. You can find more details about this in the [Airbyte documentation](https://docs.airbyte.com/connector-development/connector-builder-ui/partitioning#adding-the-partition-value-to-the-record). Here's an example of how you can add a nested field to the root of the record:

json {   "id": 999,   "author": "Jon Doe",   "note": "Great product!" }

You can add the order id by taking the following steps: - Make sure the "Current Parameter Value Identifier" is set to

order

. - Add an "Add field" transformation with "Path"

order_id

and "Value"

{{ stream_partition.order }}

. After these steps, the record will look like this:

json {   "id": 999,   "author": "Jon Doe",   "note": "Great product!",   "order_id": 123 }

Please note that this method is not recommended for doing projections (like concatenating firstname and lastname into a single "name" field). It's usually beneficial to leave these tasks to a later stage in the data pipeline ([source](https://docs.airbyte.com/connector-development/connector-builder-ui/record-processing#adding-fields)).

how to move all nested field to root level, without knowing what fields will be coming @kapa.ai

A message from kapa.ai

Airbyte currently does not support automatically flattening all nested fields to the root level without specifying the fields. The transformations available in Airbyte, such as

AddFields

and

RemoveFields

, require you to specify the field paths that you want to add or remove. However, you can use the

*

character as a placeholder to select all children at the current position in the path. For example, setting Field Path to

data

,

*

,

record

will return all records under

data

and

record

([source](https://docs.airbyte.com/connector-development/connector-builder-ui/record-processing#fields-nested-in-arrays)). If you want to flatten all nested fields to the root level, you might need to process the data after it has been loaded into your destination. This can be done using SQL or a data processing tool that supports nested data structures. For more complex transformations, you might want to consider using a tool like dbt, which is designed for transforming data in your warehouse. Airbyte has built-in support for dbt, allowing you to run dbt transformations as part of your Airbyte syncs ([source](https://docs.airbyte.com/understanding-airbyte/basic-normalization)). Remember to always check the schema of your data source and plan your transformations accordingly.