https://linen.dev logo
Join Slack
Powered by
My new airbyte deployment isn't allowing normaliza...
# replication-ask-ai
m
My new airbyte deployment isn't allowing normalization. "Normalization and Transformation operations are not supported for this connection" is what is showing. I deployed this version using helm and k8s.
k
New message text here
m
1. I don't think it's a memory issue 2. I can't see any normalization pod @kapa.ai
k
Based on the information provided, there could be a few reasons why you're not seeing any normalization pods. 1. Normalization is an optional operation in Airbyte. It's only available for certain destinations that support dbt execution. If you're not seeing any normalization pods, it could be because you haven't enabled normalization for your connections. You can enable or disable normalization in the "Normalization and Transformation" section when setting up your connection [source](https://docs.airbyte.com/understanding-airbyte/basic-normalization). 2. If you have enabled normalization but still can't see any normalization pods, it could be due to the version of Airbyte you're using. The normalization feature is tied to a specific Airbyte version and is released along with it. If you want to use a specific normalization version, you need to use the corresponding Airbyte version [source](https://docs.airbyte.com/understanding-airbyte/basic-normalization). 3. There could also be issues with your deployment. Some users have reported issues with normalization pods failing to initialize or getting stuck during initialization. These issues could be due to various factors such as memory issues, issues with the source or destination, or issues with the underlying infrastructure [source](https://discuss.airbyte.io/t/3132). If you're still having issues, I would recommend checking the logs for any error messages or clues as to what might be going wrong. You can also refer to this [video](

https://youtu.be/7sKy7VbcNSM▾

) for more details on troubleshooting normalization issues.
m
@kapa.ai if i deployed my airbyte with a helm chart using the commanfd 
helm install --values ./values.yaml airbyte airbyte/airbyte -n airbyte
but now. i want to downgrade to 0.50.19
k
A message from kapa.ai
To downgrade your Airbyte version on Kubernetes, you can follow these steps: 1. Turn off Airbyte by deleting the deployments. Replace 
<yournamespace>
with the namespace you used during the installation (in your case, it's 
airbyte
):
Copy code
bash kubectl delete deployments airbyte-db airbyte-worker airbyte-server airbyte-temporal airbyte-webapp --namespace=<yournamespace>
(Source: [Upgrading Airbyte on K8s](https://docs.airbyte.com/operator-guides/upgrading-airbyte)) 2. Update your Helm chart to use the version you want to downgrade to. In your 
values.yaml
file, specify the Airbyte version as 
0.50.19
. 3. Apply the changes:
Copy code
bash helm upgrade --install --values ./values.yaml airbyte airbyte/airbyte -n airbyte
4. After 2-5 minutes, check the status of the Airbyte pods. They should show `Running`:
Copy code
bash kubectl get pods | grep airbyte
5. Finally, run the following command to allow access to the UI/API:
Copy code
bash kubectl port-forward svc/airbyte-webapp-svc 8000:80
Please note that downgrading might cause compatibility issues depending on the changes between the two versions. Always ensure to backup your data before performing a downgrade.
m
it didn't downgrade @kapa.ai
k
A message from kapa.ai
I'm sorry to hear that the downgrade didn't work as expected. Based on the information provided, it's difficult to determine the exact cause of the issue. However, here are a few things you could try: 1. Check the Helm Chart Version: The Helm chart version and the Airbyte version are not always the same. Make sure you're using the correct Helm chart version that corresponds to the Airbyte version you want to downgrade to. Unfortunately, the extracts provided do not contain information on how to find the correct Helm chart version for a specific Airbyte version. 2. Uninstall and Reinstall: If the upgrade command didn't work, you could try uninstalling the current release and then reinstalling with the desired version. Be aware that this will delete your current deployment and any data not backed up:
Copy code
bash helm uninstall airbyte -n airbyte helm install --values ./values.yaml airbyte airbyte/airbyte --version 0.50.19 -n airbyte
3. Check for Errors: Look for any error messages in the output when you run the Helm command. This could give you clues as to why the downgrade is not working. 4. Reach Out to the Community: If you're still having trouble, consider reaching out to the Airbyte community on their [Slack channel](https://airbytehq.slack.com) or [discussion forum](https://discuss.airbyte.io). Other users may have encountered similar issues and could provide further assistance. Remember to always backup your data before making any changes to your deployment.
m
i got Error: chart "airbyte" matching 0.50.19 not found in airbyte index. (try 'helm repo update'): no chart version found for airbyte-0.50.19 @kapa.ai
k
A message from kapa.ai
I'm sorry for the confusion. The error message indicates that the Helm chart for the Airbyte version 
0.50.19
is not found in the Helm repository. This could be because the Helm chart for that specific version does not exist or it's not available in the repository. Here are a few things you could try: 1. Update the Helm Repository: Run 
helm repo update
to ensure your local Helm client has the latest version of the chart. 2. Check Available Chart Versions: You can check the available versions of the Airbyte chart in the Helm repository by running 
helm search repo airbyte
. This will list all the available versions. If 
0.50.19
is not listed, it means that there's no Helm chart available for that version. 3. Check Airbyte's GitHub Repository: Airbyte's Helm charts are stored in their GitHub repository. You can check there to see if the version you're looking for is available. However, the extracts provided do not contain a link to the GitHub repository. 4. Reach Out to the Community: If you're still having trouble, consider reaching out to the Airbyte community on their [Slack channel](https://airbytehq.slack.com) or [discussion forum](https://discuss.airbyte.io). Other users may have encountered similar issues and could provide further assistance. Remember, downgrading might cause compatibility issues depending on the changes between the two versions. Always ensure to backup your data before performing a downgrade.
m
when i check the bootloader logs i get this 2023-09-25 071129 ERROR i.a.b.Application(main):25 - Unable to bootstrap Airbyte environment. org.jooq.exception.DataAccessException: SQL [insert into "public"."actor_definition_version" ("id", "created_at", "updated_at", "actor_definition_id", "docker_repository", "docker_image_tag", "spec", "documentation_url", "protocol_version", "release_stage", "release_date", "normalization_repository", "normalization_tag", "supports_dbt", "normalization_integration_type", "allowed_hosts", "suggested_streams", "support_state") values (cast(? as uuid), cast(? as timestamp(6) with time zone), cast(? as timestamp(6) with time zone), cast(? as uuid), ?, ?, cast(? as jsonb), ?, ?, cast(? as "public"."release_stage"), cast(? as date), ?, ?, ?, ?, cast(? as jsonb), cast(? as jsonb), cast(? as "public"."support_state"))]; ERROR: null value in column "support_level" of relation "actor_definition_version" violates not-null constraint Detail: Failing row contains (ff5c87e5-aaa3-44a9-bf76-39059dd9b627, fe2b4084-3386-4d3b-9ad6-308f61a6f1e6, 2023-09-25 071129.124119+00, 2023-09-25 071129.124119+00, https://docs.airbyte.com/integrations/sources/harvest, airbyte/source-harvest, 0.1.18, {"supportsDBT": false, "advanced_auth": {"predicate_key": ["cred..., 0.2.0, null, null, null, null, null, {"hosts": ["api.harvestapp.com"]}, null, generally_available, supported, null). at org.jooq_3.17.8.POSTGRES.debug(Unknown Source) ~[?:?] at org.jooq.impl.Tools.translate(Tools.java:3313) ~[jooq-3.17.8.jar:?] at org.jooq.impl.DefaultExecuteContext.sqlException(DefaultExecuteContext.java:678) ~[jooq-3.17.8.jar:?] at org.jooq.impl.AbstractQuery.execute(AbstractQuery.java:355) ~[jooq-3.17.8.jar:?] at org.jooq.impl.AbstractDelegatingQuery.execute(AbstractDelegatingQuery.java:115) ~[jooq-3.17.8.jar:?] at io.airbyte.config.persistence.ConfigRepository.writeActorDefinitionVersion(ConfigRepository.java:3556) ~[io.airbyte.airbyte-config-config-persistence-0.0.99.jar:?] at io.airbyte.config.persistence.ConfigRepository.setActorDefinitionVersionForTagAsDefault(ConfigRepository.java:1046) ~[io.airbyte.airbyte-config-config-persistence-0.0.99.jar:?] at io.airbyte.config.persistence.ConfigRepository.writeSourceDefinitionAndDefaultVersion(ConfigRepository.java:798) ~[io.airbyte.airbyte-config-config-persistence-0.0.99.jar:?] at io.airbyte.config.persistence.ConfigRepository.lambda$writeSourceDefinitionAndDefaultVersion$30(ConfigRepository.java:773) ~[io.airbyte.airbyte-config-config-persistence-0.0.99.jar:?] at io.airbyte.db.Database.lambda$transaction$0(Database.java:27) ~[io.airbyte.airbyte-db-db-lib-0.0.99.jar:?] at org.jooq.impl.DefaultDSLContext.lambda$transactionResult0$3(DefaultDSLContext.java:552) ~[jooq-3.17.8.jar:?] at org.jooq.impl.Tools$3$1.block(Tools.java:5854) ~[jooq-3.17.8.jar:?] at java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3745) ~[?:?] at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3690) ~[?:?] at org.jooq.impl.Tools$3.get(Tools.java:5851) ~[jooq-3.17.8.jar:?] at org.jooq.impl.DefaultDSLContext.transactionResult0(DefaultDSLContext.java:600) ~[jooq-3.17.8.jar:?] at org.jooq.impl.DefaultDSLContext.transactionResult(DefaultDSLContext.java:524) ~[jooq-3.17.8.jar:?] at io.airbyte.db.Database.transaction(Database.java:27) ~[io.airbyte.airbyte-db-db-lib-0.0.99.jar:?] at io.airbyte.db.ExceptionWrappingDatabase.transaction(ExceptionWrappingDatabase.java:47) ~[io.airbyte.airbyte-db-db-lib-0.0.99.jar:?] at io.airbyte.config.persistence.ConfigRepository.writeSourceDefinitionAndDefaultVersion(ConfigRepository.java:772) ~[io.airbyte.airbyte-config-config-persistence-0.0.99.jar:?] at io.airbyte.config.init.ApplyDefinitionsHelper.applySourceDefinition(ApplyDefinitionsHelper.java:154) ~[io.airbyte.airbyte-config-init-0.0.99.jar:?] at io.airbyte.config.init.ApplyDefinitionsHelper.apply(ApplyDefinitionsHelper.java:102) ~[io.airbyte.airbyte-config-init-0.0.99.jar:?] at io.airbyte.config.init.ApplyDefinitionsHelper.apply(ApplyDefinitionsHelper.java:74) ~[io.airbyte.airbyte-config-init-0.0.99.jar:?] at io.airbyte.bootloader.DefaultPostLoadExecutor.execute(DefaultPostLoadExecutor.java:47) ~[io.airbyte-airbyte-bootloader-0.0.99.jar:?] at io.airbyte.bootloader.Bootloader.load(Bootloader.java:123) ~[io.airbyte-airbyte-bootloader-0.0.99.jar:?] at io.airbyte.bootloader.Application.main(Application.java:22) ~[io.airbyte-airbyte-bootloader-0.0.99.jar:?] Caused by: org.postgresql.util.PSQLException: ERROR: null value in column "support_level" of relation "actor_definition_version" violates not-null constraint @kapa.ai
k
A message from kapa.ai
The error message indicates that a null value is being inserted into the "support_level" column of the "actor_definition_version" table, which violates the not-null constraint of that column. This means that the "support_level" field is required and must contain a value. This could be due to a few reasons: 1. Data Issue: The data being inserted into the "actor_definition_version" table might be missing the "support_level" value. This could be due to an issue with the source data or a bug in the code that generates this data. 2. Code Issue: The code that's inserting data into the "actor_definition_version" table might be incorrectly handling null values for the "support_level" field. To resolve this issue, you would need to ensure that a valid "support_level" value is provided when inserting data into the "actor_definition_version" table. This might involve checking the source data for any missing "support_level" values or debugging the code to find out why the "support_level" value is not being set. Unfortunately, the provided extracts do not contain specific code or instructions on how to do this. You might need to refer to the Airbyte's source code or reach out to the Airbyte community on their [Slack channel](https://airbytehq.slack.com) or [discussion forum](https://discuss.airbyte.io) for further assistance.
m
@Marcos Marx (Airbyte) do you know anything about this error?
c
Did you perhaps install an older version of airbyte after having a newer version installed? I’ve seen this error recently in relation to a schema migration, and we don’t support downs in airbyte schema migrations right now due to flyway licensing
m
Oh wow. So I can't downgrade?
c
it is possible, but you will have to manually undo the database migration so you would have to reverse engineer the sql that applied this migration (e.g. write some sql that did the opposite of what the migration did), then run that against the db before downgrading. It is unfortunate that we cannot support downs with flyway, but it is a limitation of how flyway licenses their product
to be really safe you would want to walk back each migration one by one in the order in which they were applied (which is what a migrations system that applies downs would typically do). Most of the migrations would ideally not be breaking your application but it’s the only way to really guarantee a successful downgrade
in other words, if you are downgrading from 0.50.23 to 0.50.13, you would need to start with 0.50.23 sql file, run the opposite of that, and do that for every sql file, in order, for versions all the way back to the sql file of 0.50.13 (you would not need to run 0.50.13's sql file)
that is the exact mirror of the opposite of what is happening when you upgrade, btw, the application is taking these sql files and applying them one by one from the version you were on to the current version
m
I'm really lost 😓
c
suffice it to say that if you are not comfortable reverse engineering sql code in the folder I linked, there is not a straightforward path to downgrading airbyte if migrations create conflicts
l
FYI We were able to restore the airbyte database to a version prior to when we initiated the attempted upgrade (on kubernetes). This might be an easier fix than trying to reverse engineer the java migrations. There are some unfortunate tradeoffs however but we couldn't find another way
c
yeah, if you are ok with data loss restoring the db is another alternative. It is of course one of the most catastrophic ones, but it is a way
Open in Slack
PreviousNext
Powered by