https://linen.dev logo
Join Slack
Powered by
I am having the below error: ```2024-06-03 12:14:...
# ask-ai
p
I am having the below error:
Copy code
2024-06-03 12:14:09 ERROR i.a.c.Application(run):80 - Killing orchestrator because of an Exception
java.lang.IllegalStateException: baseUrl is invalid.
        at io.airbyte.api.client.generated.SourceApi.getSourceWithHttpInfo(SourceApi.kt:3690) ~[io.airbyte-airbyte-api-0.61.0.jar:?]
        at io.airbyte.api.client.generated.SourceApi.getSource(SourceApi.kt:653) ~[io.airbyte-airbyte-api-0.61.0.jar:?]
        at io.airbyte.workers.general.ReplicationWorkerFactory.create(ReplicationWorkerFactory.java:149) ~[io.airbyte-airbyte-commons-worker-0.61.0.jar:?]
        at io.airbyte.container_orchestrator.orchestrator.ReplicationJobOrchestrator.runJob(ReplicationJobOrchestrator.java:118) ~[io.airbyte-airbyte-container-orchestrator-0.61.0.jar:?]
        at io.airbyte.container_orchestrator.Application.run(Application.java:78) [io.airbyte-airbyte-container-orchestrator-0.61.0.jar:?]
        at io.airbyte.container_orchestrator.Application.main(Application.java:38) [io.airbyte-airbyte-container-orchestrator-0.61.0.jar:?
k
The error message you are encountering, 
java.lang.IllegalStateException: baseUrl is invalid
, suggests that there is an issue with the base URL configuration in your Airbyte setup. This is a known issue that has been discussed in the Airbyte community. From the knowledge sources, it appears that there was a request to make the base URL configurable in the Airbyte web application. The issue was raised because the Airbyte webapp always assumes it is located at the root 
/
, and there was no way to configure it to recognize a different base URL (e.g., 
/base/path/
). This issue was documented in the following GitHub issue: - [Webapp: Could you please provide configurable base url?](https://github.com/airbytehq/airbyte/issues/8167) Here is the relevant part of the original post from the issue: > I deployed airbyte on K8s. Instead of 
kubectl port-forward
, I used ingress and istio virtual service to make a url for airbyte webapp, like 
<https://example.com/base/path/>
. But it seems that airbyte webapp always assumes it is located at the root 
/
, and I cannot find how to configure the webapp so that it knows its base url is 
/base/path/
. Unfortunately, the issue was marked as stale and subsequently closed due to inactivity, and there is no indication that a solution was implemented: > This issue was closed because it has been inactive for 20 days since being marked as stale. Given this context, it seems that the base URL configuration issue might still be unresolved. Here are a few steps you can take to address this problem: 1. Check for Updates: Ensure you are using the latest version of Airbyte, as the issue might have been resolved in a newer release. 2. Community Support: Reach out to the Airbyte community for support. You can join the Airbyte Slack community and ask for help regarding this issue. Here is the link to join the Slack community: - [Airbyte Slack Community](https://slack.airbyte.com/) 3. Workaround: If the issue persists, consider using a workaround such as configuring your ingress and istio virtual service to map the base URL to the root path 
/
that Airbyte expects. If you need further assistance, you can refer to the [Airbyte documentation](https://docs.airbyte.com/) or the [Airbyte GitHub repository](https://github.com/airbytehq/airbyte) for more information. For more details on the original issue, you can view the closed GitHub issue [here](https://github.com/airbytehq/airbyte/issues/8167).
p
@kapa.ai, also api server is unable to start using a helm deployment on GKE. This is the error: Error: secret "airbyte-config-secrets" not found
k
It looks like the error you're encountering is due to a missing Kubernetes secret named 
airbyte-config-secrets
. This secret is essential for storing various configuration details required by Airbyte. To resolve this issue, you need to create the 
airbyte-config-secrets
secret in your Kubernetes cluster. Here are the steps to create this secret: 1. Create the Kubernetes Secret: You can create the secret using a Kubernetes manifest. Replace the placeholder values with your actual credentials and configuration details:
Copy code
yaml
   apiVersion: v1  
   kind: Secret  
   metadata:  
     name: airbyte-config-secrets  
   type: Opaque  
   stringData:  
     # Replace the example values below with your actual credentials and configuration details.
     license-key: <your-license-key>
     database-host: <your-database-host>
     database-port: <your-database-port>
     database-name: <your-database-name>
     database-user: <your-database-user>
     database-password: <your-database-password>
     instance-admin-email: <your-instance-admin-email>
     instance-admin-password: <your-instance-admin-password>
     s3-access-key-id: <your-s3-access-key-id>
     s3-secret-access-key: <your-s3-secret-access-key>
     aws-secret-manager-access-key-id: <your-aws-secret-manager-access-key-id>
     aws-secret-manager-secret-access-key: <your-aws-secret-manager-secret-access-key>
Apply this manifest using `kubectl`:
Copy code
sh
   kubectl apply -f path/to/your/secret-manifest.yaml
Alternatively, you can create the secret directly using `kubectl`:
Copy code
sh
   kubectl create secret generic airbyte-config-secrets \
     --from-literal=license-key='<your-license-key>' \
     --from-literal=database-host='<your-database-host>' \
     --from-literal=database-port='<your-database-port>' \
     --from-literal=database-name='<your-database-name>' \
     --from-literal=database-user='<your-database-user>' \
     --from-literal=database-password='<your-database-password>' \
     --from-literal=instance-admin-email='<your-instance-admin-email>' \
     --from-literal=instance-admin-password='<your-instance-admin-password>' \
     --from-literal=s3-access-key-id='<your-s3-access-key-id>' \
     --from-literal=s3-secret-access-key='<your-s3-secret-access-key>' \
     --from-literal=aws-secret-manager-access-key-id='<your-aws-secret-manager-access-key-id>' \
     --from-literal=aws-secret-manager-secret-access-key='<your-aws-secret-manager-secret-access-key>' \
     --namespace airbyte
2. Update Airbyte Configuration: Ensure that your 
airbyte.yml
configuration file references the 
airbyte-config-secrets
secret correctly. Here is an example configuration for AWS S3 users:
It looks like the error you're encountering is due to a missing Kubernetes secret named 
airbyte-config-secrets
. This secret is essential for storing various configuration details required by Airbyte. To resolve this issue, you need to create the 
airbyte-config-secrets
secret in your Kubernetes cluster. Here are the steps to create this secret: 1. Create the Kubernetes Secret: You can create the secret using a Kubernetes manifest. Replace the placeholder values with your actual credentials and configuration details:
Copy code
yaml    apiVersion: v1      kind: Secret      metadata:        name: airbyte-config-secrets      type: Opaque      stringData:        # Replace the example values below with your actual credentials and configuration details.      license-key: <your-license-key>      database-host: <your-database-host>      database-port: <your-database-port>      database-name: <your-database-name>      database-user: <your-database-user>      database-password: <your-database-password>      instance-admin-email: <your-instance-admin-email>      instance-admin-password: <your-instance-admin-password>      s3-access-key-id: <your-s3-access-key-id>      s3-secret-access-key: <your-s3-secret-access-key>      aws-secret-manager-access-key-id: <your-aws-secret-manager-access-key-id>      aws-secret-manager-secret-access-key: <your-aws-secret-manager-secret-access-key>
Apply this manifest using `kubectl`:
Copy code
sh    kubectl apply -f path/to/your/secret-manifest.yaml
Alternatively, you can create the secret directly using `kubectl`:
Copy code
sh    kubectl create secret generic airbyte-config-secrets \      --from-literal=license-key='<your-license-key>' \      --from-literal=database-host='<your-database-host>' \      --from-literal=database-port='<your-database-port>' \      --from-literal=database-name='<your-database-name>' \      --from-literal=database-user='<your-database-user>' \      --from-literal=database-password='<your-database-password>' \      --from-literal=instance-admin-email='<your-instance-admin-email>' \      --from-literal=instance-admin-password='<your-instance-admin-password>' \      --from-literal=s3-access-key-id='<your-s3-access-key-id>' \      --from-literal=s3-secret-access-key='<your-s3-secret-access-key>' \      --from-literal=aws-secret-manager-access-key-id='<your-aws-secret-manager-access-key-id>' \      --from-literal=aws-secret-manager-secret-access-key='<your-aws-secret-manager-secret-access-key>' \      --namespace airbyte
2. Update Airbyte Configuration: Ensure that your 
airbyte.yml
configuration file references the 
airbyte-config-secrets
secret correctly. Here is an example configuration for AWS S3 users: ```yaml global: storage: type: s3 storageSecretName: airbyte-config-secrets bucket: log: airbyte-storage state: airbyte-storage workloadOutput: airbyte-storage s3: region: <aws-region> authenticationType: credentials # Use "credentials" or "instanceProfile" accessKeyIdSecretKey: aws-secret-manager-access-key-id # Omit if using instanceProfile secretAccessKeySecretKey: aws-secret-manager-secret-access-key # Omit if using instanceProfile secretsManager: type: awsSecretManager storageSecret
p
@kapa.ai How can I override the secret - the secret is on GCP and it has a different name
k
A message from kapa.ai
To override the secret in Airbyte when using Google Cloud Platform (GCP) and a different secret name, you need to update your Helm values and Kubernetes secrets configuration accordingly. Here are the steps to achieve this: 1. Create the Kubernetes Secret: First, create a Kubernetes secret with your GCP credentials. Replace 
<CREDENTIALS_JSON_BLOB>
with your actual GCP credentials JSON blob:
Copy code
yaml    apiVersion: v1      kind: Secret      metadata:        name: your-custom-secret-name      type: Opaque      stringData:        gcp.json: <CREDENTIALS_JSON_BLOB>
Apply this manifest using `kubectl`:
Copy code
sh    kubectl apply -f path/to/your/secret-manifest.yaml
Alternatively, you can create the secret directly from a file:
Copy code
sh    kubectl create secret generic your-custom-secret-name --from-file=gcp.json=<path-to-your-credentials-file>.json
2. Update the Helm Values: Update your 
values.yaml
file to reference the custom secret name and the path to the GCP credentials. Here is an example configuration:
Copy code
yaml    global:        storage:          type: gcs          storageSecretName: your-custom-secret-name          bucket:            log: airbyte-storage            state: airbyte-storage            workloadOutput: airbyte-storage          gcs:            authenticationType: credentials            projectId: <your-project-id>            credentialsPath: /secrets/gcs-log-creds/gcp.json            secretsManager:          type: googleSecretManager          storageSecretName: your-custom-secret-name          googleSecretManager:            authenticationType: credentials            projectId: <your-project-id>            credentialsSecretKey: gcp.json
3. Deploy Airbyte with the Updated Configuration: Run the following command to deploy Airbyte with the updated `values.yaml`:
Copy code
sh    helm upgrade --install <RELEASE_NAME> airbyte/airbyte -n <NAMESPACE> --values /path/to/values.yaml --version 0.54.69
By following these steps, you can override the default secret name and use your custom secret stored in GCP. This ensures that Airbyte can access the necessary credentials and configuration details from the specified secret. For more details, you can refer to the [Airbyte documentation on deploying Airbyte on Kubernetes using Helm](https://docs.airbyte.com/deploying-airbyte/on-kubernetes-via-helm#update-the-valuesyaml-with-the-gcs-logging-information-below) and the [implementation guide for creating Kubernetes secrets](https://docs.airbyte.com/enterprise-setup/implementation-guide#creating-a-kubernetes-secret).
p
@kapa.ai but why is the API server looking for this secret? What value does it need to read from there?
k
A message from kapa.ai
The Airbyte API server is looking for the 
airbyte-config-secrets
secret because it contains essential configuration details and credentials required for the proper functioning of Airbyte. These details typically include sensitive information such as API keys, database credentials, and other secrets that Airbyte needs to connect to various data sources and destinations. Here are some common values that the API server might need to read from the 
airbyte-config-secrets
secret: - Database Credentials: Information required to connect to the Airbyte internal database, such as the database host, port, name, user, and password. - API Keys and Secrets: Credentials for connecting to various data sources and destinations, such as AWS S3, Google Cloud Storage, and other services. - Instance Admin Credentials: Email and password for the instance admin user. - Cloud Provider Credentials: Access keys and secret keys for cloud providers like AWS or GCP, used for storage and other integrations. For example, a typical Kubernetes secret manifest for Airbyte might look like this:
Copy code
yaml apiVersion: v1   kind: Secret   metadata:     name: airbyte-config-secrets   type: Opaque   stringData:     license-key: <your-license-key>   database-host: <your-database-host>   database-port: <your-database-port>   database-name: <your-database-name>   database-user: <your-database-user>   database-password: <your-database-password>   instance-admin-email: <your-instance-admin-email>   instance-admin-password: <your-instance-admin-password>   s3-access-key-id: <your-s3-access-key-id>   s3-secret-access-key: <your-s3-secret-access-key>   aws-secret-manager-access-key-id: <your-aws-secret-manager-access-key-id>   aws-secret-manager-secret-access-key: <your-aws-secret-manager-secret-access-key>
In your case, if the secret is stored in GCP and has a different name, you need to ensure that the Helm values and Kubernetes configuration are updated to reference the correct secret name and path to the credentials. This way, the API server can correctly locate and read the necessary values from the specified secret. For more details on configuring secrets for Airbyte, you can refer to the [Airbyte documentation on deploying Airbyte on Kubernetes using Helm](https://docs.airbyte.com/deploying-airbyte/on-kubernetes-via-helm#update-the-valuesyaml-with-the-gcs-logging-information-below) and the [implementation guide for creating Kubernetes secrets](https://docs.airbyte.com/enterprise-setup/implementation-guide#creating-a-kubernetes-secret).
a
Hi @Pedro Peixoto Do you find the solution ? I have this error and I don't understand why.
p
Hi @Aurélien Bstl, no... still struggling with lot of similar issues, tried configuring a baseUrl, but now get similar errors:
Copy code
2024-06-03 16:39:01 WARN i.t.i.a.ActivityTaskExecutors$BaseActivityTaskExecutor(execute):126 - Activity failure. ActivityId=6021724f-1aa9-3a97-8bd5-7693f4345f52, activityType=RecordWorkflowCountMetric, attempt=3
java.lang.IllegalStateException: baseUrl is invalid.
a
That's weird. I hasn't been this issue last Friday when I setup the Orchestrator.
t
I have the problem running Airbyte OSS via Docker
p
If you guys want I'm following on this issue thread - https://github.com/airbytehq/airbyte/issues/38854
t
rollback to 0.60.1 and it works..
p
Yep, that was the solution I found most suitable unfortunately @Thomas
Open in Slack
PreviousNext
Powered by