https://linen.dev logo
Join Slack
Powered by
We are using `airbytehq/airbyte` Terraform provide...
# replication-ask-ai
s
We are using 
airbytehq/airbyte
Terraform provider 
v1.3.0
against a self-managed Airbyte deployment. Every Airbyte resource fails with: failed to get token: failed to decode token response: invalid character '<' looking for beginning of value This affects declarative source definitions, generic sources, and destinations, so it appears to fail during provider authentication rather than in an individual resource configuration. Our Buildkite script obtains client credentials from AWS Secrets Manager and tries to retrieve a token from the Airbyte Public API: bash DNS_NAME="$ENDPOINT:$PORT" BEARER_TOKEN=$(curl --request POST \ --url "$DNS_NAME/api/public/v1/applications/token" \ --header 'accept: application/json' \ --header 'Content-Type: application/json' \ --data "{\"client_id\":\"$CLIENT_ID\",\"client_secret\":\"$CLIENT_SECRET\",\"grant_type\":\"client_credentials\"}" \ | jq -r .access_token) SERVER_URL="http://$DNS_NAME/api/public/v1" Terraform provider configuration is generated into 
secrets.tfvars
; we are tracing the exact provider fields now. The 
<
parse error suggests the provider’s token request receives HTML (possibly a redirect, invalid endpoint, proxy/SSO page, wrong HTTP/HTTPS scheme, or unexpected URL path). What exact provider configuration and endpoint format does 
airbytehq/airbyte
v1.3.0 expect for self-managed Airbyte authentication using client credentials? Should the provider host/server URL include 
/api/public/v1
, and what token endpoint/path should it use? Are there known issues with NodePort URLs, HTTP vs HTTPS, or Airbyte Public API client credentials that result in an HTML response instead of JSON?
k
Thinking completed The error 
invalid character '<' looking for beginning of value
occurs because Terraform is attempting to parse an HTML response (such as a 
<!DOCTYPE html>
404 page, proxy error, or redirect page) as JSON when requesting an access token. Provider Configuration & Endpoint Format For the 
airbytehq/airbyte
Terraform provider, the configuration depends on whether you let Terraform fetch the token via OAuth2 Client Credentials or pass a pre-fetched bearer token directly [airbyte Provider]. 1. Expected server_url For self-managed Airbyte deployments, the provider documentation specifies that 
server_url
must include the public API path [Terraform provider — Pre-1.0; Getting Started]:
Copy code
server_url = "http://<host>:<port>/api/public/v1"
(Defaults to 
https://api.airbyte.com/v1
for Airbyte Cloud if omitted) [airbyte Provider; Getting Started]. 2. Using client_id and client_secret (with token_url) The provider schema supports [airbyte Provider]: • 
client_id
(OAuth2 Client ID) • 
client_secret
(OAuth2 Client Secret) • 
token_url
(OAuth2 token endpoint URL) • 
server_url
(API Base URL) If you configure 
client_id
and 
client_secret
on self-managed Airbyte, ensure you also explicitly configure 
token_url
[airbyte Provider]. The token endpoint for self-managed Airbyte is [Get an access token]:
Copy code
provider "airbyte" {
  server_url    = "http://<host>:<port>/api/public/v1"
  token_url     = "http://<host>:<port>/api/public/v1/applications/token"
  client_id     = var.client_id
  client_secret = var.client_secret
}
If 
token_url
is omitted, the provider may default to Cloud or construct an incorrect path relative to 
server_url
, hitting an endpoint that serves HTML instead of JSON. 3. Alternative: Using bearer_auth Since your CI pipeline already extracts 
BEARER_TOKEN
via 
curl
, you can bypass the provider's token exchange entirely by passing the bearer token directly to the provider schema [airbyte Provider]:
Copy code
provider "airbyte" {
  server_url  = "http://<host>:<port>/api/public/v1"
  bearer_auth = var.bearer_token
}
Potential Causes for the HTML (<) Response 1. Missing or Default 
token_url
: When relying on 
client_id
and 
client_secret
, omitting 
token_url
can cause the provider to call an invalid URL or Airbyte Cloud's OAuth endpoint, returning an HTML error page [airbyte Provider]. 2. Trailing Slashes or Incorrect Paths: The documentation shows public API endpoints structured under 
/api/public/v1/applications/token
[Get an access token]. Ensure extra trailing slashes or incomplete paths are not causing reverse proxy rewrites or 404 HTML routing. 3. HTTP vs. HTTPS / NodePort & Reverse Proxies: If Airbyte or an ingress controller enforces HTTPS, hitting an 
http://
endpoint can result in an HTTP 301/302 redirect returning an HTML payload body (
<html...
) instead of following the redirect to JSON. Uncertainty and Limitations The knowledge sources do not explicitly mention the internal default value of 
token_url
when targeting a self-hosted 
server_url
, nor do they contain documented issues specifically mentioning Kubernetes NodePort behavior with the v1.3.0 provider. However, explicitly defining 
token_url
or supplying 
bearer_auth
directly avoids the provider's automatic token resolution. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
s
this is in the CI/CD push configs file # Get client ID and secret from secrets manager. instanceAdminClientId within the secret is client id CLIENT_ID=$(aws secretsmanager get-secret-value --secret-id airbytes/data-platform/${ENVIRONMENT}/auth --query SecretString --output text | jq -r .instanceAdminClientId) CLIENT_SECRET=$(aws secretsmanager get-secret-value --secret-id airbytes/data-platform/${ENVIRONMENT}/auth --query SecretString --output text | jq -r .instanceAdminClientSecret) # Get the endpoints ENDPOINT=$(kubectl get nodes -o jsonpath='{.items[0].status.addresses[?(@.type=="InternalIP")].address}') PORT=$(kubectl get service $SERVICE_NAME -n airbyte -o jsonpath='{.spec.ports[0].nodePort}') DNS_NAME="$ENDPOINT:$PORT" # if ENDPOINT is not set, or PORT is not set exit if [ -z "$DNS_NAME" ]; then echo "Could not find the airbytes load balancer" exit 1 fi # get workspace id BEARER_TOKEN=$(curl --connect-timeout 10 --request POST --url "$DNS_NAME/api/public/v1/applications/token" --header 'accept: application/json' --header 'Content-Type: application/json' --data "{\"client_id\":\"$CLIENT_ID\",\"client_secret\":\"$CLIENT_SECRET\",\"grant_type\":\"client_credentials\"}" | jq -r .access_token) WP_API_URL="$DNS_NAME/api/public/v1/workspaces?workspaceIds=&includeDeleted=false" WORKSPACE_ID=$(curl --connect-timeout 10 --request GET --url $WP_API_URL --header 'accept: application/json' --header "Authorization: Bearer $BEARER_TOKEN" | jq -r '.data[0].workspaceId') # prepare server URL SERVER_URL="http://$DNS_NAME/api/public/v1" cp -r ./airbytes_config/ ./tmp/airbytes_config
@kapa.ai, analyze the above code snippet and would any changes would suffice?
k
Thinking completed Based on the knowledge sources, there are two key issues in your code snippet: 1. The grant type field name is grant-type, not grant_type In the official Airbyte API documentation for exchanging client credentials on self-managed deployments, the JSON payload requires the field key 
grant-type
(with a hyphen), not 
grant_type
(with an underscore) [Get an access token]:
Copy code
{
  "client_id": "<YOUR_CLIENT_ID>",
  "client_secret": "<YOUR_CLIENT_SECRET>",
  "grant-type": "client_credentials"
}
2. Missing Scheme in curl URL Your script sets 
DNS_NAME="$ENDPOINT:$PORT"
and runs:
Copy code
--url "$DNS_NAME/api/public/v1/applications/token"
Because the scheme (
http://
or 
https://
) is omitted, 
curl
or intermediate network proxies may fail to route properly or hit an unexpected endpoint, leading to an HTML response. The scheme should be included [Get an access token]. Corrected Script Updating the scheme and fixing the 
grant-type
parameter:
Copy code
DNS_NAME="$ENDPOINT:$PORT"
BEARER_TOKEN=$(curl --request POST \
  --url "http://$DNS_NAME/api/public/v1/applications/token" \
  --header 'accept: application/json' \
  --header 'Content-Type: application/json' \
  --data "{\"client_id\":\"$CLIENT_ID\",\"client_secret\":\"$CLIENT_SECRET\",\"grant-type\":\"client_credentials\"}" \
  | jq -r .access_token)

SERVER_URL="http://$DNS_NAME/api/public/v1"
Will this change suffice for Terraform? I'm sorry, but there's no information about the exact contents of your generated 
secrets.tfvars
or your provider block in the knowledge sources, so whether this change alone will suffice depends on how you configure the provider: 1. If you pass 
BEARER_TOKEN
into Terraform: If you pass the pre-fetched token into Terraform's 
bearer_auth
attribute, this fix will suffice because the provider will not attempt its own token exchange [airbyte Provider]:provider "airbyte" { server_url = var.server_url bearer_auth = var.bearer_token } 2. If you configure 
client_id
and 
client_secret
in the Terraform provider: If you rely on Terraform to obtain the token directly rather than passing 
bearer_auth
, the provider itself makes the token request. In that case, you must also provide 
token_url = "http://$DNS_NAME/api/public/v1/applications/token"
in your Terraform provider block; otherwise, the provider may attempt to fetch credentials from a default or cloud token URL [airbyte Provider; Get an access token]. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
s
So @kapa.ai summarize the solutions in detail and provide the steps for it.
k
Thinking completed To resolve the 
failed to decode token response: invalid character '<'
error, you have two primary solutions. The error occurs because an endpoint is returning an HTML page instead of the JSON token response expected by either 
jq
or the Terraform provider. Solution 1: Use bearer_auth (Recommended if generating tokens in CI/CD) Since your Buildkite script already fetches an access token, pass that token directly to Terraform using the 
bearer_auth
attribute [airbyte Provider]. This bypasses the provider's internal token exchange mechanism entirely. Step 1: Fix the curl command in your Buildkite script Update the request URL to include the protocol scheme (
http://
or 
https://
), and change 
grant_type
to 
grant-type
(hyphenated, as required by the API) [Get an access token]:
Copy code
DNS_NAME="$ENDPOINT:$PORT"
SERVER_URL="http://$DNS_NAME/api/public/v1"

BEARER_TOKEN=$(curl --request POST \
  --url "http://$DNS_NAME/api/public/v1/applications/token" \
  --header 'accept: application/json' \
  --header 'Content-Type: application/json' \
  --data "{\"client_id\":\"$CLIENT_ID\",\"client_secret\":\"$CLIENT_SECRET\",\"grant-type\":\"client_credentials\"}" \
  | jq -r .access_token)
Step 2: Pass the token into secrets.tfvars Write the token and server URL to your `secrets.tfvars`:
Copy code
cat <<EOF > secrets.tfvars
server_url   = "$SERVER_URL"
bearer_token = "$BEARER_TOKEN"
EOF
Step 3: Configure the Terraform provider Use 
bearer_auth
in your 
main.tf
[airbyte Provider]:
Copy code
terraform {
  required_providers {
    airbyte = {
      source  = "airbytehq/airbyte"
      version = "1.3.0"
    }
  }
}

provider "airbyte" {
  server_url  = var.server_url
  bearer_auth = var.bearer_token
}
Solution 2: Let Terraform handle token retrieval via client_id, client_secret, and token_url If you want Terraform to handle the OAuth2 token exchange directly, you do not need the 
curl
command in your shell script [Get an access token]. However, on self-managed deployments, you must explicitly provide the 
token_url
so it does not default to Airbyte Cloud [airbyte Provider]. Step 1: Export values to secrets.tfvars Export your base URL, token endpoint URL, and credentials:
Copy code
DNS_NAME="$ENDPOINT:$PORT"

cat <<EOF > secrets.tfvars
server_url    = "http://$DNS_NAME/api/public/v1"
token_url     = "http://$DNS_NAME/api/public/v1/applications/token"
client_id     = "$CLIENT_ID"
client_secret = "$CLIENT_SECRET"
EOF
Step 2: Configure the Terraform provider Configure the provider schema with 
token_url
, 
client_id
, and 
client_secret
[airbyte Provider]:
Copy code
terraform {
  required_providers {
    airbyte = {
      source  = "airbytehq/airbyte"
      version = "1.3.0"
    }
  }
}

provider "airbyte" {
  server_url    = var.server_url
  token_url     = var.token_url
  client_id     = var.client_id
  client_secret = var.client_secret
}
👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
2 Views
Open in Slack
PreviousNext
Powered by