Hi all, I'm making all the configurations programmatically. I was able to do it for the source and destination, but I'm having issues creating the connection. To test it out, I used the PokeAPI as source & Bigquery as destination. Also I was able to create the connection manually through the UI, and it syncs just fine. I'm using Airbyte OSS deployed on GCP using compute engine. The 8006 port is exposed. Here are the 3 options that i used to create the connection via code: 1. Airbyte Terraform Provider code:

resource "airbyte_connection" "pokeapi_bigquery_connection" {
  name = "Poke API Connection"
  source_id = airbyte_source_pokeapi.my_source_pokeapi.source_id
  destination_id = airbyte_destination_bigquery.my_destination_bigquery.destination_id
  configurations = {
    streams = [
      {
        name      = "pokemon"
        sync_mode = "full_refresh_overwrite"
      }
    ]
  }
  schedule = {
    schedule_type = "manual"
  }
  namespace_definition = "custom_format"
  status = "inactive"
}

response:

│ Error: failure to invoke API
│ 
│   with airbyte_connection.pokeapi_bigquery_connection,
│   on <http://main.tf|main.tf> line 50, in resource "airbyte_connection" "pokeapi_bigquery_connection":
│   50: resource "airbyte_connection" "pokeapi_bigquery_connection" {
│ 
│ error sending request: Post "http://<server_url>:8006/v1/connections": context deadline
│ exceeded (Client.Timeout exceeded while awaiting headers)

2. POST Request API code:

import requests
from requests.auth import HTTPBasicAuth

url = "http://<server_url>:8006/v1/connections/create"

payload = {
    "name": "PokeAPI-to-Bigquery",
    "sourceId": "<source_id>",
    "destinationId": "<destination_id>",
    "workspaceId": "<workspace_id>",
    "status": "active",
    "schedule": { "scheduleType": "manual" },
    "configurations": {
        'streams': [
            {
                'name': 'pokemon',
                'syncMode': 'full_refresh_overwrite'
            }
        ],
    },
    "dataResidency": "auto",
    "namespaceDefinition": "destination",
    "namespaceFormat": None,
    "nonBreakingSchemaUpdatesBehavior": "ignore",
}

headers = {
    "accept": "application/json",
    "content-type": "application/json"
}

response = <http://requests.post|requests.post>(
    url,
    json=payload,
    headers=headers,
    auth=HTTPBasicAuth('<username>', '<password>')
)

response:

{
  "type": "about:blank",
  "status": 405
}

3. airbyte-api-python-sdk code:

import airbyte
from airbyte.models import shared

s = airbyte.Airbyte(
    server_url='http://<server_url>:8006/v1/connections',
    security=shared.Security(
        basic_auth=shared.SchemeBasicAuth(
            password="<password>",
            username="<username>",
        ),
    ),
)

req = shared.ConnectionCreateRequest(
    configurations=shared.StreamConfigurations(
        streams=[
            shared.StreamConfiguration(
                name='pokemon',
                sync_mode=shared.ConnectionSyncModeEnum.FULL_REFRESH_OVERWRITE,
            ),
        ],
    ),
    data_residency=shared.GeographyEnum.AUTO,
    destination_id='<destination_id>',
    name='PokeApi-to-BigQuery',
    namespace_definition=shared.NamespaceDefinitionEnum.SOURCE,
    namespace_format='destination',
    non_breaking_schema_updates_behavior=shared.NonBreakingSchemaUpdatesBehaviorEnum.PROPAGATE_FULLY,
    # prefix='ipsam',
    schedule=shared.ConnectionSchedule(
        cron_expression='0 0/30 * * * ?',
        schedule_type=shared.ScheduleTypeEnum.CRON,
    ),
    source_id='<source_id>',
    status=shared.ConnectionStatusEnum.ACTIVE,
)

res = s.connections.create_connection(req)

response:

SDKError: API error occurred: Status 405
{"type":"about:blank","status":405}

Could someone help to see what I've been missing? Thanks, Nico

Are you deploying on Kube or docker-compose? Depending on the answer to that question I’d ask which service (either docker compose or kube pod) is exposed via 8006?

Hello! We are facing the same issue with an Airbyte install on GKE using Helm chart. The API works perfectly fine when called through the UI. But when called by the Terraform provider we very often get a timeout:

2023-10-03T14:40:02.2628737Z Error: failure to invoke API
2023-10-03T14:40:02.2628840Z 
2023-10-03T14:40:02.2629016Z   with module.***.module.airbyte.airbyte_connection.zendesk_support_to_bigquery_connection,
2023-10-03T14:40:02.2629262Z   on ***/airbyte/connections.tf line 1, in resource "airbyte_connection" "zendesk_support_to_bigquery_connection":
2023-10-03T14:40:02.2629505Z    1: resource "airbyte_connection" "zendesk_support_to_bigquery_connection" {
2023-10-03T14:40:02.2629581Z 
2023-10-03T14:40:02.2629712Z error sending request: Get
2023-10-03T14:40:02.2630014Z "<http://airbyte-airbyte-api-server-svc.dataeng.svc.cluster.local/v1/connections/c28ddf55-ae2a-4dd7-b8f2-69442b3e9bd6>":
2023-10-03T14:40:02.2630463Z context deadline exceeded (Client.Timeout exceeded while awaiting headers)

When looking at the API, some requests go through, but at somepoint the server get stuck,

/health

call also get stuck. And after a couple of second, the following error appears in the logs:

2023-10-03 14:41:32 [1;31mERROR[m i.m.r.i.RecoveryInterceptor(resolveFallback):244 - Type [io.airbyte.api.server.forwardingClient.ConfigApiClient$Intercepted] executed with error: Read Timeout
io.micronaut.http.client.exceptions.ReadTimeoutException: Read Timeout
	at io.micronaut.http.client.exceptions.ReadTimeoutException.<clinit>(ReadTimeoutException.java:26) ~[micronaut-http-client-core-3.10.1.jar:3.10.1]
	at io.micronaut.http.client.netty.DefaultHttpClient.lambda$exchangeImpl$33(DefaultHttpClient.java:1097) ~[micronaut-http-client-3.10.1.jar:3.10.1]
	at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:94) ~[reactor-core-3.5.5.jar:3.5.5]
	at io.micronaut.reactive.reactor.instrument.ReactorSubscriber.onError(ReactorSubscriber.java:64) ~[micronaut-runtime-3.10.1.jar:3.10.1]
	at reactor.core.publisher.SerializedSubscriber.onError(SerializedSubscriber.java:124) ~[reactor-core-3.5.5.jar:3.5.5]
	at reactor.core.publisher.FluxTimeout$TimeoutMainSubscriber.handleTimeout(FluxTimeout.java:295) ~[reactor-core-3.5.5.jar:3.5.5]
	at reactor.core.publisher.FluxTimeout$TimeoutMainSubscriber.doTimeout(FluxTimeout.java:280) ~[reactor-core-3.5.5.jar:3.5.5]
	at reactor.core.publisher.FluxTimeout$TimeoutTimeoutSubscriber.onNext(FluxTimeout.java:419) ~[reactor-core-3.5.5.jar:3.5.5]
	at io.micronaut.reactive.reactor.instrument.ReactorSubscriber.onNext(ReactorSubscriber.java:57) ~[micronaut-runtime-3.10.1.jar:3.10.1]
	at reactor.core.publisher.FluxOnErrorReturn$ReturnSubscriber.onNext(FluxOnErrorReturn.java:162) ~[reactor-core-3.5.5.jar:3.5.5]
	at io.micronaut.reactive.reactor.instrument.ReactorSubscriber.onNext(ReactorSubscriber.java:57) ~[micronaut-runtime-3.10.1.jar:3.10.1]
	at reactor.core.publisher.MonoDelay$MonoDelayRunnable.propagateDelay(MonoDelay.java:271) ~[reactor-core-3.5.5.jar:3.5.5]
	at reactor.core.publisher.MonoDelay$MonoDelayRunnable.run(MonoDelay.java:286) ~[reactor-core-3.5.5.jar:3.5.5]
	at io.micronaut.reactive.reactor.instrument.ReactorInstrumentation.lambda$init$0(ReactorInstrumentation.java:62) ~[micronaut-runtime-3.10.1.jar:3.10.1]
	at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:68) ~[reactor-core-3.5.5.jar:3.5.5]
	at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:28) ~[reactor-core-3.5.5.jar:3.5.5]
	at java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
	at java.lang.Thread.run(Thread.java:1589) ~[?:?]
2023-10-03 14:41:32 [32mINFO[m i.a.a.LoggingTrackingClient(track):55 - track. version: null, userId: null, action: Airbyte_API_Call, metadata: {user_id=00000000-0000-0000-0000-000000000000, endpoint=/v1/connections, operation=GET, status_code=500}
2023-10-03 14:41:32 [1;31mERROR[m i.m.h.s.RouteExecutor(logException):444 - Unexpected error occurred: Read Timeout
io.micronaut.http.client.exceptions.ReadTimeoutException: Read Timeout
	at io.micronaut.http.client.exceptions.ReadTimeoutException.<clinit>(ReadTimeoutException.java:26) ~[micronaut-http-client-core-3.10.1.jar:3.10.1]
	at io.micronaut.http.client.netty.DefaultHttpClient.lambda$exchangeImpl$33(DefaultHttpClient.java:1097) ~[micronaut-http-client-3.10.1.jar:3.10.1]
	at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:94) ~[reactor-core-3.5.5.jar:3.5.5]
	at io.micronaut.reactive.reactor.instrument.ReactorSubscriber.onError(ReactorSubscriber.java:64) ~[micronaut-runtime-3.10.1.jar:3.10.1]
	at reactor.core.publisher.SerializedSubscriber.onError(SerializedSubscriber.java:124) ~[reactor-core-3.5.5.jar:3.5.5]
	at reactor.core.publisher.FluxTimeout$TimeoutMainSubscriber.handleTimeout(FluxTimeout.java:295) ~[reactor-core-3.5.5.jar:3.5.5]
	at reactor.core.publisher.FluxTimeout$TimeoutMainSubscriber.doTimeout(FluxTimeout.java:280) ~[reactor-core-3.5.5.jar:3.5.5]
	at reactor.core.publisher.FluxTimeout$TimeoutTimeoutSubscriber.onNext(FluxTimeout.java:419) ~[reactor-core-3.5.5.jar:3.5.5]
	at io.micronaut.reactive.reactor.instrument.ReactorSubscriber.onNext(ReactorSubscriber.java:57) ~[micronaut-runtime-3.10.1.jar:3.10.1]
	at reactor.core.publisher.FluxOnErrorReturn$ReturnSubscriber.onNext(FluxOnErrorReturn.java:162) ~[reactor-core-3.5.5.jar:3.5.5]
	at io.micronaut.reactive.reactor.instrument.ReactorSubscriber.onNext(ReactorSubscriber.java:57) ~[micronaut-runtime-3.10.1.jar:3.10.1]
	at reactor.core.publisher.MonoDelay$MonoDelayRunnable.propagateDelay(MonoDelay.java:271) ~[reactor-core-3.5.5.jar:3.5.5]
	at reactor.core.publisher.MonoDelay$MonoDelayRunnable.run(MonoDelay.java:286) ~[reactor-core-3.5.5.jar:3.5.5]
	at io.micronaut.reactive.reactor.instrument.ReactorInstrumentation.lambda$init$0(ReactorInstrumentation.java:62) ~[micronaut-runtime-3.10.1.jar:3.10.1]
	at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:68) ~[reactor-core-3.5.5.jar:3.5.5]
	at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:28) ~[reactor-core-3.5.5.jar:3.5.5]
	at java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
	at java.lang.Thread.run(Thread.java:1589) ~[?:?]

We are running Airbyte chart

0.49.4

with image versions

0.50.30

and terraform provider

0.3.4

. Which all seems to be latest at this time. This prevent us from using the terraform provider for now. Would you have any idea of this issue’s cause ?

Hi @Jon Tan (Airbyte), I used this doc to deploy airbyte on GCP via terraform. The resource instance adds a startup script to run the

run-ab-platform.sh

script and then use docker compose I'm exposing the 8006 port for the

airbyte-api-server

(or maybe the

airbyte-proxy

🤔)

airbyte-api-server:
    image: airbyte/airbyte-api-server:${VERSION}
    logging: *default-logging
    container_name: airbyte-api-server
    restart: unless-stopped
    ports:
      - "8006"
    environment:
      - AIRBYTE_VERSION=${VERSION}
      - DEPLOYMENT_MODE=${DEPLOYMENT_MODE}
      - INTERNAL_API_HOST=${INTERNAL_API_HOST}
      - AIRBYTE_API_HOST=${AIRBYTE_API_HOST}
    networks:
      - airbyte_internal
    depends_on:
      bootloader:
        condition: service_completed_successfully

As I mention before, I'm able to create resources and destionations via terraform, what I can not do is to create the connections. With the other two options I can interact with the API: list sources, create a destination, etc. But when it comes to create the connections, nothing

I face the same issue. @Jon Tan (Airbyte), to answer your question the 8006 port is exposing the

airbyte/proxy

with airbyte OSS. I missed this message and opened an new thread right after... Sorry about the duplicate.

I'm facing the same issue. I'm hitting the API directly, the

/v1/connections

endpoint and it seems to sporadically return

500

. We are using docker compose and the compose file is pretty much identical to that found here. The logs for the

airbyte-api-server

service show the error:

airbyte-api-server  | 2023-10-06 14:51:58 INFO i.a.a.LoggingTrackingClient(track):55 - track. version: null, userId: null, action: Airbyte_API_Call, metadata: {user_id=00000000-0000-0000-0000-000000000000, endpoint=/v1/connections, operation=GET, status_code=500}
airbyte-api-server  | 2023-10-06 14:51:58 ERROR i.m.h.s.RouteExecutor(logException):444 - Unexpected error occurred: Read Timeout

We are also facing this exact same problem @Jon Tan (Airbyte) and it's something that is preventing us from managing airbyte via github actions. We are only exposing the new API and we can't use port forwarding. Do you know if the team has found the issue/is working on a fix? And when it would be ready?

Hey folks, these timeouts to be caused by the fact that some of our operations do kick off longer running tasks which (due to how the platform is architected) are currently synchronous (things like running a

discoverSchema

job when creating a connection. Our current suggestion for a workaround is to increase the read timeout in micronaut way up for your instance. I believe the default is 10 seconds, but you should try to increase that possibly to a few minutes via

micronaut.http.client.read-timeout

In the long term, we have a ticket on our roadmap for investigating both speeding up those slower synchronous tasks as well as making some of them async with notifications (possibly webhooks) on failure. The longer term stuff would not happen until next year at the earliest. If the increased timeout doesn’t improve things, please let us know.

Thank you

Thanks!

How can we set this when deploying to kubernetes?

In one of the recent Airbyte OSS releases, there should be an environment variable you can set on either your docker container (for docker-compose users) or on the pod (for k8s users) called

READ_TIMEOUT

which should increase the read timeout for the Airbyte API’s micronaut client.

Thank you

Hello! Same here, we updated the timeout to 10m and even with this, the terraform provider times out way before 10m. If this can help, I noticed everytime I restart the API pod, the first try works, but then all fail.

Quick update here, we’re looking into this problem now and have a few leads we’re going to try out. Will report back on this thread as we progress.

Thanks for the update

Update: We think we’ve isolated the problem and have a fix that is currently in testing. Once that’s been validated in our test environment, we’ll release to both Cloud and OSS. I’ll post here to let you know so that you can give it a try and let us know if it resolves the timeout issues.