Hello 🙂 I successfully tested the AsyncRetriever feature with Airbyte Cloud. I exported the YAML file and am now trying to make it work with the Open Source version of Airbyte. I had to slightly modify the code to make it compliant, especially replacing the

download_target_extractor

parameter with

urls_extractor

(otherwise, I wasn't able to test my code in the Connector Builder). When I run the test, the first part works fine—I can see the generated file on the backend—but after the file is created, I get an error. I'm not sure if it occurs during the polling or the data retrieval phase.

Internal Server Error: java.io.IOException: unexpected end of stream on <http://data-airbyte-airbyte-connector-builder-server-svc/>...

My yaml is below in this thread. Does anyone have an idea of what might be causing this issue? thank you for your help

Hi Folks, on creating a connection with Snowflake connector as source where in I provide any random value to Warehouse parameter which is not present in my Snowflake account, the connection is successful in Airbyte. This should not happen technically and should throw an error. Is this a issue with Snowflake connector? Thanks & Regards

I've got a question around the connection builder. Ive got an API endpoint that takes a fromdate and a todate, as a unix timestamp. I want to fetch a year at a time, and continue to loop till today. These two parameters need to be change for each loop. Where would i start with this?

I'm creating a new destination connector. I was wondering if there was a way to get the Source Connector Name or Connection Name. I wanted to default the destination location name to include it

Connection builder - Ive got a response from an API that looks like this - i cant seem to get it to load the records successfully . How does one select the array?

[
[
"2019-09-02T18:00:00.000000Z",
"15100",
"15200",
"15100",
"15199",
"4.11970335"
],
[
"2019-09-02T17:00:00.000000Z",
"14879.75",
"15115",
"14861.99",
"15115",
"10.01840031"
]
]

Hello devs I am working on a destination where I have to add file transfer feature in it How can I add in python cdk?

hi - has anyone been able to pull the product group id column from the pinterest airbyte source?

Hello Team, We are developing custom destination connector using java. We are using latest java CDK for developing connector. We are getting following [main] ERROR io.airbyte.cdk.integrations.base.AirbyteExceptionHandler - Something went wrong in the connector. See the logs for more details. java.lang.IllegalArgumentException: org.apache.commons.cli.MissingOptionException: Missing required option: [--spec outputs the json configuration specification, --check checks the config can be used to connect, --discover outputs a catalog describing the source's catalog, --read reads the source and outputs messages to STDOUT, --write writes messages from STDIN to the integration] at io.airbyte.commons.cli.Clis.parse(Clis.kt:33) at io.airbyte.commons.cli.Clis.parse$default(Clis.kt:19) at io.airbyte.cdk.integrations.base.IntegrationCliParser$Companion.parseCommand(IntegrationCliParser.kt:71) at io.airbyte.cdk.integrations.base.IntegrationCliParser$Companion.access$parseCommand(IntegrationCliParser.kt:25) at io.airbyte.cdk.integrations.base.IntegrationCliParser.parse(IntegrationCliParser.kt:21) at io.airbyte.cdk.integrations.base.IntegrationRunner.run(IntegrationRunner.kt:117) at io.airbyte.cdk.integrations.base.IntegrationRunner.run$default(IntegrationRunner.kt:113) at io.airbyte.cdk.integrations.base.IntegrationRunner.run(IntegrationRunner.kt) at io.airbyte.integrations.destination.sdp.SDPDestination.main(SDPDestination.java:45) Caused by: org.apache.commons.cli.MissingOptionException: Missing required option: [--spec outputs the json configuration specification, --check checks the config can be used to connect, --discover outputs a catalog describing the source's catalog, --read reads the source and outputs messages to STDOUT, --write writes messages from STDIN to the integration] at org.apache.commons.cli.DefaultParser.checkRequiredOptions(DefaultParser.java:281) at org.apache.commons.cli.DefaultParser.parse(DefaultParser.java:723) at org.apache.commons.cli.DefaultParser.parse(DefaultParser.java:686) at org.apache.commons.cli.DefaultParser.parse(DefaultParser.java:667) at io.airbyte.commons.cli.Clis$RelaxedParser.parse(Clis.kt:56) at io.airbyte.commons.cli.Clis.parse(Clis.kt:28) ... 8 more We are using IntegrationRunner. We observed that airbyte platform is sending ‘spec’ as command to connector but IntegrationRunner expecting –spec. When we check CDK code we found optionGroup is built using longOpt. Do connector code expected to append ‘—’ command that it receives from airbyte platform before passing it to IntegrationRunner. Or we are missing something. Thanks in advance.

Hi All.. I am trying to connect to an API that requires an client certificate for the SSL/TLS handshake. Has anyone done this out of box, or will this require the CDK?

Quick question - I've got a custom connector, and updated it to add new streams. It's been published as version 4, but also used in sources already. Can I update my sources to use the new version, or do I need to recreate them and the connections?

Hi All! Could you, please, recommend an individual Airbyte expert for a consulting session to guide me through the set up a basic data pipeline from 2 sources: Amazon Ads API and Amazon Selling Partner API into BigQuery without any transformation?

Hey team, I used to run the

generate.sh

to scaffold a connector, i’ve just seen that the script has been removed from the codebase. What’s the proper way to get going now ?

I'm trying to write a connector to an API with limited filtering capability by last_updated date. The API can support only dates within last 30 days for the last_updated parameter. Is there a way to implement a conditional incremental sync mode so that if the cursor (or the configured connector start date) is older than 30 days we skip the last_updated parameter and just run a full refresh sync relying only on pagination strategy. If the cursor value is within last 30 days we'd use the last_updated parameter and sync only differences since the previous sync. Is this possible to configure it like this in YAML builder?

Hi All, Does current version of Pinecone connector v0.1.43 support OpenAI’s new text-embedding-3-small model (1536 dimensions)? The docs say it currently uses text-embedding-ada-002 (also 1536 dims) via the OpenAI API, so I’m wondering if 3-small isn’t available yet?

Hi all - Does anyone have an existing source connector built for The Trade Desk?

Hi, does anyone has idea about these two types of errors does anyone have the idea about it? i am trying to build the destination-bigquery connector

│  pod "q-destination-spec-c10a8ab7-a413-4b99-a7f2-1a96b9a70f67-0-xxxdw" is waiting to start: image can't be pulled for airbyte/q-destination-spec-c10a8 │

If I still want to use generate.sh now to create custom connector, is it still possible?

Hey all - I'm attempting to set up pagination in my custom connector - however, the API I'm querying requires passing a JSON array from the last record (rather than a single key) for the pagination value. Any idea how I can do this? Or a way to manually view the pagination test requests so I can debug?

Hi All, Can you confirm if my assumption is correct? Connection builder: Only to build source (not destination) Low-code: Only to build source (not destination) Python/Java: For building source & destination but can't be used in the cloud unless you register them with Airbyte?

Hey Team i am trying to create notification for failed sync throu slack webhooks. The TF picks this up but nothing happens. My code looks like that . Am i missing something ? resource "airbyte_workspace" "platform_workspace" { name = "platform_workspace" notifications = { failure = { webhook = { enabled = true url = "https://hooks.slack.com/services/abc123" } } } depends_on = [ google_compute_instance.airbyte_instance ] }

Hi evryone, I'm building my own API source The first page of the API response works fine, but the second one returns 429 Too Many Requests and does not include a Retry-After header. Do you have any advice on how to handle this case properly in a UI? Pleace help 🙏

Just submitted an update to an existing source (mixmax), and I see some failing checks. Are these expected until there is a review? https://github.com/airbytehq/airbyte/pull/59160/

Hi team! I'm a newbie so hopefully this is a quick one: is there a simple repository that shows how to create a simple non-HTTP custom connector? It will use a Python Library to ingest data from a (file/URL) source, so it's a fairly basic use case. All the examples I see are based on HttpStreams.

Is there a way on the cloud version to change a connector version? The current stripe connector released a few days ago to upgrade to the low code CDK and is breaking by not processing invoice lines correctly.

Might be a stupid question, but: is there any way to see the full logs here? (see screenshot) The error message "An unknown error has occurred" isn't particularly helpful here -- this custom connector was working on a previous version of Airbyte, and II was able to use cURL to get the data with the same configuration, so I'd like to know what's going on exactly :/

Custom connector builder stopped working for all our custom connectors right after this morning's maintenance: "Incompatible YAML Your YAML manifest is not compatible with the Connector Builder UI, due to the following error: Unauthorized There is no UI value to switch back to. Please resolve this error with the YAML editor in order to switch to UI mode. " For all of our custom connectors from various sources. Stopped working at 10:57 am EST aligning with the Airbyte maintenance window (and I see Connecter Builder was part of the update). Any guidance? Thanks!

i am having this issue while building the destination-bigquery connector, does anyone has idea about it? my airbyte version is

1.1.0

and bigquer-destination image version is

2.9.0

Failure in destination: You must upgrade your platform version to use this connector version. Either downgrade your connector or upgrade platform to 0.63.7

I am building a connector to cxone.niceincontact.com. This has oauth2 authentication This is what I am providing in consent url. I have already provided client_id and client_secret in Inputs

<https://cxone.niceincontact.com/auth/authorize?{{client_id_key}}={{client_id_value}}&{{redirect_uri_key}}={{{{redirect_uri_value}}%7CurlEncoder}}&{{state_key}}={{state_value}}&scope=openid&response_type=code>

I see this error in the console when I test the connection when airbyte calls

get_oauth_consent_url
{
    "message": "Internal Server Error: Error parsing '{{redirect_uri_value': syntax error at position 129, encountered '}', expected ':'",
    "exceptionClassName": "com.hubspot.jinjava.interpret.FatalTemplateErrorsException",
    "exceptionStack": [],
    "rootCauseExceptionStack": []
}

Is it possible to inject a JSON array into the request body for incremental syncs? The API spec I'm working with accepts filters but as arrays like this:

{
  "filters": [
    {
      "$gt": {
        "audit.modifiedDateTime": "2025-05-05T12:34:56Z"
      }
    }
  ]
}

but when I have a Incremental Sync section of the YAML like

type: DatetimeBasedCursor
cursor_field: audit.modifiedDateTime
cursor_datetime_formats:
  - '%Y-%m-%dT%H:%M:%SZ'
datetime_format: '%Y-%m-%dT%H:%M:%SZ'
start_datetime:
  type: MinMaxDatetime
  datetime: '{{ config["start_date"] }}'
  datetime_format: '%Y-%m-%dT%H:%M:%SZ'
start_time_option:
  type: RequestOption
  inject_into: body_json
  field_path:
    - filters
    - '0'
    - $gt
    - audit.modifiedDateTime

The

filters

part of the request does not become an array but a KV pair like

"filters": {
      "0": {
        "$gt": {
          "audit.modifiedDateTime": "2005-05-01T00:00:00Z"
        }
      }
    }

How can I tell Airbyte that

filters

in the request JSON body is supposed to be an array?

Has anyone found any reliable workaround to connect to databricks as a source with airbyte cloud?