Hi, does anyone know if I can use the low code cdk to grab arbitrary header values? I know there's a

WaitTimeFromHeader

backoff strategy, so it's capable of using headers for backoffs. What I'm trying to do is improve the Pardot connector to handle the 100k record limit when paginating. Essentially, after paginating through 100k records, even if there are more results the last response won't contain a

nextPageUrl

/

nextPageToken

, instead the response will contain a

Pardot-Warning

header with a value

101;Record count for nextPageToken sequence has been exceeded. No page token returned.

(see here) and the proper way to handle this is to construct a new request from the last seen record in order to carry on paginating.

Hi! I'm building custom connectors with the low-code CDK, but I can't use the connector builder because I need custom declarative components. I notice the CLI-based tutorial for the low-code CDK is gone from the official docs (replaced by a link to the no code connector builder tutorial). Does this mean building custom declarative components for the low-code CDK and writing editing the YAML manifest ourselves is now deprecated ?

Hey folks! 👋 I'm building a source connector for Hotmart, which uses an OAuth2 client credentials flow with a twist: the token request requires a custom

Authorization

header in the format

Authorization: Basic <base64(email:password)>

, in addition to the usual

client_id

and

client_secret

. While experimenting with the Connector Builder, I noticed: • The OAuth2 (legacy) option allows me to use the

client_credentials

flow, but doesn’t support injecting a custom access token header. • The OAuth2 (modern) option supports

access_token_header

, but forces an

authorization_code

flow and requires a consent URL — which Hotmart doesn't use or support. So far, I haven’t been able to combine these two behaviors. This particular auth flow seems unsupported at the moment through the UI. I also explored the Python CDK route (I have experience with Python), but the documentation appears outdated or incomplete. It’s not clear to me what the current best practices are or how to properly bootstrap a new connector using the latest tools. The custom Python connector tutorial seems deprecated or at least not aligned with the current repo structure. Any guidance on how to best approach this kind of authentication? Either using the Builder or Python CDK would be great. Open to any suggestions.

Hello 🙂 I successfully tested the AsyncRetriever feature with Airbyte Cloud. I exported the YAML file and am now trying to make it work with the Open Source version of Airbyte. I had to slightly modify the code to make it compliant, especially replacing the

download_target_extractor

parameter with

urls_extractor

(otherwise, I wasn't able to test my code in the Connector Builder). When I run the test, the first part works fine—I can see the generated file on the backend—but after the file is created, I get an error. I'm not sure if it occurs during the polling or the data retrieval phase.

Internal Server Error: java.io.IOException: unexpected end of stream on <http://data-airbyte-airbyte-connector-builder-server-svc/>...

My yaml is below in this thread. Does anyone have an idea of what might be causing this issue? thank you for your help

Hi Folks, on creating a connection with Snowflake connector as source where in I provide any random value to Warehouse parameter which is not present in my Snowflake account, the connection is successful in Airbyte. This should not happen technically and should throw an error. Is this a issue with Snowflake connector? Thanks & Regards

I've got a question around the connection builder. Ive got an API endpoint that takes a fromdate and a todate, as a unix timestamp. I want to fetch a year at a time, and continue to loop till today. These two parameters need to be change for each loop. Where would i start with this?

I'm creating a new destination connector. I was wondering if there was a way to get the Source Connector Name or Connection Name. I wanted to default the destination location name to include it

Connection builder - Ive got a response from an API that looks like this - i cant seem to get it to load the records successfully . How does one select the array?

[
[
"2019-09-02T18:00:00.000000Z",
"15100",
"15200",
"15100",
"15199",
"4.11970335"
],
[
"2019-09-02T17:00:00.000000Z",
"14879.75",
"15115",
"14861.99",
"15115",
"10.01840031"
]
]

Hello devs I am working on a destination where I have to add file transfer feature in it How can I add in python cdk?

hi - has anyone been able to pull the product group id column from the pinterest airbyte source?

Hello Team, We are developing custom destination connector using java. We are using latest java CDK for developing connector. We are getting following [main] ERROR io.airbyte.cdk.integrations.base.AirbyteExceptionHandler - Something went wrong in the connector. See the logs for more details. java.lang.IllegalArgumentException: org.apache.commons.cli.MissingOptionException: Missing required option: [--spec outputs the json configuration specification, --check checks the config can be used to connect, --discover outputs a catalog describing the source's catalog, --read reads the source and outputs messages to STDOUT, --write writes messages from STDIN to the integration] at io.airbyte.commons.cli.Clis.parse(Clis.kt:33) at io.airbyte.commons.cli.Clis.parse$default(Clis.kt:19) at io.airbyte.cdk.integrations.base.IntegrationCliParser$Companion.parseCommand(IntegrationCliParser.kt:71) at io.airbyte.cdk.integrations.base.IntegrationCliParser$Companion.access$parseCommand(IntegrationCliParser.kt:25) at io.airbyte.cdk.integrations.base.IntegrationCliParser.parse(IntegrationCliParser.kt:21) at io.airbyte.cdk.integrations.base.IntegrationRunner.run(IntegrationRunner.kt:117) at io.airbyte.cdk.integrations.base.IntegrationRunner.run$default(IntegrationRunner.kt:113) at io.airbyte.cdk.integrations.base.IntegrationRunner.run(IntegrationRunner.kt) at io.airbyte.integrations.destination.sdp.SDPDestination.main(SDPDestination.java:45) Caused by: org.apache.commons.cli.MissingOptionException: Missing required option: [--spec outputs the json configuration specification, --check checks the config can be used to connect, --discover outputs a catalog describing the source's catalog, --read reads the source and outputs messages to STDOUT, --write writes messages from STDIN to the integration] at org.apache.commons.cli.DefaultParser.checkRequiredOptions(DefaultParser.java:281) at org.apache.commons.cli.DefaultParser.parse(DefaultParser.java:723) at org.apache.commons.cli.DefaultParser.parse(DefaultParser.java:686) at org.apache.commons.cli.DefaultParser.parse(DefaultParser.java:667) at io.airbyte.commons.cli.Clis$RelaxedParser.parse(Clis.kt:56) at io.airbyte.commons.cli.Clis.parse(Clis.kt:28) ... 8 more We are using IntegrationRunner. We observed that airbyte platform is sending ‘spec’ as command to connector but IntegrationRunner expecting –spec. When we check CDK code we found optionGroup is built using longOpt. Do connector code expected to append ‘—’ command that it receives from airbyte platform before passing it to IntegrationRunner. Or we are missing something. Thanks in advance.

Hi All.. I am trying to connect to an API that requires an client certificate for the SSL/TLS handshake. Has anyone done this out of box, or will this require the CDK?

Quick question - I've got a custom connector, and updated it to add new streams. It's been published as version 4, but also used in sources already. Can I update my sources to use the new version, or do I need to recreate them and the connections?

Hi All! Could you, please, recommend an individual Airbyte expert for a consulting session to guide me through the set up a basic data pipeline from 2 sources: Amazon Ads API and Amazon Selling Partner API into BigQuery without any transformation?

Hey team, I used to run the

generate.sh

to scaffold a connector, i’ve just seen that the script has been removed from the codebase. What’s the proper way to get going now ?

I'm trying to write a connector to an API with limited filtering capability by last_updated date. The API can support only dates within last 30 days for the last_updated parameter. Is there a way to implement a conditional incremental sync mode so that if the cursor (or the configured connector start date) is older than 30 days we skip the last_updated parameter and just run a full refresh sync relying only on pagination strategy. If the cursor value is within last 30 days we'd use the last_updated parameter and sync only differences since the previous sync. Is this possible to configure it like this in YAML builder?

Hi All, Does current version of Pinecone connector v0.1.43 support OpenAI’s new text-embedding-3-small model (1536 dimensions)? The docs say it currently uses text-embedding-ada-002 (also 1536 dims) via the OpenAI API, so I’m wondering if 3-small isn’t available yet?

Hi all - Does anyone have an existing source connector built for The Trade Desk?

Hi, does anyone has idea about these two types of errors does anyone have the idea about it? i am trying to build the destination-bigquery connector

│  pod "q-destination-spec-c10a8ab7-a413-4b99-a7f2-1a96b9a70f67-0-xxxdw" is waiting to start: image can't be pulled for airbyte/q-destination-spec-c10a8 │

If I still want to use generate.sh now to create custom connector, is it still possible?

Hey all - I'm attempting to set up pagination in my custom connector - however, the API I'm querying requires passing a JSON array from the last record (rather than a single key) for the pagination value. Any idea how I can do this? Or a way to manually view the pagination test requests so I can debug?

Hi All, Can you confirm if my assumption is correct? Connection builder: Only to build source (not destination) Low-code: Only to build source (not destination) Python/Java: For building source & destination but can't be used in the cloud unless you register them with Airbyte?

Hey Team i am trying to create notification for failed sync throu slack webhooks. The TF picks this up but nothing happens. My code looks like that . Am i missing something ? resource "airbyte_workspace" "platform_workspace" { name = "platform_workspace" notifications = { failure = { webhook = { enabled = true url = "https://hooks.slack.com/services/abc123" } } } depends_on = [ google_compute_instance.airbyte_instance ] }

Hi evryone, I'm building my own API source The first page of the API response works fine, but the second one returns 429 Too Many Requests and does not include a Retry-After header. Do you have any advice on how to handle this case properly in a UI? Pleace help 🙏

Just submitted an update to an existing source (mixmax), and I see some failing checks. Are these expected until there is a review? https://github.com/airbytehq/airbyte/pull/59160/

Hi team! I'm a newbie so hopefully this is a quick one: is there a simple repository that shows how to create a simple non-HTTP custom connector? It will use a Python Library to ingest data from a (file/URL) source, so it's a fairly basic use case. All the examples I see are based on HttpStreams.

Is there a way on the cloud version to change a connector version? The current stripe connector released a few days ago to upgrade to the low code CDK and is breaking by not processing invoice lines correctly.

Might be a stupid question, but: is there any way to see the full logs here? (see screenshot) The error message "An unknown error has occurred" isn't particularly helpful here -- this custom connector was working on a previous version of Airbyte, and II was able to use cURL to get the data with the same configuration, so I'd like to know what's going on exactly :/

Custom connector builder stopped working for all our custom connectors right after this morning's maintenance: "Incompatible YAML Your YAML manifest is not compatible with the Connector Builder UI, due to the following error: Unauthorized There is no UI value to switch back to. Please resolve this error with the YAML editor in order to switch to UI mode. " For all of our custom connectors from various sources. Stopped working at 10:57 am EST aligning with the Airbyte maintenance window (and I see Connecter Builder was part of the update). Any guidance? Thanks!

i am having this issue while building the destination-bigquery connector, does anyone has idea about it? my airbyte version is

1.1.0

and bigquer-destination image version is

2.9.0

Failure in destination: You must upgrade your platform version to use this connector version. Either downgrade your connector or upgrade platform to 0.63.7