@kapa.ai Hi all, I try to use the S3 source from Airbyte. When I run with a config.json directly with

docker run

I receive a success response using this config json

{
  "provider": {
    "bucket": "************",
    "region": "eu-west-1",
    "access_key_id": "******",
    "secret_access_key": "**********",
    "endpoint": ""
  },
  "path_pattern": "preprocessed/bolcom/product-feed_baby-v2/*.csv",
  "format": {
    "format_type": "csv",
    "delimiter": "|"
  },
  "schema": "s3",
  "dataset": "bolcom"
}

When I configure the same values in the Airbyte UI I however receive a 502 error. Anything I’m doing wrong here?

when is the new update for mongo connector getting released?

Hey I am deploying Airbyte I have setup kubectl and minikube, created airbyte namespace and now i want to create values.yaml how can I do that

@kapa.ai is it possible to subscribe to a connector changelog so we could get proactively the info when a new connector version is available, please ?

@kapa.ai is it possible to use a simple shifting window or last successful pipeline run timestamp as a watermark for incremental syncs, instead of picking a cursor field from the API response? My response does have a cursor field, but it's nested in a list of results and it's not possible to set a dynamic cursor field in Airbyte

This message was deleted.

@kapa.ai Hi! I need to change data type from integer to float in stream Google Analytics 4 source. Underlying data are of type float. connector will fetch them from source, but it will only move integer values and decimal ones will have filled NULL values. Tell me how to change data types in custom report in GA4.

Hey @kapa.ai, I am trying to do some testing with the MySQL source. Airbyte creates the connection just fine but it does not see any streams at all. I’ve never seen it not able to see anything. Any tips?

is there a way to arrange columns sequence in google sheets? right now it is in alphabetical order when loading data from source to Google Sheet destinatio

Hey @kapa.ai, I'm updating the Track PMS connector and noticed that for some customer domains, the reservations and reservations_v2 streams are missing nearly half the available information (~3900 returned items vs ~7500 total items), whereas in other customer domains, those endpoints pull nearly all the available data. In all cases, the connector reports Healthy and there are no errors at all in the logs. I'm using the latest versions of abctl 0.28.0 and Airbyte helm chart 1.7.1, and had same issue when using abctl 0.25.0 and Airbyte helm chart 1.6.3. Some strange things I notice: • Separately, when I write python (via requests or asyncio) to check the API endpoint independently, I'm able to obtain all the results, indicating the issue does not appear to be the API. • When I query the API separately, the query results fields

total_items

and

page_count

are returned by every API query are consistent and correct. • Looking at the results of the endpoint

id

value that should largely be contiguous, I notice a large gap after some initial results. Could you please let me know: 1. What mechanism does Airbyte use use to determine a stream has synced successfully? Could you point me to any Airbyte source code that is used to determine a sync is complete? The connector builder indicates the python requests library is used (the Request indicates a query header of "User-Agent": "python-requests/2.32.3") when I'm testing endpoints. 2. How can I increase the amount of info in Airbyte logs so I can see what query URLs were used in the api? I can't find any URLs or Track PMS API response information in the logs, so it's hard to know exactly what was sent to the Track PMS API.

HI @kapa.ai ### Airbyte Sync Failing (v1.7) Due to Empty CSV - Schema Inference Issue Our Airbyte sync job has recently started failing, displaying the following error message in the logs:

Could not infer schema as there are no rows in year=2025/month=01/data.csv. If having an empty CSV file is expected, ignore this. Else, please contact Airbyte.

Problem Description: The sync job fails because it encounters a CSV file with no rows (e.g.,

year=2025/month=01/data.csv

), which prevents schema inference. Key Context / What's Changed: • Version Upgrade: This issue began occurring after upgrading Airbyte to version 1.7. Our previous Airbyte version handled empty CSV files in S3 without causing the sync to fail. • Expected Behavior: We do expect some CSV files for certain periods to be empty (i.e., contain headers but no data rows) as there might be no data for those specific periods. In previous Airbyte versions, this did not lead to a job failure. • File Verification: We've confirmed the specific file (

year=2025/month=01/data.csv

) is indeed empty (0KB). • Log Snippets: While the UI shows "Failed," the check phase logs still indicate "Check succeeded" and "Connector exited with exit code 0" for the initial check. The specific "Could not infer schema" error arises when the system attempts to process these empty CSVs during the actual sync. Our Goal: We need to ensure that empty CSV files (which are an expected occurrence in our data source) do not cause the entire sync job to fail in Airbyte 1.7. We require guidance on how to configure Airbyte to handle such files gracefully, ideally by skipping them or creating an empty table/stream, without stopping the sync. Could you please provide guidance on: • What changes in Airbyte version 1.7 might have altered the handling of empty CSV files during schema inference or data processing? • Are there any new or existing configuration options within the S3 source connector (or global settings) that can allow the sync to successfully proceed when encountering empty CSVs? • What is the recommended approach for handling expected empty files in S3 sources with Airbyte 1.7 to prevent job failures? We appreciate your prompt assistance in resolving this regression.

@kapa.ai SKAN fields are available in airbyte appsflyer integration

Hello @kapa.ai, my open source Airbyte installation V1.7.1 don't let me edit the marketplace connector in the UI Connector Builder, gives the error "Forbiden" when I try to test the connector in the Builder or when I try to switch from YAML to UI. It happens with any connector from the marketplace that I try to edit. Do you know how to fix this problem? Thanks in advance for your help.

Build AI using MCP server by airbyte

for docker compose setups (legacy) which containers do I need to restart to apply changes on the config

SYNC_JOB_MAX_TIMEOUT_DAYS

in the .env file?

@kapa.ai How can i change User-Agent when using OAuth authentication. I am building a custom connector

@kapa.ai has anyone used acuamatica connector APIs

@kapa.ai suddenly my Recharge connector stopped working. The error say : "The stream 'products' in your connection configuration was not found in the source. Refresh the schema in your replication settings and remove this stream from future sync attempts.", But i do not have any stream name called 'products'. Any idea what is the issue ?

@kapa.ai I am using oss version of 1.7.1 & clickhouse v2 destination connector is avaialle in this but iys not doing deduping

I'm using Airbyte's google_search_console connector (v1.3.0) in version 1.9.1 and I have several such connections. I've noticed that some connections aren't working properly: there's data in the airbyte_internal schema but no data in the destination table in the postgresql database (destination connector version v2.0.12). The synchronization logs show no errors. For information, the airbyte_internal schema table is 30GB and the destination table is 26GB. Has anyone encountered this problem? How can I fix it? @kapa.ai?

@kapa.ai when i am trying to connect to iceberg destination i am getting below error

Configuration check failed
Could not connect to the Iceberg catalog with the provided configuration. User: arn:aws:iam::413928405733:user/POS-170-spx is not authorized to perform: glue:GetTable on resource: arn:aws:glue:us-east-1:413928405733:table/testing_airbyte_connection/temp_1751633231304 because no identity-based policy allows the glue:GetTable action (Service: Glue, Status Code: 400, Request ID: fa81dd6f-6c39-43be-beb4-70750c71b5aa), root cause: AccessDeniedException(User: arn:aws:iam::413928405733:user/POS-170-spx is not authorized to perform: glue:GetTable on resource: arn:aws:glue:us-east-1:413928405733:table/testing_airbyte_connection/temp_1751633231304 because no identity-based policy allows the glue:GetTable action (Service: Glue, Status Code: 400, Request ID: fa81dd6f-6c39-43be-beb4-70750c71b5aa))

@kapa.ai when i am trying to connect to iceberg destination i am getting below error

Configuration check failed
Could not connect to the Iceberg catalog with the provided configuration. User: arn:aws:iam::413928405733:user/POS-170-spx is not authorized to perform: glue:GetTable on resource: arn:aws:glue:us-east-1:413928405733:table/testing_airbyte_connection/temp_1751633231304 because no identity-based policy allows the glue:GetTable action (Service: Glue, Status Code: 400, Request ID: fa81dd6f-6c39-43be-beb4-70750c71b5aa), root cause: AccessDeniedException(User: arn:aws:iam::413928405733:user/POS-170-spx is not authorized to perform: glue:GetTable on resource: arn:aws:glue:us-east-1:413928405733:table/testing_airbyte_connection/temp_1751633231304 because no identity-based policy allows the glue:GetTable action (Service: Glue, Status Code: 400, Request ID: fa81dd6f-6c39-43be-beb4-70750c71b5aa))

do i need to create any table in glue manually ?

How to update sync job retry limit Retry State: RetryManager(completeFailureBackoffPolicy=BackoffPolicy(minInterval=PT10S, maxInterval=PT30M, base=3), partialFailureBackoffPolicy=null, successiveCompleteFailureLimit=5, totalCompleteFailureLimit=10, successivePartialFailureLimit=1000, totalPartialFailureLimit=20, successiveCompleteFailures=1, totalCompleteFailures=1, successivePartialFailures=0, totalPartialFailures=0)

@kapa.ai, Everything was working fine and now I get this error when syncing:

Warning from destination: com.google.cloud.bigquery.BigQueryException: Incompatible table partitioning specification. Expected partitioning specification interval(type:day,field:_airbyte_extracted_at) clustering(id,_airbyte_extracted_at), but input partitioning specification is interval(type:day,field:_airbyte_extracted_at) clustering(_airbyte_extracted_at)

Why ? What can I do ?

@kapa.ai Unable to cancel airbyte running jobs This is the response while trying to get the token

>> response.text

'<!DOCTYPE html>\n<html>\n<head>\n<title>Error</title>\n<style>\nhtml { color-scheme: light dark; }\nbody { width: 35em; margin: 0 auto;\nfont-family: Tahoma, Verdana, Arial, sans-serif; }\n</style>\n</head>\n<body>\n<h1>An error occurred.</h1>\n<p>Sorry, the page you are looking for is currently unavailable.<br/>\nPlease try again later.</p>\n<p>If you are the system administrator of this resource then you should check\nthe error log for details.</p>\n<p><em>Faithfully yours, nginx.</em></p>\n</body>\n</html>\n'

@kapa.ai sabes si el problema de sincronización incremental de los conectores hechos con Connector Builder que no tienen filtrado por fecha, está en proceso de solución?

im trying to query API to create a source from a custom connector i built how can i find definitionId ? i tried using the id in the url on the builder, but i got "The requested resource could not be found."

@kapa.ai How do you get Shopify Store name? The example says either 'my-store' or 'my-store.myshopify.com' but such thing doesn't exist

@kapa.ai Unable to cancel airbyte jobs.

>> res = requests.delete(cancel_job_url, headers=header)

>> print(res.json())

{'status': 409, 'type': 'https://reference.airbyte.com/reference/errors#409-state-conflict', 'title': 'state-conflict', 'detail': 'State conflict', 'documentationUrl': None, 'data': {'message': 'Job is not currently running'}}

@kapa.ai I m using airbyte Version 1.4.1 , does this version, Support Webhooks