https://linen.dev logo
Join Slack
Powered by
<@U04S8PLJWMT>, i am trying to setup a connection,...
# replication-ask-ai
s
@kapa.ai, i am trying to setup a connection, but getting error when setting up the connection
Copy code
Internal message: Unexpected error performing DISCOVER. The exit of the connector was: 0
Failure origin: source
k
This error — 
Unexpected error performing DISCOVER. The exit of the connector was: 0
— means the connector process itself exited successfully (exit code 0), but the platform encountered an issue processing or storing the discovery result. Based on the knowledge sources, here are the known causes and solutions: Known Causes 1. Response Payload Too Large (413 Request Entity Too Large) If your source has a very large number of tables/streams (e.g., 10,000+ tables in MSSQL), the discovered catalog can exceed the platform's request size limit, causing this error even though the connector exits with code 0. [source-mssql 413 issue] To address this, you can try increasing the platform's request size limits. For example, in a Kubernetes/Helm deployment:
Copy code
global:
  env_vars:
    MICRONAUT_SERVER_MAX_REQUEST_SIZE: "104857600" # 100MB

webapp:
  ingress:
    annotations:
      nginx.ingress.kubernetes.io/proxy-body-size: "50m"

worker:
  env:
    - name: DISCOVER_JOB_TIMEOUT_MINUTES
      value: "60"
2. Invalid or Null Configuration Field If the source was created via API with an incorrect configuration (e.g., a field is 
null
or the wrong type), the connector may exit 0 during check but fail during discovery when the config is validated more strictly. [source-google-ads issue] Double-check that all required fields in your source configuration match the expected types (e.g., a field expecting a 
string
is not 
null
). 3. Connector Version Regression Some connector versions have introduced bugs in schema discovery. Downgrading to a previously working connector version has resolved this in some cases. [GA4 discovery issue] General Troubleshooting Steps 1. Check the full logs in the Airbyte UI (Job logs) for more specific error details — the root cause is often buried deeper in the stack trace. 2. Verify your source configuration — ensure all fields are correctly typed and no required fields are missing or null. 3. Check the number of streams — if your source has a very large number of tables, consider filtering to a specific schema or subset. 4. Try a different connector version — if the issue started after a connector upgrade, consider rolling back. Could you share which connector (source type) you are using? That would help narrow down the specific cause. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
s
@kapa.ai, here are the traces logs can you please check and see what might be the issue
Copy code
io.airbyte.workers.exception.WorkerException: Unexpected error performing DISCOVER. The exit of the connector was: 0
	at io.airbyte.connectorSidecar.ConnectorMessageProcessor.run(ConnectorMessageProcessor.kt:95)
	at io.airbyte.connectorSidecar.ConnectorWatcher.processConnectorOutput(ConnectorWatcher.kt:138)
	at io.airbyte.connectorSidecar.ConnectorWatcher.run(ConnectorWatcher.kt:79)
	at io.airbyte.connectorSidecar.$ConnectorWatcher$Definition.initialize(Unknown Source)
	at io.airbyte.connectorSidecar.$ConnectorWatcher$Definition.instantiate(Unknown Source)
	at io.micronaut.context.DefaultBeanContext.resolveByBeanFactory(DefaultBeanContext.java:2334)
	at io.micronaut.context.DefaultBeanContext.doCreateBean(DefaultBeanContext.java:2304)
	at io.micronaut.context.DefaultBeanContext.doCreateBean(DefaultBeanContext.java:2316)
	at io.micronaut.context.DefaultBeanContext.createRegistration(DefaultBeanContext.java:3127)
	at io.micronaut.context.SingletonScope.getOrCreate(SingletonScope.java:80)
	at io.micronaut.context.DefaultBeanContext.findOrCreateSingletonBeanRegistration(DefaultBeanContext.java:3029)
	at io.micronaut.context.DefaultBeanContext.initializeEagerBean(DefaultBeanContext.java:2702)
	at io.micronaut.context.DefaultBeanContext.initializeContext(DefaultBeanContext.java:1994)
	at io.micronaut.context.DefaultApplicationContext.initializeContext(DefaultApplicationContext.java:314)
	at io.micronaut.context.DefaultBeanContext.configureAndStartContext(DefaultBeanContext.java:3318)
	at io.micronaut.context.DefaultBeanContext.start(DefaultBeanContext.java:345)
	at io.micronaut.context.DefaultApplicationContext.start(DefaultApplicationContext.java:216)
	at io.micronaut.runtime.Micronaut.start(Micronaut.java:75)
	at io.airbyte.connectorSidecar.ApplicationKt.main(Application.kt:18)
	at io.airbyte.connectorSidecar.ApplicationKt.main(Application.kt)
Caused by: org.openapitools.client.infrastructure.ClientException: Client error : 409 Conflict
k
The stack trace reveals the root cause at the bottom:
Copy code
Caused by: org.openapitools.client.infrastructure.ClientException: Client error : 409 Conflict
This means the connector itself exited successfully (exit code 0), but when the platform's connector sidecar tried to write/store the discovery result back to the API, it received a 409 Conflict response. Based on the knowledge sources, a 409 Conflict during discovery is a known issue. One documented case involves the source-github connector, where a 409 Conflict occurs when dealing with empty repositories — the connector attempts to interact with a resource that is in a conflicting state. [source-github 409 issue] However, the knowledge sources do not contain specific documentation about a 409 Conflict occurring at the platform/sidecar level when writing the discover catalog result (i.e., at 
writeDiscoverCatalogResult
), which is what your trace suggests. What to check 1. Duplicate discovery job: A 409 Conflict at the API level can sometimes mean a discovery job for the same source is already in progress or a result already exists in a conflicting state. Try cancelling any pending jobs and retrying. 2. Source/connection state conflict: Check if there are any duplicate or conflicting source definitions in your Airbyte instance. 3. Platform version issues: This pattern (sidecar failing to write results back with a 4xx error) has been seen in platform-level bugs. [source-google-ads issue] Recommended next steps • Share which connector (source type) and Airbyte version you are using — this would help narrow down the cause significantly. • Check if there are any other jobs running for the same source simultaneously. • Look at the Airbyte server logs for more context around what triggered the 409 on the API side. The knowledge sources do not contain enough information to definitively identify the exact cause of a 409 at the catalog write step in your specific case, so I'd recommend reaching out on the Airbyte Community Slack with your connector type and platform version for more targeted help. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
s
@kapa.ai, i am setting up a connection between postgres and s3 my source is postgres which as version 3.7.2 and my destination is s3 which has version 1.8.6 and my airbyte version is 1.6.0
k
Thanks for the additional details. You're running: • Source: Postgres 3.7.2 • Destination: S3 1.8.6 • Airbyte: 1.6.0 Based on the knowledge sources, there are two relevant issues that could explain the 409 Conflict during schema discovery with your Postgres source: 1. Postgres Discovery Ignoring Schema Filters (source-postgres 3.7.0+) There is a known issue with 
source-postgres
where the discover operation fetches all tables, schemas, and primary keys without respecting the schema filter defined in the source configuration. If your Postgres database has a large number of schemas, this can cause the discovery process to be extremely resource-intensive and potentially fail. [source-postgres schema filter issue] The problematic query fetches primary keys across all schemas (passing 
null
as the schema parameter), regardless of what you configured in the source. Workaround: Try limiting your source to the 
public
schema only, or the specific schema(s) you need. Users have reported that restricting to a single schema resolves discovery failures. [Postgres discover failure] 2. Schema Discovery Timeout If the discovery is taking too long due to a large database, you can increase the timeout limits in your 
values.yaml
for Airbyte 1.6.0:
Copy code
server:
  extraEnvs:
    - name: HTTP_IDLE_TIMEOUT
      value: 20m
    - name: READ_TIMEOUT
      value: 30m
[Scaling Airbyte 1.6] Recommended Steps 1. Restrict the schema filter in your Postgres source configuration to only the schema(s) you need (e.g., 
public
). 2. Increase timeouts if your database is large. 3. Check if there are any concurrent discovery jobs running for the same source, as a 409 Conflict can indicate a resource conflict on the API side. Could you share how many schemas/tables your Postgres database has? That would help confirm whether the schema filter issue is the likely culprit. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
s
@kapa.ai, how can i restrict schema to public when setting up connection
k
When setting up your Postgres source in Airbyte, you can restrict the schema to 
public
(or any specific schema) in the Schemas field during source configuration. Here's how: In the Airbyte UI 1. Go to Sources and open your Postgres source configuration. 2. Find the Default Schemas (or Schemas) field. 3. Enter 
public
(or whichever schema you want to sync from). > Note: Schema names are case sensitive. The 
public
schema is set by default. Multiple schemas may be entered at once. [Postgres destination docs] Using Octavia CLI (if applicable) If you're configuring via Octavia CLI, in your 
sources/postgres/configuration.yaml
, set the 
schemas
field like this:
Copy code
configuration:
  schemas: ["public"]
[Octavia CLI tutorial] Why this matters for your case With 
source-postgres
version 3.7.x, there is a known issue where the discover operation fetches metadata across all schemas without respecting the schema filter, which can cause heavy load and discovery failures on databases with many schemas. Restricting to 
public
(or your specific schema) has been confirmed by users to resolve discovery failures. [source-postgres schema filter issue; Postgres discover failure] After updating the schema filter, save the source and retry the connection setup. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
s
@kapa.ai, can you tell me approptae source version to use also i am using airbyte self hosted
k
Based on the knowledge sources, here is what I can confirm about the appropriate Postgres source version for your setup: Recommended Source-Postgres Version The knowledge sources mention the following relevant versions: • 3.7.2 — This is your current version, released via [release 3.7.2] and set as the default for both Cloud and OSS users. • 3.8.0-rc.1 / 3.8.0-rc.3 — These are release candidates introducing a rewrite of source-postgres. [release 3.8.0-rc.1; release 3.8.0-rc.3] However, as release candidates, they may not be stable for production use. Important Bug Fixes to Be Aware Of • There is a known bug in certain 
pgjdbc
driver versions (low 42.7.x) that can advance the replication slot unexpectedly if you are using CDC. Airbyte addressed this by pinning the driver to version 42.6.1 in source-postgres. [pgjdbc version fix] • The schema filter issue (discovery not respecting schema filters) is present in 3.7.0+ and is still open. [schema filter issue] Recommendation The knowledge sources do not explicitly recommend a specific version beyond 3.7.2 as the current stable default for self-hosted OSS users. Since you are already on 3.7.2, the best immediate workaround for your discovery issue remains restricting the schema filter to 
public
 (or your specific schema) in the source configuration. For the most up-to-date version recommendation, I'd suggest checking the Airbyte Postgres source changelog directly, as the knowledge sources do not contain the full changelog to confirm a newer stable release beyond 3.7.2. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
Open in Slack
PreviousNext
Powered by