having a problem with bigquery connection. started after version 1.0.0 is gone live in 2026-10-02

Thinking completed The BigQuery source connector version

1.0.0

rebuilt the connector on Airbyte's Bulk CDK and introduced several breaking changes [BigQuery Migration Guide]. To fix the connection issues, follow these migration steps: 1. Refresh the Source Schema (Required) Every connection using the BigQuery source must have its schema refreshed before syncing on version

1.0.0

. Until refreshed, streams with modified column types will fail with an error requesting a schema refresh [What to do]. • Go to your connection settings in the Airbyte UI. • Click Refresh source schema and save the connection without clearing data [What to do]. 2. Provide a Service Account Key In version 1.0.0, the Service Account Key JSON field is mandatory. Earlier versions fell back to the environment credentials if this field was empty. If your source was configured without one, you must provide a valid JSON key in the connector settings [Upgrading to 1.0.0; What to do]. 3. Check Incremental Cursor Columns Version 1.0.0 restricts incremental cursors to specific supported types:

INT64

,

NUMERIC

,

BIGNUMERIC

,

FLOAT64

,

STRING

,

DATE

,

DATETIME

,

TIME

, and

TIMESTAMP

[Upgrading to 1.0.0]. • If your stream uses

BOOL

,

BYTES

,

JSON

,

STRUCT

, or

ARRAY

as a cursor column, choose a new cursor column [What to do]. • Note: Changing the cursor column causes the stream to be read again in full on the next sync [What to do]. 4. Handling Failed Syncs After Schema Refresh • If your sync fails because destination column types changed (e.g., date, datetime, time, timestamp, JSON, or base64 bytes), refresh the failing streams using Refresh and retain records [What to do]. • Avoid using Refresh and remove records or Clear on incremental streams if BigQuery no longer holds older deleted or expired records, as that would permanently delete them from your destination [Upgrading to 1.0.0]. 5. Check Connection Test Failures If the connection test itself is failing with

Discovered zero tables

, ensure the configured dataset (or the whole project if Dataset ID is left blank) contains at least one table [Upgrading to 1.0.0]. Additionally, BigQuery ML models are no longer recognized as discoverable tables/streams [Upgrading to 1.0.0]. (Note: Rolling back to connector version

0.4.5

will not resolve issues if syncs have already run on 1.0.0, as version 0.4.5 cannot read state written by 1.0.0 and will restart incremental streams from scratch [Rolling back].) 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai when trying for

Discovering schema failed

having following error:

Internal message: Unexpected error performing DISCOVER. The exit of the connector was: 0
Failure origin: source

io.airbyte.workers.exception.WorkerException: Unexpected error performing DISCOVER. The exit of the connector was: 0
	at io.airbyte.connectorSidecar.ConnectorMessageProcessor.run(ConnectorMessageProcessor.kt:114)
	at io.airbyte.connectorSidecar.ConnectorWatcher.processConnectorOutput(ConnectorWatcher.kt:154)
	at io.airbyte.connectorSidecar.ConnectorWatcher.run(ConnectorWatcher.kt:79)
	at io.airbyte.connectorSidecar.ApplicationKt.main(Application.kt:22)
	at io.airbyte.connectorSidecar.ApplicationKt.main(Application.kt)
Caused by: org.openapitools.client.infrastructure.ClientException: Client error : 413 Request Entity Too Large
	at io.airbyte.api.client.generated.SourceApi.writeDiscoverCatalogResult(SourceApi.kt:1090)
	at io.airbyte.connectorSidecar.ConnectorMessageProcessor.setOutput(ConnectorMessageProcessor.kt:202)
	at io.airbyte.connectorSidecar.ConnectorMessageProcessor.run(ConnectorMessageProcessor.kt:108)
	... 4 more

Thinking completed This error occurs when the discovered schema/catalog is too large for Airbyte's internal API to accept in a single HTTP request [source-mssql 413 Request Entity Too Large]:

Caused by: org.openapitools.client.infrastructure.ClientException: Client error : 413 Request Entity Too Large     at io.airbyte.api.client.generated.SourceApi.writeDiscoverCatalogResult(SourceApi.kt:1090)

In version 1.0.0, the BigQuery source discovers nested schemas for

STRUCT

and

ARRAY

columns and source-defined primary keys, which significantly increases the catalog size compared to earlier versions [Upgrading to 1.0.0]. If your BigQuery project or dataset contains a very large number of tables or columns, the resulting catalog payload exceeds the HTTP request size limit handled by the internal server/proxy [source-mssql 413 Request Entity Too Large]. Potential Solutions 1. Limit the Scope of the Source Configuration ◦ If the Dataset ID field is left blank in your BigQuery source settings, the connector attempts to discover all tables across the entire GCP project [Contributing to source-bigquery; Upgrading to 1.0.0]. ◦ Restrict the source to a specific Dataset ID (or create separate sources per dataset) to reduce the number of tables being discovered at once. 1. Increase Internal Request Size Limits (Self-Hosted Airbyte) ◦ The knowledge sources do not provide an officially verified resolution for this specific issue, but related cases showing

413 Request Entity Too Large

during

writeDiscoverCatalogResult

or schema discovery indicate server and proxy payload limits are being exceeded [source-mssql 413 Request Entity Too Large; 413 Request Entity Too Large nginx/1.23.2]. ◦ In self-hosted Kubernetes / Docker deployments, users often attempt to increase the payload threshold on the Micronaut server and ingress/reverse proxy (such as setting

MICRONAUT_SERVER_MAX_REQUEST_SIZE

and increasing NGINX

proxy-body-size

or

client_max_body_size

) [source-mssql 413 Request Entity Too Large; 413 Request Entity Too Large nginx/1.23.2]. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

Limit the Scope of the Source Configuration

this solved my case