@kapa.ai Getting error while reading files from gcs to send to snowflake

Thinking completed The knowledge sources do not contain enough information to identify the exact cause without your specific error log. However, when reading files from Google Cloud Storage (GCS) and syncing them to Snowflake, several common issues typically occur: 1. Content-Encoding: gzip Metadata on Uncompressed Files If the error occurs while reading CSV files from GCS (often showing decompression failures such as

TypeError: _GzipDecoder.decompress() got an unexpected keyword argument 'max_length'

or decompressive transcoding issues) [source-Google Cloud Storage (GCS)] connector fails on files with Content-Encoding: gzip in Airbyte Cloud, [SourceGCSStreamReader]: • Cause: GCS blobs sometimes carry the metadata header

Content-Encoding: gzip

even when the underlying file is uncompressed (common with files automatically generated by Google Play reports) [source-Google Cloud Storage (GCS)] connector fails on files with Content-Encoding: gzip in Airbyte Cloud. • Fix: Update to the latest version of the

source-gcs

connector, or remove the metadata header directly using

gsutil

[source-Google Cloud Storage (GCS)] connector fails on files with Content-Encoding: gzip in Airbyte Cloud:gsutil setmeta -h "Content-Encoding:" gs://your-bucket/path-to-file 2. URL Query Parameters Injected into Schema (Parquet Files) If you are reading Parquet files and notice gibberish column names (such as

...parquet?X-Goog-Algorithm

) repeatedly altering your Snowflake table schema [GCS] Unwanted column with name 2026_06_29_1782736372647_0.parquet?X-Goog-Algorithm coming in connection: • Cause: In older versions of

source-gcs

, signed-URL query parameters were mistakenly treated by the Parquet parser as Hive partition columns [GCS] Unwanted column with name 2026_06_29_1782736372647_0.parquet?X-Goog-Algorithm coming in connection, [SourceGCSStreamReader]. • Fix: Upgrade the GCS Source connector to the latest version, which uses canonical

gs://

URIs internally to prevent query string leakage into schemas [SourceGCSStreamReader]. 3. Snowflake Destination Staging Configuration If the error occurs on the destination side when staging data: • Note that starting with

destination-snowflake

version 2.0.0, external staging via GCS/S3 is no longer supported [Upgrading to 2.0.0]. • Make sure your Snowflake destination is configured to use the Internal Staging option [Upgrading to 2.0.0]. 4. GCS Permissions / Credentials Error If you receive access errors (such as

403 Forbidden

or

InvalidSecurity

) [BigQuery GCS staging failed, GCS destinaltion connection failed]: • Ensure your Google Service Account or HMAC credentials have sufficient permissions to list, read objects, and read bucket metadata [BigQuery GCS staging failed]. If none of these match your issue, please share the exact error message and stack trace from your sync logs, along with the connector versions in use, to help pinpoint the solution. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai Error: 2026-10-05 073905 platform > 2026-10-05 073905 platform > ----- START DISCOVER ----- 2026-10-05 073905 platform > 2026-10-05 073912 platform > Connector exited, processing output 2026-10-05 073912 platform > Output file jobOutput.json found 2026-10-05 073912 platform > Connector exited with exit code 1 2026-10-05 073912 platform > Reading messages from protocol version 0.2.0 2026-10-05 073912 platform > MemoryMonitor instantiated with critical threshold: 95%, anon share of usage threshold: 85%, high-pressure threshold: 90%, check interval: 5000 messages (tightens to 100 under high pressure). 2026-10-05 073912 platform > MemoryMonitor instantiated with critical threshold: 95%, anon share of usage threshold: 85%, high-pressure threshold: 90%, check interval: 5000 messages (tightens to 100 under high pressure). 2026-10-05 073912 platform > An error occurred inferring the schema. Traceback (most recent call last): File "/usr/local/lib/python3.13/site-packages/airbyte_cdk/sources/file_based/stream/default_file_based_stream.py", line 404, in _infer_file_schema return await self.get_parser().infer_schema( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ self.config, file, self.stream_reader, self.logger ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ) ^ File "/usr/local/lib/python3.13/site-packages/airbyte_cdk/sources/file_based/file_types/avro_parser.py", line 75, in infer_schema field["name"]: AvroParser._convert_avro_type_to_json( # type: ignore [index] ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^ avro_format, ^^^^^^^^^^^^ field["name"], # type: ignore [index] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ field["type"], # type: ignore [index] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ) ^ File "/usr/local/lib/python3.13/site-packages/airbyte_cdk/sources/file_based/file_types/avro_parser.py", line 168, in _convert_avro_type_to_json raise ValueError(f"Unsupported avro type: {avro_field}") ValueError: Unsupported avro type: ['null', 'long'] The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.13/site-packages/airbyte_cdk/sources/file_based/stream/default_file_based_stream.py", line 391, in _infer_schema base_schema = merge_schemas(base_schema, task.result()) ~~~~~~~~~~~^^ File "/usr/local/lib/python3.13/site-packages/airbyte_cdk/sources/file_based/stream/default_file_based_stream.py", line 412, in _infer_file_schema self._raise_schema_inference_error(exc, file) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^ File "/usr/local/lib/python3.13/site-packages/airbyte_cdk/sources/file_based/stream/default_file_based_stream.py", line 417, in _raise_schema_inference_error raise SchemaInferenceError( ...<4 lines>... ) from exc airbyte_cdk.sources.file_based.exceptions.SchemaInferenceError: Error inferring schema from files. Are the files valid? Contact Support if you need assistance. file=gs://lovelocal_dataengg_pipeline_data/ll_bronze_layer/mysql/homedelivery/db_scratchcard/2026_02_05_1770275330582_2.avro format=filetype='avro' double_as_string=False stream=db_scratchcard Traceback (most recent call last): File "/usr/local/lib/python3.13/site-packages/airbyte_cdk/sources/file_based/stream/default_file_based_stream.py", line 404, in _infer_file_schema return await self.get_parser().infer_schema( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ self.config, file, self.stream_reader, self.logger ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ) ^ File "/usr/local/lib/python3.13/site-packages/airbyte_cdk/sources/file_based/file_types/avro_parser.py", line 75, in infer_schema field["name"]: AvroParser._convert_avro_type_to_json( # type: ignore [index] ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^ avro_format, ^^^^^^^^^^^^ field["name"], # type: ignore [index] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ field["type"], # type: ignore [index] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ) ^ File "/usr/local/lib/python3.13/site-packages/airbyte_cdk/sources/file_based/file_types/avro_parser.py", line 168, in _convert_avro_type_to_json raise ValueError(f"Unsupported avro type: {avro_field}") ValueError: Unsupported avro type: ['null', 'long'] The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.13/site-packages/airbyte_cdk/sources/file_based/stream/default_file_based_stream.py", line 391, in _infer_schema base_schema = merge_schemas(base_schema, task.result()) ~~~~~~~~~~~^^ File "/usr/local/lib/python3.13/site-packages/airbyte_cdk/sources/file_based/stream/default_file_based_stream.py", line 412, in _infer_file_schema self._raise_schema_inference_error(exc, file) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^ File "/usr/local/lib/python3.13/site-packages/airbyte_cdk/sources/file_based/stream/default_file_based_stream.py", line 417, in _raise_schema_inference_error raise SchemaInferenceError( ...<4 lines>... ) from exc airbyte_cdk.sources.file_based.exceptions.SchemaInferenceError: Error inferring schema from files. Are the files valid? Contact Support if you need assistance. file=gs://lovelocal_dataengg_pipeline_data/ll_bronze_layer/mysql/homedelivery/db_scratchcard/2026_02_05_1770275330582_2.avro format=filetype='avro' double_as_string=False stream=db_scratchcard

Thinking completed The failure occurs during the schema discovery phase because the GCS source connector's Avro parser does not currently support Avro union types:

ValueError: Unsupported avro type: ['null', 'long']
airbyte_cdk.sources.file_based.exceptions.SchemaInferenceError: Error inferring schema from files.

Cause In Avro schemas, optional/nullable fields are represented as union types containing

"null"

and the base type (in this case,

['null', 'long']

). When the source connector attempts to infer the schema from your file (

2026_02_05_1770275330582_2.avro

),

AvroParser._convert_avro_type_to_json

fails because it expects a primitive type rather than a list/union representing nullable fields [JSON to Avro Conversion]. Potential Workarounds 1. Check for Connector Updates: If you are running an older version of the

source-gcs

connector, update it to the latest version to see if support for Avro union types/nullable fields has been resolved in the CDK's Avro parser. 2. Convert or Export to Parquet/JSONL/CSV: If the upstream system generating these files is under your control (or if you can sync them as Parquet or JSONL), consider writing the data in Parquet or JSON format instead, which have broader parser support in the file-based connector framework. 3. Report or Track the Issue: Because this is an unhandled case in the Python CDK's

avro_parser.py

, consider opening an issue on the Airbyte GitHub repository or contacting Airbyte Support with this stack trace so support for union/nullable Avro schemas can be patched. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.