https://linen.dev logo
Join Slack
Powered by
<@U04S8PLJWMT> Getting error while trying to repli...
# replication-ask-ai
v
@kapa.ai Getting error while trying to replicate data from gcs to snowflake
2026-06-29 173817 INFO 2026-06-29 173817 info 2026-06-29 173817 INFO 2026-06-29 173817 info ----- START DISCOVER ----- 2026-06-29 173817 INFO 2026-06-29 173817 info 2026-06-29 173822 INFO 2026-06-29 173822 info Connector exited, processing output 2026-06-29 173822 INFO 2026-06-29 173822 info Output file jobOutput.json found 2026-06-29 173822 INFO 2026-06-29 173822 info Connector exited with exit code 1 2026-06-29 173822 INFO 2026-06-29 173822 info Reading messages from protocol version 0.2.0 2026-06-29 173822 WARN 2026-06-29 173822 warn ignoring unsupported keyword arguments: ['client'] 2026-06-29 173822 ERROR 2026-06-29 173822 error An error occurred inferring the schema. Traceback (most recent call last): File "/usr/local/lib/python3.13/site-packages/airbyte_cdk/sources/file_based/stream/default_file_based_stream.py", line 404, in _infer_file_schema return await self.get_parser().infer_schema( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ self.config, file, self.stream_reader, self.logger ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ) ^ File "/usr/local/lib/python3.13/site-packages/airbyte_cdk/sources/file_based/file_types/avro_parser.py", line 75, in infer_schema field["name"]: AvroParser._convert_avro_type_to_json( # type: ignore [index] ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^ avro_format, ^^^^^^^^^^^^ field["name"], # type: ignore [index] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ field["type"], # type: ignore [index] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ) ^ File "/usr/local/lib/python3.13/site-packages/airbyte_cdk/sources/file_based/file_types/avro_parser.py", line 168, in _convert_avro_type_to_json raise ValueError(f"Unsupported avro type: {avro_field}") ValueError: Unsupported avro type: ['null', 'string'] The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.13/site-packages/airbyte_cdk/sources/file_based/stream/default_file_based_stream.py", line 391, in _infer_schema base_schema = merge_schemas(base_schema, task.result()) ~~~~~~~~~~~^^ File "/usr/local/lib/python3.13/site-packages/airbyte_cdk/sources/file_based/stream/default_file_based_stream.py", line 412, in _infer_file_schema self._raise_schema_inference_error(exc, file) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^ File "/usr/local/lib/python3.13/site-packages/airbyte_cdk/sources/file_based/stream/default_file_based_stream.py", line 417, in _raise_schema_inference_error raise SchemaInferenceError( ...<4 lines>... ) from exc airbyte_cdk.sources.file_based.exceptions.SchemaInferenceError: Error inferring schema from files. Are the files valid? Contact Support if you need assistance. file=https://storage.googleapis.com/lovelocal_dataengg_pipeline_data/ll_bronze_layer/mysql_test/homedelivery/db_banner/2026_06_29_1782733552317_0.avro?X-Goog-Algorithm=GOOG4-RSA-SHA256&amp;X-Goog-Credential=ll-gcs-snowflake-via-airbyte%40mpaanibackend.iam.gserviceaccount.com%2F20260629%2Fauto%2Fstorage%2Fgoog4_request&amp;X-Goog-Date=20260629T120822Z&amp;X-Goog-Expires=604800&amp;X-Goog-SignedHeaders=host&amp;X-Goog-Signature=33bbe47186a8a5e796f661b1dd6ab4bb2afb6244a8a475856c1ab8265ea623106dd064edcb65b6ef9a42722e9769f1b6b8cc9bb48c5d01521d57fa7b22a9fe8696d1c9fdbfe5353caf1154ed4d91699c3f3df80624292beffd60eb45c6aa1a36ed017680feebc80c8a335a2ea28e1811b144f026e5d190af6f5df5294ae12141e9786df2a445b8ae539ce30e7f76aef64ae4710d615d55dad6cff3ff20cd0d23b3a89e0aa0659852160bff1fec941db3d018308211761150deb12bf387707174704afb0881138dd02af3964fad39145c1b6ec732eaba6936ce9a9040e39e349e736910bf6f8b8155a7190fcf0c5a54c622224e2e3810464a53fb796f597afc13 format=filetype='avro' double_as_string=False stream=db_banner Traceback (most recent call last): File "/usr/local/lib/python3.13/site-packages/airbyte_cdk/sources/file_based/stream/default_file_based_stream.py", line 404, in _infer_file_schema return await self.get_parser().infer_schema( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ self.config, file, self.stream_reader, self.logger ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ) ^ File "/usr/local/lib/python3.13/site-packages/airbyte_cdk/sources/file_based/file_types/avro_parser.py", line 75, in infer_schema field["name"]: AvroParser._convert_avro_type_to_json( # type: ignore [index] ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^ avro_format, ^^^^^^^^^^^^ field["name"], # type: ignore [index] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ field["type"], # type: ignore [index] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ) ^ File "/usr/local/lib/python3.13/site-packages/airbyte_cdk/sources/file_based/file_types/avro_parser.py", line 168, in _convert_avro_type_to_json raise ValueError(f"Unsupported avro type: {avro_field}") ValueError: Unsupported avro type: ['null', 'string'] The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.13/site-packages/airbyte_cdk/sources/file_based/stream/default_file_based_stream.py", line 391, in _infer_schema base_schema = merge_schemas(base_schema, task.result()) ~~~~~~~~~~~^^ File "/usr/local/lib/python3.13/site-packages/airbyte_cdk/sources/file_based/stream/default_file_based_stream.py", line 412, in _infer_file_schema self._raise_schema_inference_error(exc, file) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^ File "/usr/local/lib/python3.13/site-packages/airbyte_cdk/sources/file_based/stream/default_file_based_stream.py", line 417, in _raise_schema_inference_error raise SchemaInferenceError( ...<4 lines>... ) from exc airbyte_cdk.sources.file_based.exceptions.SchemaInferenceError: Error inferring schema from files. Are the files valid? Contact Support if you need assistance. file=https://storage.googleapis.com/lovelocal_dataengg_pipeline_data/ll_bronze_layer/mysql_test/homedelivery/db_banner/2026_06_29_1782733552317_0.avro?X-Goog-Algorithm=GOOG4-RSA-SHA256&amp;X-Goog-Credential=ll-gcs-snowflake-via-airbyte%40mpaanibackend.iam.gserviceaccount.com%2F20260629%2Fauto%2Fstorage%2Fgoog4_request&amp;X-Goog-Date=20260629T120822Z&amp;X-Goog-Expires=604800&amp;X-Goog-SignedHeaders=host&amp;X-Goog-Signature=33bbe47186a8a5e796f661b1dd6ab4bb2afb6244a8a475856c1ab8265ea623106dd064edcb65b6ef9a42722e9769f1b6b8cc9bb48c5d01521d57fa7b22a9fe8696d1c9fdbfe5353caf1154ed4d91699c3f3df80624292beffd60eb45c6aa1a36ed017680feebc80c8a335a2ea28e1811b144f026e5d190af6f5df5294ae12141e9786df2a445b8ae539ce30e7f76aef64ae4710d615d55dad6cff3ff20cd0d23b3a89e0aa0659852160bff1fec941db3d018308211761150deb12bf387707174704afb0881138dd02af3964fad39145c1b6ec732eaba6936ce9a9040e39e349e736910bf6f8b8155a7190fcf0c5a54c622224e2e3810464a53fb796f597afc13 format=filetype='avro' double_as_string=False stream=db_banner
k
The knowledge sources don't contain enough specific information about a GCS source to Snowflake destination pipeline error to give you a precise answer. However, based on what's available, here are some relevant points and common issues: GCS → Snowflake: Known Issues & Tips 1. Snowflake No Longer Supports GCS/S3 Staging (Destination Side) If you're using an older version of the Snowflake destination connector, note that: > Snowflake no longer supports GCS/S3. Please migrate to the Internal Staging option. This is recommended by Snowflake and is cheaper and faster. [Snowflake Migration Guide] 2. GCS Credential Issues A common cause of GCS connection failures is invalid or misconfigured credentials. One forum case showed this error when using HMAC keys: > "The provided security credentials are not valid. (Service: Amazon S3; Status Code: 403; Error Code: InvalidSecurity)" Suggestions from that case: • Try creating new credentials with sufficient permissions. • Be lenient with permissions initially to rule out access issues. [GCS Destination Connection Failed] 3. Empty Stream Failure (Snowflake Destination v4.0.31) If your GCS source returns empty streams, there is a known open bug in 
destination-snowflake
v4.0.31 where syncs fail with: > 
Expected partition_id and id in additional properties (got [id])
There is currently no user-side workaround — it requires a code fix. [Snowflake Empty Stream Bug] To help diagnose your specific issue, could you share: • The exact error message you're seeing? • The connector versions (source-gcs and destination-snowflake)? • The Airbyte version you're running? 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
2 Views
Open in Slack
PreviousNext
Powered by