@kapa.ai What is the error when using AWS S3 as data source?

2025-07-24 16:41:29 error An error occurred inferring the schema. 
 Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/airbyte_cdk/sources/file_based/stream/default_file_based_stream.py", line 385, in _infer_file_schema
    return await self.get_parser().infer_schema(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/airbyte_cdk/sources/file_based/file_types/avro_parser.py", line 74, in infer_schema
    json_schema = {
                  ^
  File "/usr/local/lib/python3.11/site-packages/airbyte_cdk/sources/file_based/file_types/avro_parser.py", line 75, in <dictcomp>
    field["name"]: AvroParser._convert_avro_type_to_json(  # type: ignore [index]
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/airbyte_cdk/sources/file_based/file_types/avro_parser.py", line 168, in _convert_avro_type_to_json
    raise ValueError(f"Unsupported avro type: {avro_field}")
ValueError: Unsupported avro type: ['null', 'long']

The error message you are seeing:

ValueError: Unsupported avro type: ['null', 'long']

indicates that the Airbyte S3 source connector encountered an Avro field with the type

['null', 'long']

and was unable to convert it to a JSON schema type during schema inference. This type of error typically occurs when the Avro file contains union types (e.g., fields that can be either

null

or

long

), which is a common pattern for representing nullable fields in Avro. Airbyte's Avro parser is expected to handle these, as the mapping from JSON schema to Avro for nullable fields is usually

["null", "<type>"]

(for example,

["null", "long"]

for nullable integers) [JSON to Avro Conversion Rules](https://docs.airbyte.com/platform/understanding-airbyte/json-avro-conversion#conversion-rules). However, the error suggests that the parser does not support this union type in your current version or configuration. This could be due to a bug or a missing feature in the Avro parser implementation. There have been related issues in the past where certain logical types or union types were not handled correctly, leading to similar errors during schema inference or parsing [[source]](https://github.com/airbytehq/airbyte/issues/36920). What you can do: - Make sure you are using the latest version of the Airbyte S3 source connector, as there have been updates to improve Avro logical type and union type handling. - If the issue persists on the latest version, it may be a bug or limitation in the current Avro parser. In that case, consider opening an issue on the Airbyte GitHub repository with details about your Avro schema and the error message. In summary, the error is due to the Airbyte Avro parser not supporting the union type

['null', 'long']

during schema inference, even though this is a standard way to represent nullable fields in Avro [JSON to Avro Conversion Rules](https://docs.airbyte.com/platform/understanding-airbyte/json-avro-conversion#conversion-rules).