Hi there 👋 I’m having issues setting up a source in S3 where the files have Avro format and gzip compression. The Airbyte version I use is: 0.39.35-alpha The s3 source version is: 0.1.25 And this is the error I see in the logs:

2022-11-22 12:13:20 [1;31mERROR[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):95 - 'avro.schema'
Traceback (most recent call last):
  File "/airbyte/integration_code/main.py", line 13, in <module>
    launch(source, sys.argv[1:])
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/entrypoint.py", line 131, in launch
    for message in source_entrypoint.run(parsed_args):
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/entrypoint.py", line 116, in run
    catalog = self.source.discover(self.logger, config)
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 72, in discover
    streams = [stream.as_airbyte_stream() for stream in self.streams(config=config)]
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 72, in <listcomp>
    streams = [stream.as_airbyte_stream() for stream in self.streams(config=config)]
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/streams/core.py", line 113, in as_airbyte_stream
    stream = AirbyteStream(name=self.name, json_schema=dict(self.get_json_schema()), supported_sync_modes=[SyncMode.full_refresh])
  File "/airbyte/integration_code/source_s3/source_files_abstract/stream.py", line 184, in get_json_schema
    for column, typ in self._get_schema_map().items()
  File "/airbyte/integration_code/source_s3/source_files_abstract/stream.py", line 169, in _get_schema_map
    return_schema = self._get_master_schema()
  File "/airbyte/integration_code/source_s3/source_files_abstract/stream.py", line 234, in _get_master_schema
    this_schema = file_reader.get_inferred_schema(f, file_info)
  File "/airbyte/integration_code/source_s3/source_files_abstract/formats/avro_parser.py", line 79, in get_inferred_schema
    avro_schema = self._get_avro_schema(file)
  File "/airbyte/integration_code/source_s3/source_files_abstract/formats/avro_parser.py", line 65, in _get_avro_schema
    avro_reader = fastavro.reader(file)
  File "fastavro/_read.pyx", line 1134, in fastavro._read.reader.__init__
  File "fastavro/_read.pyx", line 1095, in fastavro._read.file_reader.__init__
KeyError: 'avro.schema'

I get the same error when I try to load the gzipped file with fastavro, so my guess is that Airbyte is not detecting the compression format. Is Avro gzipped supported for S3 source? I could not see how to specify the compression format either. Any help is appreciated, thank you!

Today only avro without compressoin are supported. See more: https://docs.airbyte.com/integrations/sources/s3#avro

Thank you Marcos for making it clear I did not realise this from the documentation. I see the issue is in the backlog, unclaimed and unscoped. Do you have any rough estimate on when this could be picked up? I’m asking because depending on how long this could take, it could make sense from my side to change the format in the data producers instead.

The issue isn’t in team backlog prioritize so isn’t possible to give you an ETA