@helpful-optician-78938 There is something wrong with the Avro mapping when it comes to logical types. I might be doing something wrong, so I'd like to know if you can take a look. I created a test case that mimics what my source is doing. -- The Setup -- In Iceberg, there is a

DecimalType

that I am trying to map to a

NumberTypeClass

. I think this mapping makes sense and it's what I can see in schema_util.py. This map is relying on the Avro

logical_type

property. The logical map is being used here. -- The Problem -- What I see when my test runs is

actual_schema

is of type

avro.schema.BytesDecimalSchema

(class definition here). It is not setting a

logicalType

property with

set_prop()

, so when

schema_util

tries to use it, the returned value is

None

and the

decimal

key mapping is never used. If I change the code in schema_util to use the

logical_type

Python property, everything works. I don't know if I explained it well with all my code references 😅 , but here is a simple test case to reproduce the problem:

def test_avro():
    avro_schema = {'type': 'record', 'name': '__struct_', 'fields': [{'name': 'name', 'type': {'type': 'bytes', 'logicalType': 'decimal', 'precision': 3, 'scale': 2, 'native_data_type': 'decimal(3, 2)', '_nullable': True}}]}
    newfields = schema_util.avro_schema_to_mce_fields(
        json.dumps(avro_schema), default_nullable=True
    )
    assert len(newfields) == 1
    schema_field: SchemaField = newfields[0]
    assert isinstance(schema_field.type.type, NumberTypeClass)

In this code,

avro_schema

is what my Iceberg source is generating for an Iceberg DecimalType. I expect to get a

NumberTypeClass

as a Datahub type, but I get a

BytesTypeClass

.