Thanks for the bug confirmation <@U028L1V9BE1>. I...
# integrate-iceberg-datahub
Thanks for the bug confirmation @helpful-optician-78938. It will probably fix DecimalType, as well as TimeType, Timestamp and TimestampZ types... They all use avro logical types. I have another thing that bugs me regarding field descriptions and Avro. This will be easier to show once I update my PR. I will do that tomorrow morning and let you know.
👍 1
Hi @modern-monitor-81461, btw, the fix for this is now merged to master:
👍 1
@helpful-optician-78938 I've updated my code with the latest changes from
and I still have an issue with logical types. In my test, I'm trying to map a
to an avro
logical type and when I look at
, I see the following:
self._actual_schema = {"type": {"type": "int", "logicalType": "date", "native_data_type": "date", "_nullable": false}, "name": "required_field", "doc": "required field documentation"}
is not under
, so that code is not matching and that if is by-passing the lookup in the mapping table.. Am I missing something?
The temp fix you proposed the other day worked
Copy code
              getattr(actual_schema, "logical_type", None)
              or actual_schema.props.get("logicalType")
Thanks @modern-monitor-81461! Could you share the output of
pip freeze | grep avro
Copy code
@helpful-optician-78938 Any update on this? ^ Were you able to reproduce it?
I apologize for the lengthy post... 😊 I took the time to look into this I now understand why @helpful-optician-78938 asked for the avro version. Turns out there is a bug in version
and there is a fix in
, which hasn't been released yet 😢 (I have inquired when it would be. Still waiting...) Here is my documentation of the problem. I wrote a test case to show the problem when Decimal logical type is used:
Copy code
def test_avro_nullable():
    import avro.schema
    decimal_avro_schema_string = """{"type": "record", "name": "__struct_", "fields": [{"name": "required_field", "type": {"type": "bytes", "logicalType": "decimal", "precision": 3, "scale": 2, "native_data_type": "decimal(3, 2)", "_nullable": false}, "doc": "required field documentation"}]}"""
    decimal_avro_schema = avro.schema.parse(decimal_avro_schema_string)
    print(f"Before: {decimal_avro_schema_string}")
    print(f"After:  {decimal_avro_schema}")

    boolean_avro_schema_string = """{"type": "record", "name": "__struct_", "fields": [{"name": "required_field", "type": {"type": "boolean", "native_data_type": "boolean", "_nullable": false}, "doc": "required field documentation"}]}"""
    boolean_avro_schema = avro.schema.parse(boolean_avro_schema_string)
    print(f"Before: {boolean_avro_schema_string}")
    print(f"After:  {boolean_avro_schema}")
the output is:
Copy code
Before: {"type": "record", "name": "__struct_", "fields": [{"name": "required_field", "type": {"type": "bytes", "logicalType": "decimal", "precision": 3, "scale": 2, "native_data_type": "decimal(3, 2)", "_nullable": false}, "doc": "required field documentation"}]}
After:  {"type": "record", "name": "__struct_", "fields": [{"type": {"type": "bytes", "logicalType": "decimal", "precision": 3, "scale": 2, "native_data_type": "decimal(3, 2)", "_nullable": false}, "name": "required_field", "doc": "required field documentation"}]}

Before: {"type": "record", "name": "__struct_", "fields": [{"name": "required_field", "type": {"type": "boolean", "native_data_type": "boolean", "_nullable": false}, "doc": "required field documentation"}]}
After:  {"type": "record", "name": "__struct_", "fields": [{"type": {"type": "boolean", "native_data_type": "boolean", "_nullable": false}, "name": "required_field", "doc": "required field documentation"}]}
Notice for the Decimal output,
are dropped. In the avro code, those are considered
. In avro 1.10.2, here is how
is created: In avro 1.11.1 (master branch), here is the fix: So after looking some more at avro's code, I noticed that there is an alternate path for Decimal type. I added the following test and now I'm able to get
and `native_data_type`:
Copy code
decimal_fixed_avro_schema_string = """{"type": "record", "name": "__struct_", "fields": [{"name": "required_field", "type": {"type": "fixed", "size": 16, "name": "bogusName", "logicalType": "decimal", "precision": 3, "scale": 2, "native_data_type": "decimal(3, 2)", "_nullable": false}, "doc": "required field documentation"}]}"""
    decimal_fixed_avro_schema = avro.schema.parse(decimal_fixed_avro_schema_string)
    print("\nDecimal (fixed)")
    print(f"Before: {decimal_fixed_avro_schema_string}")
    print(f"After:  {decimal_fixed_avro_schema}")
returns this output:
Copy code
Decimal (fixed)
Before: {"type": "record", "name": "__struct_", "fields": [{"name": "required_field", "type": {"type": "fixed", "size": 16, "name": "bogusName", "logicalType": "decimal", "precision": 3, "scale": 2, "native_data_type": "decimal(3, 2)", "_nullable": false}, "doc": "required field documentation"}]}
After:  {"type": "record", "name": "__struct_", "fields": [{"type": {"type": "fixed", "logicalType": "decimal", "precision": 3, "scale": 2, "native_data_type": "decimal(3, 2)", "_nullable": false, "name": "bogusName", "size": 16}, "name": "required_field", "doc": "required field documentation"}]}
All of this to say that @helpful-optician-78938, you don't have to look into this anymore (in case it was on your to-do list). @mammoth-bear-12532 I've said this at the beginning, but decided to still go that route to go faster, but I really don't like the fact that we go through Avro to generate SchemaFields. I understand it was to avoid duplicated code in many sources, but I would be curious to get your opinion on whether we could implement a builder to do that. The builder would track the field paths in a nested record... I might be missing something, but there's got to be a better way then going through avro and being at the mercy of bugs like this. Thoughts?
Hi @modern-monitor-81461, thanks for digging into this. I'll get back to you by tomorrow with the answers to your questions.
Just to add to my previous comment about questioning the use of avro, I'm facing another "bug" with timestamp. Iceberg support timestamps with and without timezone. The avro spec supports local-timestamp-millis and local-timestamp-micros, but the actual Python implementation does not (but the Java one does...). I tried to use
and it wasn't being recognized. I had to dig into avro's code to figure it has been "forgotten". I totally back the need for having common code that can be re-used across ingestion sources, but I'm afraid having to rely on avro is not the best dev-experience. I'm just relating my experience for you guys to maybe make things better...
@helpful-optician-78938 If I may ask since you will answer my questions 😅 , I am trying to figure out why my Arrays are always showing with a
. I have looked around and I can't figure out where we would be setting this. The only place I can see is here in, but we are always going to call the
with no argument, so effectively setting
. Am I right to believe that the current code will never set the array nested type?
will look into this as well.
Totally agree @modern-monitor-81461. We need to plan this though. Will be a medium-large size project.