https://datahubproject.io logo
#integrate-iceberg-datahub
Title
# integrate-iceberg-datahub
h

helpful-optician-78938

03/15/2022, 11:40 PM
Hi @modern-monitor-81461, thanks for reporting it. This is indeed a bug. Just built a fix and tested it. Will clean it up, add test coverage and raise the PR to OSS soon with the fix. In the meantime, if you want to unblock yourself, you can change this code to
Copy code
type=self._converter._get_column_type(
         actual_schema.type,
          (
              getattr(actual_schema, "logical_type", None)
              or actual_schema.props.get("logicalType")
           ),
),
m

modern-monitor-81461

03/16/2022, 11:04 AM
@helpful-optician-78938 While you are in this file, would it be possible to add missing Avro logical types about time and timestamps? Right now we only support those:
Copy code
field_logical_type_mapping: Dict[str, Any] = {
        "date": DateTypeClass,
        "timestamp-millis": TimeTypeClass,
        "decimal": NumberTypeClass,
    }
But the Avro spec has a few more: • time-millis • time-micros • timestamp-micros • local-timestamp-millis • local-timestamp-micros • duration • uuid In Iceberg, I need to map the following types: • UUIDType -> mapped as an Avro String right now, would be nice to use
uuid
logical type • TimeType -> right mapping would be
time-micros
• TimestampType (with local and UTC options) -> right mapping would be
timestamp-micros
or
local-timestamp-micros
@helpful-optician-78938 Another thing I'm having issues with is the
nullable
field. I can't figure out how to use it with Avro. I know the magic is happening here, but depending on the Avro mapping used, I get different results. I just can't figure out the pattern to use... I've updated my Iceberg PR with my latest try to make nullable work. It almost works, but it fails for DecimalType. I can go into details, but first I'd like to know if what I'm doing is the right way or not. I thought I could use
_nullable
in Avro like this. Am I right?