Hi guys. I had a question about Hive metastore an...
# troubleshootingb
Benjamin Wootton
02/20/2023, 1:22 PMHi guys. I had a question about Hive metastore and Hudi.
In Flink SQL I create a catalog pointing to Hive metastore.
I then create a database and a Hudi table. The Hudi table is created and Hive Metastore learns about the database and the table.
The problem is that Hive does not know about the columns, only the table object. Is this by design or am I doing something wrong?
n
Nicholas Jiang
02/20/2023, 1:23 PMDid you use the
HoodieHiveCatalogb
Benjamin Wootton
02/20/2023, 1:24 PMI didn’t…. Google doesnt return much on HoodieHiveCatalog?
n
Nicholas Jiang
02/20/2023, 1:26 PMYou could refer to https://github.com/apache/hudi/blob/master/hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/catalog/HoodieHiveCatalog.java. Meanwhile, you should use
type=hudimodeHMSb
Benjamin Wootton
02/20/2023, 1:27 PMok trying that…
Benjamin Wootton
02/20/2023, 1:37 PMthat seems to have helped, thanks.
Benjamin Wootton
02/20/2023, 1:38 PMWhy does Hive metastore need to know about Hudi? It should be able to store the table metadata independent of the storage format no?
n
Nicholas Jiang
02/20/2023, 1:40 PMGreat. Because HoodieHiveCatalog gets the Hudi metadata from HMS which is special to HMS.
b
Benjamin Wootton
02/20/2023, 1:43 PMI think if I’m creating a Hudi or an Iceberg table from Flink, with an external catalog HMS configured, I think we should be able to write the metadata thorugh the hive catalog. It seems to work like that but doesnt serialise the column names.
Benjamin Wootton
02/20/2023, 1:43 PMMaybe Im misunderstanding or maybe i hit a bug….
Benjamin Wootton
02/20/2023, 1:43 PMthanks for the help though. will continue….
Benjamin Wootton
02/20/2023, 1:44 PMI thought HMS was a service independent of the underlying storage formats
n
Nicholas Jiang
02/20/2023, 1:44 PMDifferent table format store different metadata of format itself.
Nicholas Jiang
02/20/2023, 1:45 PMThe meta info isn’t unified.
b
Benjamin Wootton
02/20/2023, 1:45 PMwould it be a similar pattern for iceberg tables?
n
Nicholas Jiang
02/20/2023, 1:45 PMNo.
Nicholas Jiang
02/20/2023, 1:48 PMBut this requires the storage meta unfied.
2 Views