George Leonard
07/26/2024, 3:17 PMCREATE TABLE t_f_unnested_sales (
`store_id` STRING,
`product` STRING,
`brand` STRING,
`saleValue` DOUBLE,
`category` STRING,
`saleDateTime_Ltz` STRING,
`saleTimestamp_Epoc` STRING,
`saleTimestamp_WM` AS TO_TIMESTAMP(FROM_UNIXTIME(CAST(`saleTimestamp_Epoc` AS BIGINT) / 1000)),
WATERMARK FOR `saleTimestamp_WM` AS `saleTimestamp_WM`
) WITH (
'connector' = 'kafka',
'topic' = 'unnested_sales',
'properties.bootstrap.servers' = 'broker:29092',
'properties.group.id' = 'testGroup',
'scan.startup.mode' = 'earliest-offset',
'value.format' = 'avro-confluent',
'value.avro-confluent.url' = '<http://schema-registry:9081>',
'value.fields-include' = 'ALL'
);
insert into t_f_unnested_sales
SELECT
`store`.`id` as `store_id`,
bi.`name` AS `product`,
bi.`brand` AS `brand`,
bi.`price` * bi.`quantity` AS `saleValue`,
bi.`category` AS `category`,
`saleDateTime_Ltz` as saleDateTime_Ltz,
`saleTimestamp_Epoc` as saleTimestamp_Epoc
FROM t_f_avro_salescompleted_x -- assuming avro_salescompleted_x is a table function
CROSS JOIN UNNEST(`basketItems`) AS bi;
I've got for now a hive-metastore-standalone locally deployed, with internal database, the problems i'm having is not cause by the internal database as I've tried this with a external postgresql store also.
I now try and create a table/output being pushed to S3/minio in iceberg format
CREATE TABLE t_i_unnested_sales WITH (
'connector' = 'iceberg',
'catalog-type'='hive',
'catalog-name'='dev',
'warehouse' = '<s3a://warehouse>',
'hive-conf-dir' = './conf')
LIKE t_f_unnested_sales;
Insert into t_i_unnested_sales
SELECT * FROM t_f_unnested_sales;
and end with this error.
-- > [ERROR] Could not execute SQL statement. Reason:
-- > java.lang.UnsupportedOperationException: Creating table with computed columns is not supported yet.
I created my catalog using:
CREATE CATALOG c_iceberg_hive WITH (
'type' = 'iceberg',
'catalog-type' = 'hive',
'warehouse' = '<s3a://warehouse>',
'hive-conf-dir' = './conf'
);
USE CATALOG `c_iceberg_hive`;
CREATE DATABASE `c_iceberg_hive`.`db01`;
to get around the problem I also tried creating the table with a saleTimestamp timestamp(3) column and doing the conversion in the insert statement resulting in a error saying watermarks are not supported.
the ask... what is the solution, watermarks are such a fundamental thing, a catalog is required bit of all of this...
Iceberg is the data format, referred to as the OTF as i understand, the catalog is the repository containing the structures/definitions of whats stored in a object store, in lab case this is minio, HIVE plays the role of catalog, seems to be the top pick, the Hive catalog is store in a hive metastore, with what looks like primarily one back end storage engine (in standalone configuration anyhow), PostgreSQL.
My problem from above, from what i can figure out, hive does not support the derived column in the table, nor the watermark column.
whats my options.George Leonard
07/27/2024, 10:57 AM