ok, running into something i can't think is realit...
# random
g
ok, running into something i can't think is reality... this is one of my tables, it gets data from a kafka topic, as can be seen my timestamp assigned comes in as a string, saleTimestamp_Epoc which I then need to work with to get to a proper format so that it can be used to assign a water mark, as water marks are needed for grouping/time based windows.
Copy code
CREATE TABLE t_f_unnested_sales (
    `store_id` STRING,
    `product` STRING,
    `brand` STRING,
    `saleValue` DOUBLE,
    `category` STRING,
    `saleDateTime_Ltz` STRING,
    `saleTimestamp_Epoc` STRING,
    `saleTimestamp_WM` AS TO_TIMESTAMP(FROM_UNIXTIME(CAST(`saleTimestamp_Epoc` AS BIGINT) / 1000)),
    WATERMARK FOR `saleTimestamp_WM` AS `saleTimestamp_WM`
) WITH (
    'connector' = 'kafka',
    'topic' = 'unnested_sales',
    'properties.bootstrap.servers' = 'broker:29092',
    'properties.group.id' = 'testGroup',
    'scan.startup.mode' = 'earliest-offset',
    'value.format' = 'avro-confluent',
    'value.avro-confluent.url' = '<http://schema-registry:9081>',
    'value.fields-include' = 'ALL'
);

insert into t_f_unnested_sales
SELECT
      `store`.`id` as `store_id`,
      bi.`name` AS `product`,
      bi.`brand` AS `brand`,
      bi.`price` * bi.`quantity` AS `saleValue`,
      bi.`category` AS `category`,
      `saleDateTime_Ltz` as saleDateTime_Ltz,
      `saleTimestamp_Epoc` as saleTimestamp_Epoc
    FROM t_f_avro_salescompleted_x  -- assuming avro_salescompleted_x is a table function
    CROSS JOIN UNNEST(`basketItems`) AS bi;
I've got for now a hive-metastore-standalone locally deployed, with internal database, the problems i'm having is not cause by the internal database as I've tried this with a external postgresql store also. I now try and create a table/output being pushed to S3/minio in iceberg format
Copy code
CREATE TABLE t_i_unnested_sales WITH (
  'connector' = 'iceberg',
  'catalog-type'='hive',
  'catalog-name'='dev',
  'warehouse' = '<s3a://warehouse>',
  'hive-conf-dir' = './conf')
LIKE t_f_unnested_sales;

Insert into t_i_unnested_sales
  SELECT * FROM t_f_unnested_sales;
and end with this error.
Copy code
-- > [ERROR] Could not execute SQL statement. Reason:
-- > java.lang.UnsupportedOperationException: Creating table with computed columns is not supported yet.
I created my catalog using:
Copy code
CREATE CATALOG c_iceberg_hive WITH (
   'type'            = 'iceberg',
   'catalog-type'    = 'hive',
   'warehouse'       = '<s3a://warehouse>',
   'hive-conf-dir'   = './conf'
);

USE CATALOG `c_iceberg_hive`;
CREATE DATABASE `c_iceberg_hive`.`db01`;
to get around the problem I also tried creating the table with a saleTimestamp timestamp(3) column and doing the conversion in the insert statement resulting in a error saying watermarks are not supported. the ask... what is the solution, watermarks are such a fundamental thing, a catalog is required bit of all of this... Iceberg is the data format, referred to as the OTF as i understand, the catalog is the repository containing the structures/definitions of whats stored in a object store, in lab case this is minio, HIVE plays the role of catalog, seems to be the top pick, the Hive catalog is store in a hive metastore, with what looks like primarily one back end storage engine (in standalone configuration anyhow), PostgreSQL. My problem from above, from what i can figure out, hive does not support the derived column in the table, nor the watermark column. whats my options.
Resolved... figured out... will post a what i did ltr. for now its with the hms standalone with derby db, but pretty confident it will work with the postgresql backend also