I'm trying to get the following snippet to work on AWS Managed Apache Flink:

CREATE TABLE input_stream (
  `proctime` AS PROCTIME(),
  `eventtime` TIMESTAMP(3),
  `key` INT,
  WATERMARK FOR `eventtime` as `eventtime`
)
PARTITIONED BY `key`
WITH ( 'connector' = 'kinesis', 'stream' = '...')

CREATE TABLE large_read_table (
  `key` INT,
  `annotation` INT,
  PRIMARY KEY (`key`) NOT ENFORCED
)
WITH ( 'connector' = 'iceberg', 'warehouse' = '...', ... )

CREATE TABLE output_stream (
  `proctime` TIMESTAMP_LTZ(3),
  `eventtime` TIMESTAMP(3),
  `key` INT
  `annotation` INT,
  WATERMARK FOR `eventtime` as `eventtime`
)
PARTITIONED BY `key`
WITH ( 'connector' = 'kinesis', 'stream' = '...')

INSERT INTO output_stream live
SELECT
  s.`proctime`,
  s.`eventtime`,
  s.`key`,
  t.`annotation`
FROM input_stream s
LEFT JOIN large_read_table t
ON s.`key` = t.`key`

This give me the error:

Table sink '...output_stream' doesn't support consuming update and delete changes which is produced by node Join.

Joining with

LEFT JOIN large_read_table FOR SYSTEM_TIME AS OF s.proctime AS t

instead, I get the error:

Processing-time temporal join is not supported yet.

Is there a way to make annotation from lookup table work and have a stream at the end?

Is there a way to make annotation from lookup table work and have a stream at the end?

You could use the lookup work on event time, so you would have an event time temporal join

so if I do

for SYSTEM_TIME AS OF s.eventtime

it should work?

I would expect so yes

Cool. Thanks. Ill try it out now.

You don't have a primary key defined on your kinesis stream

Thank you. I'll give it a try.

is it possible that Iceberg doesn't support being a lookup table?

Definitely. I don't know if they've built in support for that

okay. ill ask over on their what is possible with iceberg. Thank you so much for you time Martijn.