Apache Flink

Has anyone tried using Iceberg table as Sink, but writing to multiple Iceberg tables based on some properties of RowData in runtime?

It does not look like this is supported.
<https://github.com/apache/iceberg/issues/2208>

I am thinking about how I can implement either a workaround or a custom Sink function that can write to multiple Iceberg tables in runtime

You could split the stream in a ProcessFunction (using side outputs), and have multiple sinks, but you’d need to know the set of possible values (and this set couldn’t be very big).

<@U03J4C16PM2> Yeah won’t work in our use case. The sink Iceberg tables will change over time, and will have to be dynamically determined.

I am re-writing IcebergFilesComitter and IcebergFilesWriter to support this use case specifically for our use case.

I ended up re-writing `IcebergFilesCommitter` and `IcebergFilesWriter` to write to multiple iceberg tables in runtime based on certain conditions of the record property!

Interesting - might be worth a blog post :slightly_smiling_face:

good idea. I will write one in a couple of weeks :laughing: !