Hi All Is there any way of creating permanent tables without Apache Flink #troubleshooting

Hi All, Is there any way of creating permanent tab...

Sharath Gururaj

09/12/2023, 1:20 PM

Hi All, Is there any way of creating permanent tables without using hive catalog? It seems a bit much to create a hive cluster just to store metadata. If i use the InMemoryCatalog, all the state is lost if I restart the cluster. The JDBCCatalog does not support creating flink managed tables. So it seems like hive is required for any non-trivial production flink cluster.. am I right?

Jane Chan

09/12/2023, 1:22 PM

You might interested in https://paimon.apache.org/docs/master/engines/flink/

Sharath Gururaj

09/12/2023, 1:25 PM

thanks I will check it out.. just wondering whether most of the community indeed uses hive for their catalog..

Sharath Gururaj

09/12/2023, 1:29 PM

the other catalog are simply not an option at all.. and If we can't store state durably, its pretty much impossible to write any non-trivial SQL flink application

Sharath Gururaj

09/12/2023, 1:36 PM

Furthermore, the InMemoryCatalog does not support CTAS (

create table foo as select * from ...

). This means I cant use "materialized views" to share the computation between several pipelines. i.e., The InMemoryCatalog only supports "temporary tables" which inlines the computation, so there will be duplication of computation even if I reuse the temporary table across different pipelines

Brian Koehler

09/12/2023, 8:31 PM

You can use side outputs if you need to reuse a temp table for multiple "pipeline outputs". https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/side_output/

Open in Slack

Previous Next