curious issue, configured a HMS as catalog. then c...
# troubleshooting
g
curious issue, configured a HMS as catalog. then created the below catalogs in it via flink and the associated databases nside the catalogs. notice how the postgres_catalog has the 2 databases from the kafka_catalog listed. The commands are from a totally new, nothing exist environment. catalog is backed by a postgre sql database. I understood each catalog is it's own namespace... if the same is done using a in memory catalog then this does not happen.
Copy code
Flink SQL> CREATE CATALOG kafka_catalog WITH (
>   'type'          = 'hive',
>   'hive-conf-dir' = './conf/'
> );
[INFO] Execute statement succeeded.

Flink SQL> use catalog kafka_catalog;
> 
[INFO] Execute statement succeeded.

Flink SQL> CREATE DATABASE IF NOT EXISTS kafka_catalog.inbound;  
> 
[INFO] Execute statement succeeded.

Flink SQL> CREATE DATABASE IF NOT EXISTS kafka_catalog.outbound;  
> 
[INFO] Execute statement succeeded.

Flink SQL> CREATE CATALOG postgres_catalog WITH (
>   'type'          = 'hive',
>   'hive-conf-dir' = './conf/'
> );
[INFO] Execute statement succeeded.

Flink SQL> use catalog kafka_catalog;
[INFO] Execute statement succeeded.

Flink SQL> show databases;
+---------------+
| database name |
+---------------+
|       default |
|       inbound |
|      outbound |
+---------------+
3 rows in set

Flink SQL> use catalog postgres_catalog;
[INFO] Execute statement succeeded.

Flink SQL> show databases;
+---------------+
| database name |
+---------------+
|       default |
|       inbound |
|      outbound |
+---------------+
3 rows in set
Copy code
Flink SQL> use catalog kafka_catalog;
[INFO] Execute statement succeeded.

Flink SQL> create database test;
[INFO] Execute statement succeeded.

Flink SQL> use catalog postgres_catalog;
[INFO] Execute statement succeeded.

Flink SQL> show databases;
+---------------+
| database name |
+---------------+
|       default |
|       inbound |
|      outbound |
|          test |
+---------------+
j
It is because Flink catalogs provide logical organization of tables and other objects but not physical isolation. Since both are configured to point to the same Hive Metastore, they are essentially two different views into the same central metadata repository.
🫣 1
g
this is like creating 2 tables in the same database in 2 schema's but because the storage is the same database it's the same table... so does not make sense coming from a DB world for the last 25yrs... but smile grin and move on, guess. thanks.
the storage in this case will be inbound from postgress or kafka and outbound to kafka and/or paimon.