Hi, noob question, when I create a new instance of Hive catalog in a Flink SQL client - I expected it to persist when I exit the client and connect again ( reset the session ), I am not seeing that happen, is that expected? isn't catalog persisted in HMS?

CREATE CATALOG myhive WITH (
    'type' = 'hive',
    'default-database' = 'mydatabase',
    'hive-conf-dir' = '/opt/hive-conf'
);
-- set the HiveCatalog as the current catalog of the session
USE CATALOG myhive;

cc @Danny Chen

I am not seeing that happen, is that expected?

Yes, that is as expected. The Hive catalog is an external, persisted catalog. However, your connection to Hive itself is not persisted by Flink SQL. But you can fix that with an initialize file during your session, as outlined on https://nightlies.apache.org/flink/flink-docs-stable/docs/dev/table/sqlclient/#initialize-session-using-sql-files

Sorry, this is confusing - is this because Flink doesn't pull HMS data to populate context of the client?

Flink SQL doesn't store anything itself, so the connection to Hive isn't stored in Flink SQL

I see, but if in another session, I do a SHOW CATALOGS or a SHOW TABLES, that should be picked up by connecting to HMS, is that correct assumption?

You mean you close your session where you established the connection, then start Flink SQL again and then do a

SHOW CATALOGS

?

Also, in the link the SQL file, has CREATE TABLE statements, so with flink - that doesn't necessarily means creating a table?

Then no, that is not a correct assumption. That's because your new session doesn't have any information that it needs to connect to HMS, since that information isn't persisted

oh then how are tables resued in the Flink? isn't catalog.database.table a three part identifier?

I think you are mixing up the difference between persistence in HMS itself, and persistence of the connection from Flink SQL to HMS

Gotcha, I understand now, my confusion was probably coming from the three part naming conventionally used in HMS ( schema.database.table ), looks like catalog is just a connection string of sorts.

looks like catalog is just a connection string of sorts.

Mostly yes. This is how it's explained in the docs

Catalog enables users to reference existing metadata in their data systems, and automatically maps them to Flink’s corresponding metadata.

A tangential question: Usually in Flink deployments, do you see HiveCatalog setup by the persona writing the jobs or cluster owners? By setup I mean moving around all the libraries as mentioned in the docs.

That depends on the organizational setup. Mostly I see platform teams who operate Flink and provide the tools for platform teams that build and maintain Flink jobs

Thanks @Martijn Visser this was really helpful discussion!