Hi all, I'm facing a problem when create a catalog...
# troubleshooting
l
Hi all, I'm facing a problem when create a catalog from Flink SQL client -> the catalog just exist in a local context, even when I'm using HiveMetastore (edited) So everytime I'm create a new Flink SQL session, I have to re-create a new catalog. This is my sample catalog in Flink Sql client:
Copy code
CREATE CATALOG iceberg_catalog WITH (
   'type' = 'iceberg',
   'catalog-type' = 'hive',
   'uri' = '<thrift://hivems.lakehouse.db-prod:9083>'
)
Is there anyway I could use only 1 catalog which persistent with HMS?
e
Recently, I had the chance to experiment with Flink catalogs. I recommend this article—it explains in simple terms how Flink works with catalogs and the trade-offs between different implementations. In the end, I went with the FileCatalogStore. If you take the same route, don’t forget to mount the catalog path as a volume in the container running the SQL Client, Job Manager, or Task Managers (assuming you’re running with Docker). As for the Hive Metastore, I tried multiple times but eventually gave up on it. Unless you specifically need to expose the catalog to external systems like Trino, I don’t see much benefit in going through the extra complexity of HMS. https://www.decodable.co/blog/catalogs-in-flink-sql-a-primer
🙏 1
👍 2
l
ah I think your point's correct. HMS do not store catalog definition. So I need to use FileCatalogStore or JdbcCatalogStore and store catalogs definition to MySQL/PostgreSQL
I'm using HMS because I want to store data with Parquet and Iceberg and HMS store databases/tables information.
With FileCatalogStore, data will be lost if the service goes down without a persistent volume. Using a PVC can prevent data loss, but I still prefer JdbcCatalog and storing catalog definitions in MySQL for better portability and reliability.
r