<@U0A71G31CDV> in `CachingIcebergCatalog.java` (St...
# questions-and-troubleshooting
r
@Rocky in
CachingIcebergCatalog.java
(StarRocks FE), we are caching the Iceberg catalog for tables but not for views. Are there technical limitations to not do so?
r
Based on the source code and the evolution of Iceberg support in StarRocks, here is why
CachingIcebergCatalog
currently caches tables but not views: 1. Implementation Priority and Feature Maturity Iceberg View support is a relatively recent addition to the StarRocks Iceberg connector (significantly expanded in versions 3.3 and 4.0). When
CachingIcebergCatalog
was initially designed, it focused on the most performance-critical metadata: Iceberg Tables. * Table metadata in Iceberg can be extremely large, consisting of a chain of snapshots, manifest lists, and thousands of manifest files. Repeatedly fetching and parsing this metadata from remote storage (S3/HDFS) or a metastore (Glue/Hive) causes significant latency. * View metadata is comparatively tiny. An Iceberg view is essentially a small JSON file containing the SQL definition and some properties. Fetching a view typically requires only a single metadata file read or one RPC call to the catalog, making the performance gain from caching less dramatic than for tables. 2. No Fundamental Technical Limitations There are no known technical limitations that strictly prevent views from being cached. Just like tables, Iceberg views are identified by a
TableIdentifier
and represented by a
View
object in the Iceberg SDK. * The
Table
cache in
CachingIcebergCatalog
uses a Guava
LoadingCache
to manage memory and expiration (TTL). * A similar
viewCache
could be implemented following the same pattern. The reason it is missing is likely because the performance bottleneck for views has not been as severe as for tables, and thus it hasn't been prioritized. 3. Current Behavior As you noted in the code: * Tables:
loadTable(TableIdentifier identifier)
calls
tableCache.get(identifier)
, which manages the lifecycle and refresh of the table object. * Views:
loadView(TableIdentifier identifier)
currently delegates directly to the underlying catalog (
delegate.asViewCatalog().loadView(identifier)
), meaning it fetches the view definition from the source every time it is needed during query planning. Summary The lack of view caching is likely an implementation gap rather than a deliberate technical restriction. As StarRocks continues to optimize its "Catalog-Centric" architecture (particularly for REST catalogs and Glue in version 4.1), view caching may be added to further reduce RPC overhead for complex workloads involving many views. For now, if you are experiencing latency specifically related to view resolution in high-concurrency environments, you might consider reaching out on the StarRocks GitHub to request this enhancement.