@Rocky I am getting large ingress into my VPC doing somewhat basic queries via StarRocks in AWS against Azure storage for Iceberg tables. What can be done to help reduce the ingress costs?

Large ingress costs when querying cross-cloud (Azure to AWS) are usually caused by StarRocks fetching data files and metadata from Azure for every query. To reduce these costs, you should implement caching and data-reduction strategies to ensure data is transferred only when necessary. 1. Enable Data Cache (Block Cache) This is the most effective way to reduce ingress. StarRocks can cache the data blocks fetched from Azure on the local NVMe/SSD disks of your BE (Backend) or CN (Compute Node) nodes in AWS. Subsequent queries will read from the local AWS cache instead of re-fetching from Azure. * How to enable: In your

be.conf

(or

cn.conf

), set:

properties
    # Enable the unified data cache
    datacache_enable = true
    # Specify the path for the cache (use high-speed SSDs/NVMe)
    storage_root_path = /path/to/fast/disk
    # Set the cache size (e.g., 100GB)
    datacache_disk_size = 107374182400

* Effect: Only the first query for a specific data block incurs the ingress cost. 2. Optimize Metadata Caching StarRocks caches Iceberg metadata (manifests, snapshots) to speed up planning. If your Iceberg tables have many manifest files, fetching them repeatedly from Azure adds to the cost. * Enable Disk Metadata Cache: On your FE (Frontend), enable disk-based metadata caching to persist this across FE restarts:

sql
    -- Set via ADMIN SET FRONTEND CONFIG
    ADMIN SET FRONTEND CONFIG ("enable_iceberg_metadata_disk_cache" = "true");
    ADMIN SET FRONTEND CONFIG ("iceberg_metadata_disk_cache_capacity" = "2147483648"); -- 2GB

* Catalog Properties: Ensure your Iceberg catalog has

enable_iceberg_metadata_cache

set to

true

(default in recent versions). 3. Use Asynchronous Materialized Views (MVs) If you have "hot" datasets or specific aggregations used frequently, create an asynchronous Materialized View inside StarRocks. * How it helps: The MV stores the data locally in StarRocks (on AWS EBS or S3). The ingress from Azure happens only once during the MV refresh (which can be incremental for Iceberg). * Automatic Rewriting: Users can still query the original Iceberg table; StarRocks will automatically rewrite the query to use the local MV if it matches. 4. Query & Storage Best Practices * Projection Pushdown: Avoid

SELECT *

. Because StarRocks and Iceberg use columnar formats (Parquet/ORC), selecting only the columns you need drastically reduces the amount of data transferred across the cloud boundary. * Predicate Pushdown: Ensure your queries use filters on partition columns. This allows StarRocks to prune files on the Azure side before downloading them. * Collect Statistics: Set

enable_get_stats_from_external_metadata = true

in your Iceberg catalog properties. This allows the StarRocks optimizer to make better pruning decisions by reading metadata statistics. * Compaction: If your Iceberg table has many small files, StarRocks must make many small HTTPS requests to Azure, which is slow and can increase overhead. Perform regular compaction on the Azure side to maintain larger (e.g., 128MB–512MB) Parquet files. 5. Check Network Path Ensure you are not accidentally routing traffic through a high-cost NAT Gateway. If possible, use a VPC Endpoint or verify if a dedicated connection (like AWS Direct Connect linked to Azure ExpressRoute) is applicable, though caching is generally the more cost-effective first step. References • using_starrocks/async_mv/use_cases/data_lake_query_acceleration_with_materialized_views.md • data_source/catalog/iceberg/iceberg_catalog.md