Hello I'm new to StarRocks - and had a couple questions about options for loading data to an Iceberg table with starrocks • I've succescfully ingested data into StarRocks OLAP tables (internal catalog) via post requests to the stream load endpoint. I haven't been able to figure out if there's a way to use this endpoint to ingest data into an external catalog's Iceberg table. is there a way to do this? • As an alternative, i've been able to POC the following ◦ ingest data via http (_stream_load) to OLAP table ◦ SUBMIT TASK - to INSERT OVERWRITE from OLAP table to iceberg table with Dynamic Overwrite ◦ While this works, I'm essentially duplicating data • INSERT FROM FILES() only supports parquet at the moment. Without having to introduce additional logic on my end, if there's a way to load JSON files that'd be ideal ◦ I've looked at Broker Load What other options might i have? Currently I can send JSON data to S3 or to an HTTP endpoint, and would like to get it into an iceberg table

Root cause: The

iceberg_meta_cache_ttl_sec

fix is at the wrong layer. That controls metadata refresh. The STS credentials are cached separately at the BE file client level, so you can force metadata to refresh every 30min but the BE still holds stale credentials until it gets the 400. Fixes in order of simplicity: If you're on 4.0.6, downgrade to 4.0.5. Cliff's finding around Polaris credential re-establishment lines up with what I'd expect from a regression there. Easiest path until a patch lands. For 4.0.1, or while waiting on a fix, set the credential TTL below the STS expiry at the catalog level:

ALTER CATALOG mdlh_context_store SET (
  "iceberg_meta_cache_ttl_sec" = "1800",
  "credential_cache_ttl_sec" = "1800"
);

The goal is to force a re-fetch well before the 1hr STS expiry hits. On the Polaris side, you can also increase the vended STS token duration. AWS allows up to 12 hours for role assumption, so if Polaris is issuing 1hr tokens, bumping that buys you headroom while the caching issue gets resolved. If a GitHub issue doesn't already exist for the 4.0.6 regression, worth filing one. Cliff's repro is a solid data point. The metadata TTL config isn't wrong to have but it's not solving the actual problem here.

Thanks for getting back to me John! Sorry if my question wasn't clear, I think essentially I'm asking if I can: • make http

/api/{db}/{table}/_stream_load

but somehow specify external catalog? • if now, what other option might i have for regularly bringing in json data into an iceberg table with starrocks

No options. It should work with StarRocks internal tables.

Sorry Raghav, I answered the wrong question here!