Hi team :wave: We're running StarRocks 4.0.x with ...
# questions-and-troubleshooting
b
Hi team πŸ‘‹ We're running StarRocks 4.0.x with Apache Polaris (Iceberg REST Catalog) on AWS S3 using vended credentials, and hitting a BE-side STS token expiry bug after ~1 hour. Environment: - StarRocks: 4.0.1 (K8s) / 4.0.6 (local) - Catalog: Iceberg REST β†’ Apache Polaris - Storage: AWS S3 (vended STS credentials) - Iceberg: 1.10.0 Error (after ~1hr of inactivity):
Copy code
ERROR 1064 (HY000): BE access S3 file failed,
SdkResponseCode=400, SdkErrorType=100,
SdkErrorMessage=Unable to parse ExceptionName: ExpiredToken
Message: The provided token has expired.
file = <s3://atlan-vcluster-home-mt-1h16pa456xhq/atlan-wh/>
atlan-ns/Table/data/00001-1-....parquet
BE:10001
we have tried this config
Copy code
ALTER CATALOG mdlh_context_store SET (
  "iceberg_meta_cache_ttl_sec" = "1800"
);
This partially helps (forces metadata refresh every 30min) but doesn't reliably fix the credential expiry since they are cached at a different layer. please suggests.
c
Not sure if it helps but we noticed a regression with 4.0.6 not re-establishing Polaris credentials after token refresh failure. Moving back to 4.0.5 fixed this for us.
j
Yes, StarRocks is a strong fit for this use case, with one distinction worth clarifying. Streaming DML (inserts and updates every second): Fully supported. Use a Primary Key table with Routine Load from Kafka. The Primary Key model handles real-time upserts natively β€” row-level updates and inserts at high frequency are exactly what it's designed for. This is actually where StarRocks beats ClickHouse significantly, since ClickHouse's mutation model for updates is asynchronous and heavyweight. DDL changes at runtime (column additions/deletions from Kafka): StarRocks supports online DDL β€” you can add or drop columns on a live table without downtime. However, StarRocks won't automatically parse a Kafka message that says "add column X" and apply it. You'd need an application layer consumer that reads those DDL events from Kafka and executes the corresponding
ALTER TABLE
statements against StarRocks. That's not a limitation unique to StarRocks β€” no database natively consumes schema change instructions from a message queue without some middleware. Recommended architecture: CDC pipeline (Flink CDC or Kafka Connect) β†’ StarRocks Primary Key table via Routine Load or Flink StarRocks connector β€’ Separate consumer for DDL events β†’ executes
ALTER TABLE
on StarRocks This pattern is commonly used and well-documented. If you're already using Kafka, the migration from ClickHouse to StarRocks for this use case is straightforward.