@Rocky what date will vector indexes be available for shared data clusters for starrocks/cn-ubuntu docker images?

As of current documentation and release notes (including StarRocks v4.1, released in April 2026), vector indexes are not yet supported for shared-data clusters. They are currently only supported in shared-nothing clusters (v3.4 or later). Current Status and Roadmap * Availability: Vector index support is currently restricted to shared-nothing architectures. * Roadmap: "Vector index" for shared-data (Native Storage) is listed on the StarRocks 2026 Roadmap. While it is a high-priority item for the 2026 development cycle, a specific release version (e.g., v4.2 or v4.3) or exact date has not been officially confirmed. * Docker Images: The

starrocks/cn-ubuntu

images typically follow the main release cycle. Once the feature is merged into a stable or beta release for shared-data, it will be available in the corresponding Docker image tag. Summary of Supported Indexes (Shared-data) If you are using a shared-data cluster on v4.1 or later, you can currently use: * Built-in Inverted Index: Supported in shared-data clusters since v4.1 (Beta) for text filtering and full-text search. * Bitmap / Bloom Filter / Prefix Indexes: Standard support across both architectures. For the most up-to-date schedule, I recommend monitoring the StarRocks GitHub Roadmap or the official release notes. References • deployment/shared_data/feature-support-shared-data.md • table_design/indexes/vector_index.md • table_design/indexes/inverted_index.md • faq/shared_data_faq.md

@Rocky try analyzing recent pull requests related to supporting vector indexes on shared data clusters

Based on recent activity in the StarRocks GitHub repository and the official 2026 Roadmap, here is the current status of vector index support for shared-data clusters: Current Development Status Vector index support for shared-data (lake/cloud-native) mode is currently in active development and is a high-priority item for the 2026 roadmap. Several critical pull requests (PRs) have been merged or are under review as of May 2026: * Read Path Support: PR #72074 ("[Enhancement] Support vector index read in shared-data mode") was recently merged (approx. 3 weeks ago). This enables the Compute Nodes (CN) to read and process vector indexes stored in remote object storage. * Build & Compaction: PR #72926 ("[Enhancement] Compaction-aware async vector index build scheduling") is currently in progress. This addresses how indexes are built and maintained during data compaction in a shared-data environment. * Schema & Metadata: Work is also visible in PRs like #72128, which optimizes metadata lookups for tablets in shared-data mode, a prerequisite for efficient index management. Estimated Availability While StarRocks v4.1 (released April 2026) added the Built-in Inverted Index for shared-data, it did not yet include GA support for Vector Indexes. * Target Version: Given that "read support" has landed and "build support" is in progress, the feature is likely targeting StarRocks v4.2 (expected mid-2026) or a late-cycle v4.1 patch release. * Docker Images: The

starrocks/cn-ubuntu

images for the v4.2.0-rc or v4.2.0 tags will be the first to officially support this. If you need it sooner, it may appear in

dev

or

nightly

tags once the build-path PRs are merged. Summary of Limitations (As of v4.1) Until the full shared-data implementation is released: 1. Architecture: Vector indexes (HNSW/IVFPQ) remain restricted to shared-nothing (Local Storage) clusters. 2. Alternative: For shared-data clusters on v4.1, you can use the Inverted Index for text-based search, but it does not support vector similarity (ANN) yet. Recommendation: Keep an eye on the StarRocks v4.2 Release Plan or the specific PR #72926 to track when the "write" portion of the feature is finalized.

@Rocky what is the best approach to currently incorporate vector embedding searches with shared data clusters

While full vector indexing (HNSW/IVFPQ) is not yet supported for shared-data clusters (expected in v4.2), you can still perform vector embedding searches using the following "Brute Force + Optimization" approach. For many use cases, StarRocks is fast enough to handle brute-force vector scans on millions of rows without a specialized index, provided you optimize the execution environment. 1. Use Precise Distance Functions StarRocks provides built-in functions for calculating vector similarity that work regardless of the cluster architecture. * L2 Distance:

l2_distance(array1, array2)

* Cosine Similarity:

cosine_similarity(array1, array2)

Example Query:

sql
SELECT id, l2_distance([0.1, 0.2, 0.3], vector_column) as dist
FROM my_table
WHERE category = 'electronics' -- Pre-filter to reduce scan
ORDER BY dist ASC
LIMIT 10;

2. Enable and Tune the Data Cache In shared-data clusters, the bottleneck for brute-force vector search is often the latency of reading data from object storage (S3/GCS). * Recommendation: Ensure Data Cache is enabled on your Compute Nodes (CN). This stores frequently accessed vector data on local NVMe/SSD, allowing brute-force scans to run at near-local speeds. * Tip: If you have a specific "hot" dataset for vector search, ensure the CN nodes have enough disk space to keep the entire vector column in the cache. 3. Pre-Filtering (Hybrid Search) The most effective way to speed up search without a vector index is to reduce the number of rows being scanned. * Scalar Filtering: Use standard columns (like

user_id

,

timestamp

, or

category

) in your

WHERE

clause. * Inverted Indexes (v4.1+): Since shared-data clusters now support the Built-in Inverted Index, you can use it for high-performance text filtering. By filtering rows based on keywords before calculating vector distances, you can drastically reduce the compute load. 4. Manual "Coarse" Clustering (Advanced Workaround) If you have billions of rows and cannot wait for v4.2, you can implement a manual version of the IVF (Inverted File) logic: 1. Generate Clusters: Use an external tool (like Scikit-learn or Faiss) to pre-calculate $K$ cluster centroids for your embeddings. 2. Assign IDs: When loading data into StarRocks, include a

cluster_id

column for each row (the ID of the nearest centroid). 3. Two-Step Search: * Find the nearest centroids to your query vector. * Query StarRocks only for rows where

cluster_id

matches those nearest centroids. * This effectively simulates an IVF index, reducing a global scan to a targeted scan of specific clusters. Summary of Current Roadmap * v4.1 (Available now): Inverted Index support for shared-data (useful for pre-filtering). * v4.2 (Upcoming): Native HNSW and IVFPQ support for shared-data clusters, which will automate the acceleration you are-mentioned "Coarse Clustering" and Search" and provide much higher performance with less manual effort. References • table_design/indexes/vector_index.md • table_design/indexes/inverted_index.md

@Rocky explain the process of creating vector embedding tables in shared data cluster and supporting queries that join vector queries and typical lakehouse queries, and later once HNSW indexing is available in 4.2 give a migration plan for updating queries / tables

Currently, in StarRocks v4.1, vector indexes (HNSW/IVFPQ) are only supported in shared-nothing (local storage) clusters. For shared-data (lake/cloud-native) clusters, you can still store and query embeddings using brute-force calculations. Support for native HNSW/IVFPQ in shared-data mode is targeted for the v4.2 release (mid-2026). Below is the guide for current implementation and the transition plan for v4.2. ────────── Phase 1: Current Implementation (v4.1) In v4.1, you can store embeddings as

ARRAY<FLOAT>

and use precise distance functions. These queries are parallelized and accelerated by the StarRocks SIMD-vectorized execution engine. 1. Creating the Table When creating a table in a shared-data cluster, ensure you enable the Data Cache. This keeps embedding data on the local NVMe of your Compute Nodes (CN), drastically speeding up the brute-force scans.

sql
CREATE TABLE product_embeddings (
    product_id BIGINT NOT NULL,
    category STRING,
    embedding ARRAY<FLOAT> NOT NULL
) 
ENGINE=OLAP
DUPLICATE KEY(product_id)
DISTRIBUTED BY HASH(product_id)
PROPERTIES (
    "datacache.enable" = "true",        -- Crucial for performance in shared-data
    "datacache.partition_duration" = "1 MONTH"
);

2. Hybrid Lakehouse Queries (JOINs) StarRocks allows you to join local vector tables with typical lakehouse data (e.g., Iceberg/Hive) or other OLAP tables. Example: Join local embeddings with an Iceberg table to filter by user preferences

sql
-- Querying a local table and joining with an external Iceberg catalog
SELECT 
    p.product_id, 
    i.product_name,
    l2_distance(p.embedding, [0.1, 0.5, 0.8]) as dist
FROM product_embeddings p
JOIN iceberg_catalog.sales_db.product_metadata i ON p.product_id = i.id
WHERE i.status = 'active'             -- Filter on Lakehouse data
  AND p.category = 'electronics'      -- Filter on local metadata
ORDER BY dist ASC
LIMIT 10;

────────── Phase 2: Migration Plan for StarRocks v4.2 (HNSW) Once you upgrade to v4.2, follow this plan to move from brute-force scans to indexed Approximate Nearest Neighbor Search (ANNS). 1. Add the Index (Metadata Update) You won't need to recreate your table. You can add the index via an

ALTER

statement. This will trigger an asynchronous index build in the background.

sql
-- Migration Step: Add the HNSW index to your existing column
ALTER TABLE product_embeddings 
ADD INDEX idx_vector (embedding) USING VECTOR (
    "index_type" = "hnsw",
    "metric_type" = "l2_distance",
    "dim" = "1536"                    -- Dimension must match your embedding
);

-- Monitor build progress
SHOW ALTER TABLE COLUMN FROM product_embeddings;

2. Update Query Syntax To trigger the index, you must switch from precise functions to their "approximate" counterparts. The index will not be used unless you use the

approx_

functions and include a

LIMIT

clause. Current (v4.1) | Migrated (v4.2+)

l2_distance()

|

approx_l2_distance()

cosine_similarity()

|

approx_cosine_similarity()

Updated Query:

sql
SELECT product_id, approx_l2_distance(embedding, [...]) as dist
FROM product_embeddings
ORDER BY dist ASC
LIMIT 10;

3. Tuning for Production After migration, use session variables or hints to tune the recall (accuracy) vs. speed tradeoff. HNSW uses the

efsearch

parameter.

sql
SELECT /*+ SET_VAR(ann_params='{efsearch=128}') */
    product_id, 
    approx_l2_distance(embedding, [...]) as dist
FROM product_embeddings
ORDER BY dist ASC
LIMIT 10;

Summary of Differences * Storage: No change (both use

ARRAY<FLOAT>

). * Compute: v4.1 uses brute-force (fully parallelized); v4.2 uses the HNSW graph to skip data scans. * Infrastructure: For both versions, Ensure your

starrocks/cn-ubuntu

containers have sufficient

storage_root_path

disk space to cache the embeddings. References