@Rocky Is there a way to speed-up Bitmap index creation? It takes very long time.

Creating a Bitmap index in StarRocks is an asynchronous Schema Change process. It involves reading the existing data in the column, generating the bitmap data structures, and writing new versions of the data segments (rowsets). Here are the primary ways to speed up or manage the creation process: 1. Increase Schema Change Parallelism (BE Config) StarRocks uses a specific thread pool for schema change tasks (including index creation). You can increase the number of concurrent threads on each Backend (BE) node to process tablets faster. *

alter_tablet_worker_count

: This BE configuration defines the number of threads used for Schema Change. The default is typically 3. Increasing this (e.g., to 10 or more, depending on your CPU/Disk IO) allows more tablets to be processed in parallel.

bash
    # Example: Increase to 10 threads
    # This usually requires a BE restart or using the HTTP API for runtime updates
    # (Check if your version supports mutable config updates)

2. Increase Memory Limits (BE Config) Building bitmap indexes for columns with very high cardinality can be memory-intensive. If a task exceeds its memory limit, it may spill to disk or fail, slowing down the process. *

memory_limitation_per_thread_for_schema_change

: Controls the memory allocated to each schema change thread (default is 2 GB). If you have high-cardinality columns and sufficient RAM, consider increasing this to 4 GB or 8 GB. 3. Monitor Progress Since the process is asynchronous, you should monitor the state to see if it is progressing or stuck:

sql
SHOW ALTER TABLE COLUMN;

* Look at the State column (it should be

RUNNING

or

FINISHED

). * Check the Progress column to see the percentage completion of tablets. 4. Optimize Column Cardinality StarRocks bitmap indexes are most efficient for low-to-medium cardinality columns (e.g., gender, city, category). * High Cardinality: While StarRocks supports bitmap indexes on high-cardinality columns (unlike some other systems), building them is significantly slower because the resulting index size is much larger. * If your column has millions of unique values, consider whether a Bloom Filter index or the newer Inverted Index (available in v3.3+) might be more appropriate. 5. Best Practices & Limitations * One Task at a Time: StarRocks only allows one schema change task per table at a time. If you need to add multiple indexes, you must wait for the first to finish or define them all in a single

ALTER TABLE

statement if supported. * Tablet Size: If your tablets are very large (e.g., > 1GB compressed), the individual tasks for each tablet will take longer. Optimal tablet size is generally 100MB–1GB. * System Load: Creating indexes is an IO-intensive operation. Avoid running these tasks during peak data ingestion or heavy query periods, as they compete for the same disk IO and CPU resources. Note for 3.x Users: If you are using StarRocks 3.3 or later, evaluate the new Inverted Index (Lucene-based) for high-performance text or range or keyword filtering, as it is often more robust for complex filtering scenarios than the legacy Bitmap Index. References • table_design/indexes/Bitmap_index.md • best_practices/query_tuning/schema_tuning.md • knowledge/usage/best-practice.md