@Rocky can you please let me know how i can build my local environment on windows?

To build a local StarRocks environment on Windows, the most recommended and streamlined approach is using Docker (preferably with the WSL2 backend). StarRocks is a Linux-based system, so it does not run natively on Windows without a compatibility layer. Here are the two ways to set this up depending on your goal: Option 1: Quick Start (To Use/Test StarRocks) If you just want to run StarRocks locally to run queries and test features, use the "All-in-1" Docker image. 1. Install Docker Desktop: Ensure you have Docker Desktop installed and configured to use the WSL2 engine. 2. Resource Allocation: In Docker Desktop settings, ensure you have allocated at least 4 GB of RAM (8 GB is recommended). 3. Run the Container: Open PowerShell or Command Prompt and run:

bash
    docker pull starrocks/allin1-ubuntu
    docker run -p 9030:9030 -p 8030:8030 -p 8040:8040 -itd --name starrocks-local starrocks/allin1-ubuntu

4. Connect: You can connect using any MySQL-compatible client (like DBeaver or the MySQL CLI). If you have the MySQL client installed locally:

bash
    mysql -P 9030 -h 127.0.0.1 -u root

────────── Option 2: Development Environment (To Compile StarRocks) If you want to build StarRocks from source or contribute to the project, you should use the official development environment image. 1. Clone the Repository:

bash
    git clone <https://github.com/StarRocks/starrocks.git>
    cd starrocks

2. Use the Dev Container (Recommended for VS Code): • Install the Remote - Containers extension in VS Code. • Open the StarRocks folder. VS Code should detect the

.devcontainer

and ask to "Reopen in Container". • This will automatically pull the

starrocks/dev-env-ubuntu

image and set up all toolchains (GCC, CMake, JDK, etc.). 3. Manual Compilation via Docker: If you prefer the command line:

bash
    docker run -it -v <path_to_starrocks_code>:/root/starrocks starrocks/dev-env-ubuntu:latest
    cd /root/starrocks
    ./build.sh

Important Tips for Windows Users: * WSL2: Always use WSL2 instead of Hyper-V for better performance. * File System: If you are compiling from source, keep your code inside the WSL2 file system (e.g.,

\\wsl$\Ubuntu\home\...

) rather than the Windows mount (

/mnt/c/...

) to avoid extremely slow I/O and permission issues. * Memory: StarRocks is a high-performance database; if the container keeps crashing, check your Docker Desktop memory limit (it usually needs more than the default 2GB). Which path are you looking to take? I can provide more specific steps for either! References • quick_start/shared-nothing.md • developers/build-starrocks/Build_in_docker.md • developers/development-environment/dev-env-setup.md • quick_start/iceberg.md

I'm trying to setup my local dev environment. that means if i update the codebase, that should be reflected. what workflow is best for this? should i rebuild everytime? @Rocky

To set up a local development environment for StarRocks that reflects your changes efficiently, you definitely do not need to rebuild everything every time. The best workflow depends on whether you are working on the Frontend (FE), Backend (BE), or both. Here is the recommended approach: 1. Use the Right Build Command Instead of a full

./build.sh

, use targeted flags to only compile what you've changed: * Only Frontend (Java):

./build.sh --fe

* Only Backend (C++):

./build.sh --be

* Only Unit Tests:

./run-fe-ut.sh

or

./run-be-ut.sh

2. Recommended Development Setup Most developers use a Remote Development or Docker workflow because building the Backend (BE) is resource-intensive and requires a Linux environment. * For FE (Java): Use IntelliJ IDEA. You can run and debug FE directly on your Mac or Windows machine. * Run

mvn install -DskipTests

in the

fe

directory first to generate necessary Thrift/Protobuf classes. * Set up

STARROCKS_HOME

and other env variables in your IDEA Run Configuration. * For BE (C++): Use CLion with a Remote Toolchain. * Keep your code on your local machine and sync it to a high-performance Ubuntu server (16GB+ RAM recommended). * CLion handles the synchronization and allows you to use the server's compilers while coding locally. 3. Workflow for Fast Iteration (Incremental Builds) The build system is designed to be incremental. Here’s how to make it even faster: * Ccache: Ensure

ccache

is installed on your build machine. StarRocks' build scripts detect it automatically, which significantly reduces re-compilation time for BE. * Fast Linker: Linking is often the bottleneck for BE. You can use the

lld

linker to speed it up:

bash
    STARROCKS_LINKER=lld ./build.sh --be

* Docker-based Workflow: If you don't want to manage dependencies, use the provided Docker tools:

bash
    ./build-in-docker.sh --be  # Compiles BE inside a pre-configured container

4. Summary Workflow 1. First time: Run a full

./build.sh

(this will take a while as it builds all third-party dependencies). 2. Code Change (FE): Run

./build.sh --fe

(usually takes < 1 min). 3. Code Change (BE): Run

./build.sh --be

(thanks to

ccache

, only changed files and their dependents are recompiled). 4. Test: Run specific unit tests using the

--gtest_filter

(for BE) to avoid running the entire test suite. Pro-tip: If you are only changing Java code in FE, you can often just use the "Build Project" feature in IntelliJ IDEA and restart the FE process within the IDE, which takes only seconds. References • developers/build-starrocks/handbook.md • developers/build-starrocks/Build_in_docker.md • developers/development-environment/dev-env-setup.md • developers/build-starrocks/build_starrocks_on_ubuntu.md

should i use docker? how about using Remote Development? @Rocky

Choosing between Docker and Remote Development depends on whether you are looking to simply run/evaluate StarRocks or if you want to develop/contribute to the StarRocks codebase. 1. Using Docker Docker is the fastest way to get started if your goal is to use StarRocks as a database for testing or local development. * Quick Start: You can pull the

starrocks/allin1-ubuntu

image to run a full StarRocks cluster (FE and BE) in a single container.

bash
    docker run -p 9030:9030 -p 8030:8030 -p 8040:8040 -itd \
    --name quickstart starrocks/allin1-ubuntu

* Development Environment: StarRocks provides a specific

starrocks/dev-env-ubuntu

image that contains all the necessary dependencies (gcc, cmake, jdk, etc.) to compile the source code. * Best for: Trying out features, local testing, and maintaining a clean machine without installing many dependencies. 2. Remote Development Remote Development is highly recommended if you are building or contributing to StarRocks, especially for the Backend (BE) which is C++ based and resource-intensive. * Why use it? Compiling the StarRocks BE requires significant resources (at least 16GB RAM and multiple CPU cores). Many developers use a local laptop (like a MacBook) for coding and a powerful remote Linux server (Ubuntu 22.04) for actual compilation and execution. * IDE Integration: * VS Code: You can use the "Dev Containers" extension with the

starrocks/dev-env-ubuntu

image or use the "Remote - SSH" extension to work directly on a Linux server. * CLion/IntelliJ: These support "Remote Development" where the IDE runs locally, but the code is synchronized to and compiled on a remote server. * Debugging: * FE (Java): Supports standard remote JVM debugging (default port 5005). * BE (C++): Usually requires GDB on the remote server. Rocky's Recommendation: * If you just want to run queries: Use Docker. It’s isolated and takes minutes to set up. * If you want to modify the StarRocks source code: Use Remote Development. Set up a powerful Ubuntu server and connect to it via VS Code or CLion. This avoids draining your local laptop's battery and RAM during heavy C++ compilations. For more details on setting up your environment, you can check the StarRocks Developer Guide. References • developers/development-environment/ide-setup.md • developers/development-environment/IDEA.md • developers/development-environment/dev-env-setup.md

can i build my environment without those? @Rocky

It depends on what specific "those" refers to, but generally, StarRocks is quite flexible. Here are the most common "can I do this without..." scenarios for building a StarRocks environment: 1. Without multiple machines (Single-node) Yes! For development or testing, you can deploy both the FE (Frontend) and BE (Backend) on the same physical or virtual machine. This is often called a "pseudo-distributed" or "all-in-one" setup. 2. Without high-end hardware (Low specs) While production recommendations suggest 16+ cores and 64GB+ RAM: * CPU/RAM: You can run a small environment on as little as 2–4 cores and 8GB of RAM for testing, though performance will be limited. * Storage: You can use HDDs instead of SSDs. Note that if you use Primary Key tables with persistent indexes, SSDs are strongly recommended for performance. * Network: You can use 1GbE instead of the recommended 10GbE. 3. Without external storage (HDFS or S3) Yes. StarRocks supports a Shared-nothing architecture, where data is stored directly on the local disks of the BE nodes. You do not need HDFS, S3, or any external object storage to run a fully functional StarRocks cluster. 4. Without ZooKeeper Yes. Unlike some other distributed systems, StarRocks does not use or require ZooKeeper. It manages its own metadata and consistency using a built-in BDBJE (Berkeley DB Java Edition) protocol for FE nodes. 5. Without AVX2 instructions This is the one "hard" requirement for the standard x86 BE binaries. The BE service relies on AVX2 to power its vectorization engine. * If your CPU doesn't support AVX2 (common in very old hardware or some budget VMs), the BE will likely fail to start. _ _Note:* There are ARM64 builds available if you are on an ARM architecture (like Apple Silicon or AWS Graviton), which don't use AVX2. 6. Without Docker Yes. While Docker is the fastest way to get started, you can manually deploy StarRocks by downloading the binary

.tar.gz

and running the start scripts (

start_fe.sh

and

start_be.sh

) directly on your Linux host. *What you DO need:* * Linux: StarRocks must run on Linux (CentOS, Ubuntu, RedHat, etc.). * JDK: You need JDK 11 or 17 (depending on the StarRocks version) to run the FE. * AVX2 (for x86): Most modern CPUs from the last 10 years support this. Are you referring to a specific component or a hardware requirement mentioned earlier? Let me know so I can give you a more precise answer! References • faq/Deploy_faq.md • faq/Others.md • deployment/deployment_prerequisites.md • developers/build-starrocks/build_starrocks_on_ubuntu.md • deployment/plan_cluster.md • developers/mac-compile-run-test.md • developers/build-starrocks/Build_in_docker.md • deployment/prepare_deployment_files.md

I'm trying to setup my local dev environment with docker on Windows. that means if i update the codebase, that should be reflected. what workflow is best for this? should i rebuild every time? @Rocky

Setting up a local development environment for StarRocks on Windows using Docker is a great way to ensure consistency. To ensure code changes are reflected without rebuilding the entire Docker image every time, you should use a volume-mounted development container workflow. Here is the best workflow for this: 1. Recommended Environment: WSL2 For the best performance and compatibility on Windows, always run Docker within WSL2 (Windows Subsystem for Linux). * Clone the StarRocks repository directly into the WSL2 filesystem (e.g.,

~/starrocks

) rather than on the Windows C: drive (

/mnt/c/

). This significantly improves I/O performance for builds. 2. The Development Workflow (No Image Rebuilds) StarRocks provides a pre-built development environment image and helper scripts that mount your local code into the container. * Initial Setup: Navigate to your cloned StarRocks directory and use the provided scripts:

bash
    # Use the helper script to enter an interactive shell
    ./docker-dev.sh shell

This command starts a container based on

starrocks/dev-env-ubuntu

, mounting your local source code to

/workspace

inside the container. * Making Changes: You can edit the code using your favorite IDE on Windows (like VS Code or IntelliJ). Because your source directory is volume-mounted, any change you save on Windows is immediately visible inside the container. * Incremental Compiling (The "No Rebuild" Part): Instead of rebuilding the Docker image, you just run the build script inside the container's shell. The build system (Maven for FE, CMake for BE) is designed for incremental builds, so it will only recompile what you changed.

bash
    # Inside the docker-dev.sh shell:
    ./build.sh --fe  # Build only the Frontend
    ./build.sh --be  # Build only the Backend

The artifacts will be placed in the

output/

directory of your local repository. 3. Alternative: VS Code Dev Containers If you use VS Code, the most seamless workflow is using the Dev Containers extension. 1. Open the StarRocks folder in VS Code. 2. The repository includes configuration for Dev Containers. VS Code will prompt you to "Reopen in Container." 3. This sets up the environment automatically. You can then use the integrated terminal to run

./build.sh

commands. Summary of the Workflow: 1. Edit code on Windows/WSL. 2. Run

./docker-dev.sh shell

(keep this terminal open). 3. Compile inside the container using

./build.sh --fe

or

--be

. 4. Repeat step 1 & 3. Note on Maven/Cache: The

docker-dev.sh

script automatically mounts your Maven cache (

~/.m2

), so you don't have to re-download dependencies every time you start the container. Do I need to rebuild the image? Only if you need to change the underlying operating system dependencies or the toolchain version. For 99% of development tasks (fixing bugs, adding features), you never need to rebuild the Docker image. References • developers/development-environment/dev-env-setup.md • developers/build-starrocks/Build_in_docker.md

do you think ubuntu on vm is good for this project? which is better? @Rocky

Ubuntu 22.04 is one of the officially supported operating systems for StarRocks, and running it on a Virtual Machine (VM) is a common and fully supported deployment method. Here is a breakdown of why this setup works well and what you should consider when choosing between a VM and other options: 1. Is Ubuntu on a VM good? Yes, it is a solid choice, especially for development, testing, and cloud-native environments where flexibility and scaling are important. * Support: StarRocks provides binary distribution packages specifically for Ubuntu (v22.04). * Flexibility: VMs allow you to easily adjust resources (CPU/RAM) and take snapshots, which is great for managing a database cluster. 2. Which is "better"? (VM vs. Bare Metal) While VMs are convenient, the "better" choice depends on your specific use case: * Bare Metal (Physical Machines): Usually "better" for maximum performance in production. Since StarRocks is a high-performance OLAP engine, running it directly on hardware eliminates virtualization overhead. It also ensures consistent I/O performance, which is critical for large-scale data processing. * VMs: Better for ease of management and resource isolation. In modern cloud environments (like AWS, Azure, or private clouds), StarRocks performs very well on VMs as long as the underlying hardware is powerful enough. 3. Critical Considerations for Your VM Regardless of the OS, if you use a VM, ensure the following for the best experience: * Instruction Sets (AVX2): StarRocks relies on AVX2 for its vectorization engine. Ensure your VM hypervisor (like VMware, KVM, or Hyper-V) is configured to "pass through" the host CPU's instruction sets so the guest OS can see AVX2. You can check this by running:

bash
    cat /proc/cpuinfo | grep avx2

* Storage: Use SSD storage for the backend (BE) nodes. StarRocks' performance is heavily tied to disk I/O, and SSDs are highly recommended (and required for Primary Key tables with persistent indexes). * Resources: For a production-like environment, try to allocate at least 16 CPU cores and 64 GB of RAM for BE nodes, and 8 cores / 16 GB RAM for FE nodes. Summary Recommendation If you are just starting or building a cloud-based environment, Ubuntu 22.04 on a VM is excellent. If you are building a massive, performance-critical production cluster with tight latency requirements, Bare Metal would be the superior choice for raw power. References • deployment/plan_cluster.md • faq/Deploy_faq.md • deployment/deployment_prerequisites.md • deployment/deployment_overview.md