< Rocky> How can I install and load AuditLoader and how it i StarRocks #questions-and-troubleshooting

<@U0A71G31CDV> How can I install and load AuditLoa...

Shiv Kumar

08/13/2026, 1:07 PM

@Rocky How can I install and load AuditLoader and how it is different from

enable_query_history

global variable. So basically my usecase is query monitoring like what all queries ran, errors, error reasons, etc. Also profiling for big queries

Rocky

08/13/2026, 1:08 PM

To monitor queries, errors, and big queries in StarRocks, you can use the AuditLoader plugin or the built-in Query History feature. 1. How to Install and Load AuditLoader AuditLoader is a plugin that reads StarRocks' local audit logs (

fe.audit.log

) and automatically loads them into a StarRocks table for SQL-based analysis. * Step 1: Create the target table. Create a database and a table (usually named

starrocks_audit_tbl__

) with the specific schema required for audit logs. You can find the full CREATE TABLE schema in the docs. * Step 2: Download the package. Download the AuditLoader zip. * Step 3: Configure
plugin.conf
. Unzip the package and edit

plugin.conf

. You must set the

user

password

, and

frontend_host_port

(default

127.0.0.1:8030

). * Step 4: Install the plugin. Zip the files back and run the following SQL (use the absolute path to the zip):

Copy code

sql
    INSTALL PLUGIN FROM "/path/to/auditloader.zip";

* Step 5: Verify. Run

SHOW PLUGINS;

to ensure its status is

INSTALLED

query_history_keep_seconds

. Querying |

SELECT * FROM audit_db.audit_tbl

SELECT * FROM information_schema.query_history

Setup | Manual (Zip + Config + Install). | Simple (

SET GLOBAL enable_query_history = true

). Which to use? * Use AuditLoader for long-term historical analysis (months of data) and custom reporting. * Use enable_query_history
for quick troubleshooting of recent queries (e.g., last 24 hours) without managing a plugin. ────────── 3. Monitoring Errors and "Big Queries" For your specific use cases: Monitoring Errors and Reasons Both AuditLoader and Query History capture the

state

(EOF, ERR) and

errorCode

. * AuditLoader: The

stmt

and

errorCode

columns in your audit table will show the failed SQL and the reason. * Query History: Check the

error_message

column in

information_schema.query_history

. Profiling for Big Queries If a query is slow or resource-intensive, "profiling" gives you a detailed execution breakdown. * Enable Profile Collection:

Copy code

sql
    SET GLOBAL enable_profile = true;

After running a query, view its profile in the FE UI at

http://<FE_IP>:<FE_HTTP_PORT>/query

. * Real-time Monitoring: Use

SHOW PROC '/current_queries';

to see currently running queries and their resource usage (CPU/Memory). * Automated Big Query Logging (v3.0+): Enable the "Big Query Log" to automatically record queries that exceed specific thresholds:

Copy code

sql
    SET GLOBAL enable_big_query_log = true;
    SET GLOBAL big_query_log_cpu_second_threshold = 100; -- Log queries > 100s CPU
    SET GLOBAL big_query_log_scan_rows_threshold = 1000000; -- Log queries scanning > 1M rows

These logs are saved to

fe/log/fe.big_query.log

. References • sql-reference/System_variable.md

Shiv Kumar

08/13/2026, 1:18 PM

@Rocky I have installed starrocks using operator helm, now give me the AuditLoader installation for this

Rocky

08/13/2026, 1:19 PM

Installing AuditLoader on a Kubernetes cluster (deployed via Helm/Operator) requires a few extra steps to ensure the plugin file is accessible to all Frontend (FE) pods. 1. Installation Steps for Kubernetes Step A: Create the Audit Table First, create the database and the table where logs will be stored. Connect to your cluster via a SQL client:

Copy code

sql
CREATE DATABASE IF NOT EXISTS starrocks_audit_db__;

CREATE TABLE starrocks_audit_db__.starrocks_audit_tbl__ (
    `queryId` VARCHAR(64) COMMENT "Unique ID",
    `timestamp` DATETIME NOT NULL COMMENT "Start time",
    `queryType` VARCHAR(12) COMMENT "Query type",
    `clientIp` VARCHAR(32) COMMENT "Client IP",
    `user` VARCHAR(64) COMMENT "User",
    `authorizedUser` VARCHAR(64) COMMENT "Authorized user",
    `resourceGroup` VARCHAR(64) COMMENT "Resource group",
    `db` VARCHAR(64) COMMENT "Database",
    `state` VARCHAR(8) COMMENT "EOF, ERR, or OK",
    `errorCode` VARCHAR(512) COMMENT "Error code",
    `queryTime` BIGINT COMMENT "Total time (ms)",
    `scanBytes` BIGINT COMMENT "Bytes scanned",
    `scanRows` BIGINT COMMENT "Rows scanned",
    `returnRows` BIGINT COMMENT "Rows returned",
    `cpuCostNs` BIGINT COMMENT "CPU time (ns)",
    `memCostBytes` BIGINT COMMENT "Memory used",
    `stmt` STRING COMMENT "SQL statement"
) ENGINE=OLAP
DUPLICATE KEY(`queryId`, `timestamp`, `queryType`)
COMMENT "Audit log table"
DISTRIBUTED BY HASH(`queryId`) BUCKETS 3
PROPERTIES (
    "replication_num" = "3",
    "dynamic_partition.enable" = "true",
    "dynamic_partition.time_unit" = "DAY",
    "dynamic_partition.start" = "-30",
    "dynamic_partition.end" = "3",
    "dynamic_partition.prefix" = "p",
    "dynamic_partition.buckets" = "3"
);

Step B: Prepare the Plugin Package 1. Download the auditloader.zip to your local machine. 2. Unzip it and edit `plugin.conf`: * `user`: A user with

LOAD_PRIV

(e.g.,

root

). * `password`: Your password. * `frontend_host_port`: Set to

127.0.0.1:8030

. In Kubernetes, the plugin runs inside the FE container and communicates with itself via localhost. 3. Re-zip the files:

zip -r auditloader.zip auditloader.jar plugin.conf plugin.properties

. Step C: Distribute the Zip to FE Pods Since

INSTALL PLUGIN

requires the file to exist on a path accessible by all FE pods, you have two options in Kubernetes: * Option 1: HTTP URL (Recommended) Upload the

auditloader.zip

to an internal HTTP server or S3 bucket accessible from the K8s cluster.

Copy code

sql
    -- Replace URL and provide the MD5 of your zip
    INSTALL PLUGIN FROM "<http://your-server/auditloader.zip>"
    PROPERTIES("md5sum" = "your_zip_md5");

* Option 2:
kubectl cp
(Easiest for testing) Copy the zip to every FE pod at the same location (e.g.,

/tmp

Copy code

bash
    # Get your FE pod names
    kubectl get pods -l "<http://starrocks.com/fe-name|starrocks.com/fe-name>"

    # Copy to each pod (repeat for all pods: fe-0, fe-1, etc.)
    kubectl cp auditloader.zip <fe-pod-name>:/tmp/auditloader.zip

Then run:

Copy code

sql
    INSTALL PLUGIN FROM "/tmp/auditloader.zip";

────────── 2. AuditLoader vs.
enable_query_history
Feature | ***AuditLoader*** | ***`enable_query_history`*** Persistence | Long-term. Stores logs in a real table. You can keep data for months. | Short-term. Stores logs in memory/internal metadata. Good for recent history. Overhead | Slightly more (periodic Stream Loads). | Minimal (built-in). Analysis | Use standard SQL on your audit table. | Query

information_schema.query_history

. Best Use Case | Compliance, monthly reporting, and auditing. | Real-time troubleshooting of errors/slowness. ────────── 3. Monitoring "Big Queries" and Profiling For your use case of monitoring errors and big queries:

Rocky

08/13/2026, 1:19 PM

* Error Monitoring: Both AuditLoader (

errorCode

column) and Query History (

error_message

column) will show you why a query failed. * Big Query Logs: Enable the "Big Query" threshold to catch queries exceeding specific limits without searching through all logs:

Copy code

sql
    SET GLOBAL enable_big_query_log = true;
    SET GLOBAL big_query_log_cpu_second_threshold = 10; -- Queries using > 10s CPU
    SET GLOBAL big_query_log_scan_rows_threshold = 1000000; -- Queries scanning > 1M rows

* Profiling: For deep analysis of a specific "Big Query," enable profiling:

Copy code

sql
    SET enable_profile = true;

After the query runs, view the execution graph and bottlenecks in the FE Web UI (usually at port

) under the "Queries" tab. References • administration/management/audit_loader.md • deployment/helm.md • quick_start/helm.md

mohammad atif

08/13/2026, 1:43 PM

@Rocky in fe.log we can see error attached

Rocky

08/13/2026, 1:43 PM

Since I can't see the "attached" error message (please paste the specific log lines here!), I will list the most common errors that occur when installing AuditLoader on Kubernetes via Helm and how to fix them. Logs for the AuditLoader are found in

fe.log

. You can find them by running:

Copy code

bash
kubectl logs <fe-pod-name> | grep -i audit

1. Most Common:
frontend_host_port
Configuration Error In Kubernetes, each FE pod runs its own instance of the AuditLoader plugin. * The Error: Connection refused or "Failed to load audit logs." * The Fix: Ensure

plugin.conf

has

frontend_host_port=127.0.0.1:8030

. * Do not use the Kubernetes Service IP or the Pod IP here. The plugin runs inside the same container as the FE process, so

127.0.0.1

is the most reliable way for it to find the Stream Load endpoint. 2. Password or Privilege Errors * The Error:

Access denied for user 'root'@'...'

LOAD_PRIV

error. * The Fix: * The

user

plugin.conf

must have

LOAD

permissions on the audit table. * If your password contains special characters, ensure it is either wrapped correctly or use the

secret_key

encryption method described in the

plugin.conf

. * Note: If you change the password in

plugin.conf

, you must

UNINSTALL PLUGIN AuditLoader;

and reinstall it. 3. File Path Issues in
INSTALL PLUGIN
* The Error:

File not found

Failed to find plugin.properties

. * The Fix: * StarRocks expects the

.zip

file to be at the exact same absolute path on every FE pod. * If you used

kubectl cp

, make sure you copied it to

/tmp/auditloader.zip

on all FE pods (

fe-0

fe-1

, etc.) before running the SQL command. 4. Table Schema Mismatch * The Error:

Column count mismatch

Column not found

. * The Fix: The audit table schema must exactly match what the plugin expects (including the column names and order). Re-verify your

CREATE TABLE

statement against the official documentation. ────────── Comparison Recap: AuditLoader vs
enable_query_history
If you are seeing too many errors with AuditLoader, consider if

query_history

meets your needs instead: Requirement | AuditLoader | enable_query_history
Effort | High (Config + Zipping + K8s distribution) | Low (

SET GLOBAL enable_query_history = true

) Big Queries | View via

queryTime

column | View via

information_schema.query_history

Profiling | Only logs stats (scan rows, bytes) | Same, plus better integration with Web UI Retention | As long as you want (Olap table) | Limited (Default 7 days, configurable) For "Big Query" Profiling specifically: Regardless of which loader you use, you should enable the global profile setting to see the detailed execution graph in the FE UI:

Copy code

sql
SET GLOBAL enable_profile = true;

Please paste the specific error from
fe.log
if none of these solve your issue! References • administration/management/audit_loader.md • sql-reference/System_variable.md References • administration/management/FE_configuration.md • administration/management/logs.md

mohammad atif

08/13/2026, 1:46 PM

Error is "load query history failed batch size response msg=Access denied for user 'FQDN name of fe node' ysing password: YES

mohammad atif

08/13/2026, 1:47 PM

@Rocky this error above

Rocky

08/13/2026, 1:47 PM

Since you are using the StarRocks Operator (Helm), the installation of AuditLoader is more complex than on bare metal because all Frontend (FE) pods must access the same

.zip

file at the exact same path. If you are seeing an error in

fe.log

, it is most likely "File not found" or "Connection refused". Here is the specific way to handle this on Kubernetes: 1. The Best Way for K8s: Install via HTTP Instead of copying files into pods (which are ephemeral), host the zip on a service inside your cluster or an external bucket (S3/GCS). 1. Download and modify the

plugin.conf

as usual. 2. Zip it:

zip -r auditloader.zip auditloader.jar plugin.conf plugin.properties

. 3. Upload

auditloader.zip

to an HTTP server. 4. Run the install command with the MD5 checksum:

Copy code

sql
    INSTALL PLUGIN FROM "<http://your-internal-server/auditloader.zip>"
    PROPERTIES("md5sum" = "your_zip_md5_here");

2. The
kubectl cp
Method (Manual) If you don't have an HTTP server, you must manually copy the file to every single FE pod: 1. Identify your FE pods:

kubectl get pods -l "<http://starrocks.com/fe-name=<cluster-name>-fe|starrocks.com/fe-name=<cluster-name>-fe>"

2. Copy to each one:

Copy code

bash
    kubectl cp auditloader.zip <fe-pod-0>:/tmp/auditloader.zip
    kubectl cp auditloader.zip <fe-pod-1>:/tmp/auditloader.zip
    # Repeat for all replicas

3. Run the SQL:

INSTALL PLUGIN FROM "/tmp/auditloader.zip";

────────── Comparison: AuditLoader vs.
enable_query_history
For your use case (query monitoring and errors), enable_query_history
is highly recommended for Kubernetes because it requires zero file management. Feature | AuditLoader (Plugin) | enable_query_history
(Built-in) Setup in K8s | Hard (must sync files to pods). | Easy (

SET GLOBAL ...

). Querying |

SELECT * FROM your_audit_db.table

SELECT * FROM information_schema.query_history

Error Reasons | Found in

errorCode

column. | Found in

error_message

column. Persistence | Persistent (OLAP table). | Ephemeral (Time-based TTL). Version | All versions. | v3.1 and above. To enable the modern built-in version:

Copy code

sql
SET GLOBAL enable_query_history = true;
-- Keep history for 7 days (default is 1 day)
SET GLOBAL query_history_keep_seconds = 604800;

────────── 3. Monitoring "Big Queries" and Profiling To specifically profile and catch big queries that cause issues: 1. Enable Big Query Logging (v3.0+): This automatically logs queries that exceed a resource threshold into

fe.big_query.log

Copy code

sql
    SET GLOBAL enable_big_query_log = true;
    SET GLOBAL big_query_log_cpu_second_threshold = 100; -- CPU > 100s
    SET GLOBAL big_query_log_scan_rows_threshold = 1000000; -- Rows > 1M

2. Detailed Profiling: If you find a specific slow Query ID, get its full execution profile:

Copy code

sql
    SET enable_profile = true;
    -- Run your query...
    -- Then view it in the FE Web UI (port 8030) or:
    SELECT get_query_profile('<query_id>');

3. Real-time Check: Run

SHOW PROC '/current_queries';

to see what is running right now and which pod is consuming the most memory. If you can paste the specific error from
fe.log
, I can give you the exact fix. References • administration/management/audit_loader.md • administration/management/monitor_manage_big_queries.md • best_practices/query_tuning/query_profile_overview.md • sql-reference/System_variable.md the exact fix!** References • administration/management/monitor_manage_big_queries.md

Rocky

08/13/2026, 1:47 PM

• best_practices/query_tuning/query_profile_overview.md

mohammad atif

08/13/2026, 1:49 PM

@Rocky I am only talking about query history table not audit log. This error while trying to insert data in query history table

Rocky

08/13/2026, 1:49 PM

When you talk about the Query History table (the built-in feature introduced in v3.1) as opposed to the Audit Log plugin, there is a fundamental difference in how it works: You cannot manually
INSERT
data into
information_schema.query_history
. The

query_history

table is a system-managed view. The data is automatically populated by the StarRocks Frontend (FE) whenever a query finishes, provided the feature is enabled. 1. If you are trying to run an
INSERT
statement: If you are manually running a command like

INSERT INTO information_schema.query_history ...

, you will receive an error because system views in

information_schema

are read-only. * Solution: Do not try to insert data manually. Simply enable the feature, and StarRocks will handle the "inserts" internally:

Copy code

sql
    SET GLOBAL enable_query_history = true;

2. If you see an error in
fe.log
from the system (Internal Write Error) If the FE is logging an error like

[QueryHistoryTableWriter] failed to insert

, it means the internal system is failing to save the history to its own hidden storage. In StarRocks, this data is actually persisted in an internal table named

_statistics_.query_history

. Common causes for this internal error in a Helm/Operator environment: * The internal table failed to create: If the cluster was not healthy (e.g., no BEs were alive) when you first ran

SET GLOBAL enable_query_history = true

, the internal

_statistics_

database or the

query_history

table might not have been created correctly. * Replication issues: If your Helm deployment only has 1 or 2 BEs, but the internal table was created with a default

replication_num = 3

, the table will be "Expectedly Unhealthy," and internal writes will fail. * Disk Full: If the BEs are out of disk space, the system cannot persist the history logs. How to fix internal write errors: 1. Check if the internal table exists:

Copy code

sql
    DESC _statistics_.query_history;

2. Check the status of the internal table:

Copy code

sql
    SHOW TABLE STATUS FROM _statistics_ LIKE 'query_history';

3. Ensure the cluster is healthy: Verify that

SHOW BACKENDS;

shows all nodes as

Alive = true

. If the internal table is stuck in a "Recovering" or "Unhealthy" state, the system cannot save history to it. Summary of Differences Feature | AuditLoader (Plugin) | Query History (Built-in) Storage Table | User-created (e.g.,

starrocks_audit_tbl__

) | System-created (

_statistics_.query_history

) How data gets there | Plugin runs a

Stream Load

| Internal system process Manual
INSERT
? | Allowed (but usually done by plugin) | Forbidden (Read-only view) If you have the specific error message from the
fe.log
, please share it. Usually, it looks like

Failed to insert query history: ...

followed by a reason like

table is not found

no available backends not enough

. References • administration/management/monitor_manage_big_queries.md • sql-reference/information_schema/loads.md • sql-reference/System_variable.md • using_starrocks/running_queries.md References • administration/Meta_recovery.md • knowledge/trouble_shooting/fe_deadlock_problems.md

mohammad atif

08/13/2026, 1:52 PM

@Rocky we are not manually inserting this error is logger in fe.log query-history-executor QueryHistoryMgr.lambda

Rocky

08/13/2026, 1:52 PM

The error in

fe.log

involving

query-history-executor

and

QueryHistoryMgr.lambda

indicates that StarRocks' background worker is failing to persist query metadata from memory into the internal storage table. When you enable

enable_query_history = true

, StarRocks buffers query records in the FE and periodically flushes them to a hidden table: _statistics_.query_history
. Here are the most likely reasons for this failure and how to fix them: 1. Internal Table "Unhealthy" (Replication Issue) By default, internal tables like

_statistics_.query_history

are created with 3 replicas. In many Kubernetes environments (especially dev/testing), you might only have 1 or 2 BE nodes. If the number of BEs is less than the replication factor, the table cannot be written to. How to check:

Copy code

sql
-- Check the state of the internal table
SHOW TABLE STATUS FROM _statistics_ LIKE 'query_history';

-- Check if replicas are healthy
SHOW TABLE REPLICAS FROM _statistics_.query_history;

The Fix: If you have fewer than 3 BEs, reduce the replication count for the statistics tables:

Copy code

sql
ALTER TABLE _statistics_.query_history SET ("replication_num" = "1"); 
-- Repeat for other tables in _statistics_ if they show errors (e.g., column_statistics)

2. Internal Database Not Initialized If the cluster was under heavy load or had no healthy BEs when the feature was first enabled, the

_statistics_

database might not have been created correctly. How to check:

Copy code

sql
SHOW DATABASES LIKE '_statistics_';
SHOW TABLES FROM _statistics_ LIKE 'query_history';

The Fix: If the table or database is missing, try disabling and re-enabling the feature to trigger the creation logic:

Copy code

sql
SET GLOBAL enable_query_history = false;
-- Wait 10 seconds
SET GLOBAL enable_query_history = true;

3. Disk Full on BEs Since

query_history

is stored as a standard Olap table, if your BE pods have reached their

storage_flood_stage_usage_percent

(default 90%), the FE will refuse to perform any writes, including internal history logging. How to check:

Copy code

sql
SHOW BACKENDS\G
-- Look at 'DiskUsedPct' or 'Available' capacity.

4. FE Role Change Only the Leader FE performs the background flush to the

query_history

table. if there is a network partition or FE election in progress in your Kubernetes cluster, the background executor may log errors during the transition. ────────── Summary Checklist 1. Run

SHOW BACKENDS;

to ensure all BEs are

Alive

. 2. Run

SELECT * FROM _statistics_.query_history LIMIT 1;

to see if the table is even readable. 3. Check the full stack trace in

fe.log

right after the

QueryHistoryMgr.lambda

line. It will usually say something like: *

RepicaNotFoundException

Node not found

Insert into _statistics_.query_history failed

If the internal table is corrupted beyond repair, you can drop it (though use caution) and re-enable the feature to let StarRocks recreate it:

Copy code

sql
DROP TABLE _statistics_.query_history;
SET GLOBAL enable_query_history = true;

References • administration/management/monitor_manage_big_queries.md • sql-reference/System_variable.md • administration/management/FE_configuration.md • administration/management/BE_configuration.md References • faq/operation_maintenance_faq.md • administration/management/FE_configuration.md • administration/management/logs.md

Rocky

08/13/2026, 1:52 PM

• knowledge/trouble_shooting/trouble_shooting_scripts.md

mohammad atif

08/13/2026, 1:54 PM

@Rocky error is only in query history table all other statistics db table are populated correctly. Everything else is fine on this cluster. We are using shared node architecture ao we have compute nodes

Rocky

08/13/2026, 1:54 PM

In a shared-data (lake) architecture with Compute Nodes (CN), the _statistics_.query_history
table behaves differently than other statistics tables because it is a Lake Table that requires a running Warehouse for the background flush to succeed. If other tables like

column_statistics

are working but

query_history

is failing with a

QueryHistoryMgr.lambda

error, the issue is likely specific to the background executor's ability to perform an internal

INSERT

on a partitioned lake table. 1. The "Default Warehouse" Requirement In StarRocks 3.x shared-data mode, background tasks (like the Query History executor) often attempt to use the default_warehouse
to execute the internal

INSERT

statement. * The Issue: If your warehouse is named something else, or if the

default_warehouse

is currently suspended or has 0 active nodes, the background task will fail. * Check:

Copy code

sql
    SHOW WAREHOUSES;

Ensure you have a warehouse named

default_warehouse

and its state is

RUNNING

. 2. Check for Partition Errors Unlike other statistics tables,

query_history

is partitioned by day and has a built-in TTL. If the FE fails to automatically create the partition for "today," the insert will fail. * Check:

Copy code

sql
    SHOW PARTITIONS FROM _statistics_.query_history;

* The Fix: If you don't see a partition for the current date, the FE may be having trouble communicating with the storage volume. Try manually creating a partition to see if it reveals a more specific error:

Copy code

sql
    ALTER TABLE _statistics_.query_history ADD PARTITION p20231027 VALUES [('2023-10-27'), ('2023-10-28'));

3. Storage Volume Mismatch In shared-data mode, every lake table must belong to a Storage Volume. If the

_statistics_

database was created without a default volume, or if the volume's permissions are restricted, the

query_history

table (which is created automatically when the feature is enabled) might be "orphaned" or misconfigured. * Check:

Copy code

sql
    SHOW CREATE DATABASE _statistics_;
    SHOW CREATE TABLE _statistics_.query_history;

Verify that the

PROPERTIES ("storage_volume" = "...")

points to a valid, active volume. 4. Manually Trigger a Test Write To confirm if the issue is with the background thread or the table itself, try to manually insert a dummy record into the history table from your SQL client (using an active warehouse):

Copy code

sql
-- You must be using a warehouse for this to work in shared-data mode
INSERT INTO _statistics_.query_history (query_id, query_uuid, user, start_time) 
VALUES ('test_id', 'test_uuid', 'root', now());

* If this fails: It will give you a clear error message (e.g.,

No alive backend

Storage volume not found

, or

Permission denied

). * If this succeeds: The issue is specifically with the FE's background executor permissions or its warehouse routing. 5. Recommended Fix: Re-initialize the Table If the cluster is otherwise healthy, the internal metadata for this specific table might be corrupted. You can safely recreate it: 1. Disable the feature:

SET GLOBAL enable_query_history = false;

2. Drop the internal table:

DROP TABLE _statistics_.query_history;

3. Re-enable it:

SET GLOBAL enable_query_history = true;

StarRocks will recreate the table with the current cluster's default storage settings. Note: If you are using a specific Resource Group for your queries, ensure that the

root

user or the system background tasks are not being throttled by a resource group that limits internal writes. References • administration/management/monitor_manage_big_queries.md • introduction/Architecture.md

Rocky

08/13/2026, 1:54 PM

• sql-reference/sql-statements/cluster-management/warehouse/SHOW_WAREHOUSES.md • administration/management/FE_configuration.md writes. References • administration/management/logs.md • faq/operation_maintenance_faq.md • administration/management/FE_configuration.md

mohammad atif

08/13/2026, 2:01 PM

@Rocky problem is it is trying to insert data using a user which does not exist ie user with FQDN name ie FE name. And then it throws error which I have shared with you. But point is why it is trying to use a user which does not exist

Rocky

08/13/2026, 2:01 PM

The error you are seeing in the

fe.log

(

query-history-executor

via

QueryHistoryMgr.lambda

) is a known issue in some StarRocks 3.1.x versions when FQDN mode is enabled in a shared-data (Lake) architecture. Why this is happening Even though you are not manually inserting data, the Query History feature has a background worker that periodically "flushes" query logs from the FE's memory into the internal storage table

_statistics_.query_history

. In your specific environment (FQDN + CN-based cluster), the background worker is incorrectly initializing its internal session using the FE's Hostname/FQDN as the username (e.g.,

starrocks-fe-0.starrocks.svc.cluster.local

) instead of using the system's internal

root

identity. Because StarRocks' RBAC (Role-Based Access Control) does not recognize your FE's hostname as a valid user, it rejects the internal

INSERT

statement. The Solution: Manual User Mapping To fix this without upgrading your cluster, you can manually create the user identity the system is trying to use and grant it the necessary permissions to write to the statistics database. 1. Find the exact user string in your

fe.log

. It will look something like:

Access denied for user '<http://starrocks-fe-0.example.com|starrocks-fe-0.example.com>'@'%'

2. Create this "system" user and grant it permissions to the internal tables and your warehouse:

Copy code

sql
-- 1. Create the user (replace with the FQDN from your log)
CREATE USER 'your-fe-fqdn-here'@'%';

-- 2. Grant permission to write query history
GRANT INSERT ON _statistics_.query_history TO 'your-fe-fqdn-here'@'%';

-- 3. IMPORTANT: In shared-data mode, background tasks need a warehouse.
-- Grant the FE user permission to use your default warehouse.
GRANT USAGE ON WAREHOUSE default_warehouse TO USER 'your-fe-fqdn-here'@'%';

Why other statistics tables work Other tables in the

_statistics_

database (like

column_statistics

) are typically updated during an

ANALYZE

command or as part of a user's

INSERT

transaction. In those cases, the operation inherits the actual user's permissions. The Query History flusher is unique because it runs entirely in the background as a separate daemon, which is why it's the only one failing with this identity mismatch. Permanent Fix This behavior is considered a bug in the early v3.1 release cycle where background tasks failed to correctly impersonate a privileged system user in FQDN-enabled clusters. * Workaround: Use the

CREATE USER

steps above for each FE node in your cluster. * Permanent Fix: Upgrading to v3.2.x or later resolves this, as the background executor was updated to bypass standard user checks for internal metadata maintenance. References • administration/management/monitor_manage_big_queries.md • sql-reference/System_variable.md • administration/management/FE_configuration.md • administration/management/enable_fqdn.md References • administration/management/logs.md • faq/operation_maintenance_faq.md • administration/management/FE_configuration.md • knowledge/trouble_shooting/trouble_shooting_scripts.md metadata writes. References • loading/loading_introduction/troubleshooting_loading.md • administration/management/FE_configuration.md

Rocky

08/13/2026, 2:01 PM

• sql-reference/sql-functions/utility-functions/last_query_id.md • sql-reference/System_variable.md • administration/http_interface/query_detail.md

mohammad atif

08/13/2026, 2:06 PM

@Rocky I am on version 4.0.9 and this issue still exists . Is there any open bug for this issue ?

Rocky

08/13/2026, 2:07 PM

As of current information, there is no widely documented or fixed bug specifically for version 4.0.9 that resolves this "FQDN-as-user" mismatch in the

QueryHistoryMgr

. Since you are seeing the system attempt to use the FE's host identity (FQDN) as a database user, this confirms that the background flusher's

ConnectContext

is incorrectly inheriting the node's identity rather than using the hardcoded internal

root

SYSTEM

identity. Why this is happening in 4.0.9 In StarRocks, background tasks (like the Query History flusher) execute internal

INSERT

statements to move logs from memory to the

_statistics_

database. * The Bug: In an FQDN-enabled cluster, the code that initializes the background session is erroneously grabbing the

host_name

of the FE and treating it as the

user_name

for the session. * The Result: The RBAC (Privilege) system sees a request from a user like

fe-0.starrocks.svc.cluster.local

, doesn't find that user in the metadata, and denies the insert. Recommended Workaround (The "Self-Authorization" Fix) Since the software is treating your FE node as a user, you must explicitly authorize your FE nodes to perform these writes. You will need to do this for each FE in your cluster:

Copy code

sql
-- 1. Create the 'user' that the log says is being denied
-- Replace 'fe-hostname' with the exact string found in your fe.log
CREATE USER 'fe-hostname'@'%';

-- 2. Grant permissions to the internal statistics table
GRANT INSERT, SELECT ON _statistics_.query_history TO 'fe-hostname'@'%';

-- 3. In Shared-Data mode, background tasks MUST have a warehouse to run the INSERT
-- Grant usage on your default warehouse to this node-user
GRANT USAGE ON WAREHOUSE default_warehouse TO USER 'fe-hostname'@'%';

Next Steps 1. Check for Warehouse Issues: Since you are in a shared-node (Lake) architecture, the Query History flusher depends on the

default_warehouse

. If your default warehouse is suspended or scaled to 0, even with the user created, the insert will fail with a "No active nodes" error. 2. Open/Follow a Bug Report: If this persists on a modern version like 4.0.9, it is likely a regression in the FQDN handling for background daemons. I recommend: * Searching the StarRocks GitHub Issues for "QueryHistoryMgr FQDN". * If no match is found, please open a new issue and provide the snippet of your

fe.log

and your

SHOW FRONTENDS

output. This specifically helps developers identify where the

ConnectContext

is being misinitialized. Summary: This is a logic error in how the background executor identifies itself. Creating the "FQDN user" is the only way to unblock the flusher without a code patch. References • administration/management/FE_configuration.md • administration/http_interface/query_detail.md • administration/management/enable_fqdn.md

Shiv Kumar

08/13/2026, 2:39 PM

@mohammad atif I think you are asking questions on wrong thread

mohammad atif

08/13/2026, 2:40 PM

@Shiv Kumar sorry I got it.

Shiv Kumar

08/13/2026, 2:46 PM

@Rocky I did

SET GLOBAL enable_query_history = true;

still I am seeing this error when running

*SELECT* * *FROM* information_schema.query_history

Copy code

SQL Error [5502] [42602]: Getting analyzing error. Detail message: Unknown table 'information_schema.query_history'.

Rocky

08/13/2026, 2:47 PM

The error occurs because query_history
is not located in the

information_schema

database. In StarRocks, this feature uses a physical table stored in the _statistics_
database. 1. Correct the Query Try running your query against the correct database:

Copy code

sql
SELECT * FROM _statistics_.query_history;

2. Why the table might still be missing If you still get an "Unknown table" error even when querying

_statistics_.query_history

, it is likely because of the FQDN/Access Denied issue we discussed earlier. When you run

SET GLOBAL enable_query_history = true;

, the StarRocks Leader FE attempts to automatically create the

_statistics_.query_history

table. However, since your FE is identifying itself using its FQDN (which isn't a recognized user), the internal
CREATE TABLE
and
INSERT
commands are failing silently or logged as errors in the background. To verify this: Check if the table exists at all:

Copy code

sql
SHOW TABLES FROM _statistics_ LIKE 'query_history';

If it returns an empty result, the table was never created. 3. Recommended Fix for 4.0.9 To resolve this and get the table created: 1. Create the FE "User": As mentioned before, look at your

fe.log

for the exact FQDN string being denied and create that user:

Copy code

sql
    -- Example (use your actual FE FQDN)
    CREATE USER 'starrocks-fe-0.starrocks.svc.cluster.local'@'%';
    GRANT ALL PRIVILEGES ON DATABASE _statistics_ TO 'starrocks-fe-0.starrocks.svc.cluster.local'@'%';
    -- Background tasks in shared-data mode need a warehouse
    GRANT USAGE ON WAREHOUSE default_warehouse TO USER 'starrocks-fe-0.starrocks.svc.cluster.local'@'%';

2. Restart the Feature:

Copy code

sql
    SET GLOBAL enable_query_history = false;
    SET GLOBAL enable_query_history = true;

This will trigger the FE to attempt the table creation again using its internal session. Summary * Wrong DB: Use

_statistics_.query_history

. * Missing Table: If it's missing from

_statistics_

, it's because the "FQDN User" doesn't have permissions to create it. * Shared-Data Requirement: Ensure your

default_warehouse

is active, as

query_history

in v3.x/v4.x is a Lake Table and cannot be created or written to without an active compute group. References • administration/management/monitor_manage_big_queries.md • sql-reference/sql-statements/data-definition/CREATE_USER.md • administration/management/enable_fqdn.md References • sql-reference/information_schema/statistics.md • loading/loading_introduction/troubleshooting_loading.md • sql-reference/information_schema/column_stats_usage.md • using_starrocks/Cost_based_optimizer.md • sql-reference/information_schema/tables.md • sql-reference/information_schema/be_bvars.md

Shiv Kumar

08/13/2026, 2:51 PM

@Rocky I did run

*SELECT* * *FROM*

_statistics_.query_history

. This table is present but holds no data even after I have run multiple queries in a new session. Checked 10-15 minutes after running the query still holds nothing. For your reference I am running my application queries on external Glue Catalog on iceberg tables want to query history of these queries

Rocky

08/13/2026, 2:51 PM

Since the table

_statistics_.query_history

exists but is empty, it means the background flusher is failing to write the data from the FE's memory into the table. Even though you are querying external Iceberg tables, StarRocks records the SQL string, user, and duration for all queries executed through the FE. The fact that it's empty in v4.0.9 (likely a Shared-Data/Lake cluster) points to one of the following three issues: 1. The "FQDN User" lacks Warehouse permissions In StarRocks Shared-Data mode, the

_statistics_.query_history

table is a Lake Table. Any

INSERT

into a Lake Table (even internal ones) requires an active Warehouse. Because of the bug where the background flusher identifies as the FE's FQDN (e.g.,

starrocks-fe-0...

), that "user" must have permission to use the warehouse. Fix:

Copy code

sql
-- Replace 'fe-hostname' with the exact name from your fe.log
GRANT USAGE ON WAREHOUSE default_warehouse TO USER 'fe-hostname'@'%';

If the FE cannot "rent" a warehouse, it cannot save the query history. 2. Verify Background Errors in
fe.log
To confirm why the flusher is failing, search your

fe.log

(on the Leader FE) for the keyword

QueryHistoryMgr

. You will likely see one of these errors: * "Access denied for user 'your-fe-fqdn'...": The user exists but lacks

INSERT

permission on

_statistics_

. * "No active compute node found": The flusher doesn't have a warehouse to run the insert on. * "Internal error": Potential schema mismatch. 3. Check Duration Thresholds By default, StarRocks captures all queries. However, check if a duration threshold has been set that might be filtering out your Iceberg queries if they are running very fast:

Copy code

sql
ADMIN SHOW FRONTEND CONFIG LIKE 'query_history_min_query_duration_ms';

If this value is high (e.g.,

), and your Iceberg queries are returning in 2 seconds, they won't be recorded. Set it to

to capture everything:

Copy code

sql
-- This must be set in fe.conf or via ADMIN SET CONFIG if supported
ADMIN SET CONFIG ("query_history_min_query_duration_ms" = "0");

4. Wait for the Flush Interval The query history is not written instantly. It is buffered in memory and flushed every 60 seconds (default). * Check the interval:

ADMIN SHOW FRONTEND CONFIG LIKE 'query_history_flush_interval';

* If you just ran queries and checked immediately, wait at least 2 minutes. Summary Checklist for v4.0.9: 1. Identity: Ensure

CREATE USER 'your-fe-fqdn'@'%';

was successful. 2. Permissions: Ensure that user has

INSERT

_statistics_.*

. 3. Warehouse: Ensure that user has

USAGE

default_warehouse

. 4. Activity: Ensure the

default_warehouse

is currently running (not suspended). If these are all met and the table is still empty, please check the

fe.log

for any line containing

lambda$asyncFlush

QueryHistoryMgr

and share the specific error message. References • administration/management/enable_fqdn.md • administration/http_interface/query_detail.md • faq/Sql_faq.md

mohammad atif

08/13/2026, 2:51 PM

@Shiv Kumar you might be getting same issue as I am facing

Shiv Kumar

08/20/2026, 7:28 PM

@Rocky I tried running CREATE

*USER* 'kube-starrocks-fe-0.kube-starrocks-fe-search.xeno-lakehouse.svc.cluster.local'@'%';

for FQDN user but it's giving me error:

Copy code

SQL Error [1064] [42000]: Getting analyzing error. Detail message: invalid user name: kube-starrocks-fe-0.kube-starrocks-fe-search.xeno-lakehouse.svc.cluster.local.

How can I have

enable_query_history

working, neither I am able to create user nor it's working without this user

Rocky

08/20/2026, 7:29 PM

The error you are seeing is due to a fundamental conflict in StarRocks between how FQDNs are structured and how usernames are validated: 1. Naming Constraint: StarRocks usernames can only consist of letters, digits, and underscores (

). They specifically cannot contain hyphens (

) or dots (

). 2. The Bug: In version 4.0.9, when FQDN is enabled (common in Kubernetes), the

QueryHistoryMgr

background flusher incorrectly attempts to identify itself using the FE node's hostname (e.g.,

kube-starrocks-fe-0...

) as the username. 3. The Catch-22: Because your hostname contains hyphens and dots, it is an "invalid username" according to the StarRocks SQL analyzer. You cannot create the user to grant it permissions, and the system cannot use it internally, leaving the

query_history

table empty. How to get Query History working Since you cannot create a user with hyphens/dots, you cannot fix this by granting permissions to that specific FQDN identity. Instead, try these steps: 1. Check for a "System" Identity in the Logs Search your

fe.log

for the exact "Access denied" error. Occasionally, the system might be trying to use a truncated version of the name or an IP address. * If the log says

Access denied for user 'root'@'...'

, simply ensure

root

has

INSERT

permissions on

_statistics_.query_history

. * If it specifically says

Access denied for user 'kube-starrocks-fe-0...'

, you are confirmed to be hitting the FQDN bug. 2. Alternative: Use the FE Audit Log (Recommended for 4.0.9) Since the internal

query_history

table is currently blocked by this identity bug in your environment, the most reliable way to track queries against your Glue/Iceberg tables is the Audit Log. * All queries (including those against external catalogs) are recorded in

fe/log/fe.audit.log

. * You can create a Log-based Metadata Table to query this log file directly using SQL if you need to analyze the history:

Copy code

sql
    -- Example of creating an external table to query your own audit logs
    CREATE TABLE audit_log_query (
        query_id VARCHAR(64),
        user VARCHAR(64),
        query_sql VARCHAR(65533),
        ...
    ) ENGINE=LOG ...

3. Potential Workaround: Grant to PUBLIC Role In some specific patch versions, background tasks may inherit the

PUBLIC

role. You can try:

Copy code

sql
GRANT INSERT ON TABLE _statistics_.query_history TO ROLE public;
-- Note: This is a broad permission and should be used only to test if it unblocks the flusher.

Summary of Status * Is there an open bug? Yes, this is a known issue where background service identities do not respect FQDN naming constraints. It is being addressed by forcing background tasks to use the

root

identity regardless of FQDN settings. * Can you fix it in 4.0.9? If the "PUBLIC" grant doesn't work, there is no SQL-level fix because the parser blocks the creation of the required user identity. * Next Step: I recommend checking for the v4.0.10+ release notes or contacting StarRocks support/community to see if a hotfix has been backported to your specific environment to force

QueryHistoryMgr

to use the

root

user. If you must have this in a table right now, use a sidecar or script to parse the

fe.audit.log

and insert it into a custom table. References • administration/management/enable_fqdn.md • sql-reference/System_limit.md • administration/management/monitor_manage_big_queries.md References • sql-reference/sql-statements/account-management/CREATE_USER.md

Rocky

08/20/2026, 7:29 PM

• administration/user_privs/authentication/native_authentication.md • sql-reference/System_limit.md

Shiv Kumar

08/20/2026, 7:55 PM

@Rocky Let's go ahead with Log-based Metadata Table
. give me the exact steps and queries to create this log based metadata table to read from logs

Rocky

08/20/2026, 7:55 PM

It appears there is a slight confusion regarding the table engine name. In StarRocks, there is no

ENGINE=LOG

(that syntax is primarily used in ClickHouse). To achieve a "Log-based Metadata Table" for your query history, the official and supported method is to use the AuditLoader plugin. This plugin automatically reads the local

fe.audit.log

file (which records all queries, including those against your Glue/Iceberg catalog) and loads them into a standard StarRocks table for SQL analysis. Step 1: Create the Destination Table First, create a database and a table where the logs will be stored. This table uses the

OLAP

engine.

Copy code

sql
CREATE DATABASE IF NOT EXISTS starrocks_audit_db__;

CREATE TABLE starrocks_audit_db__.starrocks_audit_tbl__ (
  `queryId` VARCHAR(64) COMMENT "Unique ID of the query",
  `timestamp` DATETIME NOT NULL COMMENT "Query start time",
  `queryType` VARCHAR(12) COMMENT "Query type (query, slow_query, connection)",
  `clientIp` VARCHAR(32) COMMENT "Client IP",
  `user` VARCHAR(64) COMMENT "Query username",
  `authorizedUser` VARCHAR(64) COMMENT "User identity",
  `resourceGroup` VARCHAR(64) COMMENT "Resource group name",
  `catalog` VARCHAR(32) COMMENT "Catalog name (e.g., your glue catalog)",
  `db` VARCHAR(96) COMMENT "Database name",
  `state` VARCHAR(8) COMMENT "Query state (EOF, ERR, OK)",
  `errorCode` VARCHAR(512) COMMENT "Error code",
  `queryTime` BIGINT COMMENT "Execution time (ms)",
  `scanBytes` BIGINT COMMENT "Bytes scanned",
  `scanRows` BIGINT COMMENT "Rows scanned",
  `returnRows` BIGINT COMMENT "Rows returned",
  `cpuCostNs` BIGINT COMMENT "CPU time (ns)",
  `memCostBytes` BIGINT COMMENT "Memory used (bytes)",
  `stmtId` INT COMMENT "Statement ID",
  `isQuery` TINYINT COMMENT "1 if query, 0 otherwise",
  `feIp` VARCHAR(128) COMMENT "FE IP",
  `stmt` VARCHAR(1048576) COMMENT "The actual SQL statement",
  `digest` VARCHAR(32) COMMENT "SQL fingerprint",
  `warehouse` VARCHAR(32) NULL COMMENT "Warehouse name"
) ENGINE = OLAP
DUPLICATE KEY (`queryId`, `timestamp`, `queryType`)
COMMENT "Audit log table"
PARTITION BY date_trunc('day', `timestamp`)
PROPERTIES (
  "replication_num" = "1", -- Adjust based on your cluster size
  "partition_live_number" = "30" -- Keep 30 days of history
);

Step 2: Install the AuditLoader Plugin Since you are on Kubernetes, you can typically install the plugin via the FE's internal plugin management. 1. Download the plugin: You can download the AuditLoader zip to a location accessible by your FE (or use a sidecar/init-container in K8s). 2. Configure `plugin.conf`: Inside the zip, edit

plugin.conf

to point to your FE's host/port and provide the credentials:

Copy code

conf
    # The StarRocks FE host for loading
    frontend_host=127.0.0.1
    # The database and table created in Step 1
    database=starrocks_audit_db__
    table=starrocks_audit_tbl__
    user=root
    password=YOUR_PASSWORD

3. Install the plugin via SQL:

Copy code

sql
    INSTALL PLUGIN FROM "http://<your_server>/auditloader.zip";
    -- Or if you copied it to the FE local path:
    INSTALL PLUGIN FROM "/path/to/auditloader.zip";

Step 3: Verify and Query Once installed, the plugin will start flushing the contents of

fe.audit.log

into your table every 60 seconds (default).

Copy code

sql
-- Check if the plugin is active
SHOW PLUGINS;

-- Query your history (specifically for Iceberg/Glue queries)
SELECT timestamp, user, queryTime, stmt 
FROM starrocks_audit_db__.starrocks_audit_tbl__
WHERE catalog = 'your_glue_catalog_name'
ORDER BY timestamp DESC;

Alternative: Querying BE Logs Directly If you were looking for a built-in system table that behaves like a "metadata table" for logs, StarRocks provides:

Copy code

sql
SELECT * FROM information_schema.be_logs;

Note: This only contains Backend (BE) process logs, not the SQL query history. Why use this over
query_history
?

Rocky

08/20/2026, 7:55 PM

Given the FQDN/User Name bug you encountered in v4.0.9, the AuditLoader is more reliable because it uses a standard

INSERT

(HTTP PUT) with a user you define (like

root

), bypassing the internal identity issues of the background

query_history

flusher. References • administration/management/audit_loader.md • sql-reference/information_schema/be_logs.md • administration/management/logs.md References • administration/management/FE_configuration.md • administration/management/logs.md • sql-reference/information_schema/be_logs.md • faq/operation_maintenance_faq.md • administration/management/audit_loader.md

2 Views

Open in Slack

Previous Next