@Rocky what do you recommend for setting alerts on query failures? Basically reporting exact query and failed reason to be reported to other systems like Sentry / Pager tools

To report exact query failures and reasons to systems like Sentry or PagerDuty, I recommend a combination of Structured Audit Logging and Log Forwarding. StarRocks does not have a native "push to Sentry" feature, but you can achieve this by following these steps: 1. Enable Structured JSON Audit Logs By default, StarRocks logs query information in a pipe-separated format in

fe.audit.log

. To make these logs easily consumable by external tools, enable the JSON format (available from v3.2.7+). Set the following FE configuration:

sql
ADMIN SET FRONTEND CONFIG ("audit_log_json_format" = "true");

This ensures that the

stmt

(SQL query) and

state

(EOF/ERR) are exported as structured JSON. 2. Use a Log Collector (Recommended) Use a tool like Filebeat, Fluentd, or Vector to tail the

fe.audit.log

on your FE nodes. These tools are better suited for sending the "exact query" and "failure reason" to Sentry than a metrics-based system like Prometheus. * Filter: Configure the collector to only pick up lines where

"state": "ERR"

. * Target Sentry: Use the Sentry API or a webhook to send the JSON log. * Target PagerDuty: You can use the PagerDuty Events API to create incidents for critical query failures. 3. Alternative: Querying the

AuditLoader

Table If you prefer to stay within the database environment, you can use the AuditLoader plugin to load audit logs into a StarRocks table. 1. Install the

AuditLoader

plugin. 2. Query the generated table for errors:

sql
   SELECT stmt, errorCode, queryTime
   FROM starrocks_audit_db*.starrocks_audit_tbl*
   WHERE state = 'ERR';

3. You can then run a small cron job or an external script to poll this table and forward new errors to Sentry/PagerDuty. 4. Metrics-based Alerting (PagerDuty via Alertmanager) If you just need an alert when failure rates spike (rather than the content of every single failed query), use the Prometheus metrics: * Metric:

starrocks_fe_query_err_rate

* PromSQL:

sum by (instance)(starrocks_fe_query_err_rate) * 100 > 10

* Workflow: Prometheus -> Alertmanager -> PagerDuty. Summary Recommendation: For high-fidelity reporting (exact query + stack trace/reason), use Method 1 & 2: JSON Audit Logs + a Log Collector (Filebeat/Fluentd) pointed at Sentry. This is the most robust way to get full context into your error-error tracking system-tracking tools. References • administration/management/logs.md • administration/management/FE_configuration.md • administration/management/audit_loader.md • faq/operation_maintenance_faq.md