@Rocky hello

Hello! I'm Rocky, your StarRocks assistant. How can I help you today?

@Rocky how to add error log counter in starrock to grafana

StarRocks does not provide a single built-in metric that counts the total number of lines in its system log files (like

be.ERROR.log

). However, you can achieve an "error log counter" in Grafana using two different methods depending on what you specifically need to monitor. Method 1: Using Built-in StarRocks Metrics (Recommended) StarRocks exposes specific error counters via its Prometheus metrics endpoint. These are usually sufficient for monitoring the health of the cluster without parsing log files. * Query Failures: Use the metric

starrocks_fe_query_err

. * Grafana PromQL:

rate(starrocks_fe_query_err[1m])

* This tracks the rate of SQL queries that failed (due to syntax, timeouts, or system errors). * Data Load Errors: Use

starrocks_fe_routine_load_error_rows

. * This tracks the total number of rows that failed to load during Routine Load operations. * Resource Group Errors: Use

starrocks_fe_query_resource_group_err

. * Tracks queries that failed specifically due to resource group limits (e.g., memory/CPU throttling). * Storage Errors: Use

starrocks_be_segment_file_not_found_total

. * Tracks critical storage issues where segment files are missing. To add these, ensure your StarRocks cluster is connected to Prometheus and then import the official StarRocks Overview Grafana dashboard, which already includes many of these "Error" panels. Method 2: Counting Lines in

be.ERROR.log

(Custom Log Pipeline) If you specifically want to count occurrences of the string "ERROR" inside the physical log files (e.g.,

be.ERROR.log

or

fe.warn.log

), you must use a log collector like Grafana Promtail or Fluentd. Example using Promtail: 1. Install Promtail on your BE/FE nodes. 2. Configure a

scrape_config

to tail the StarRocks log directory (usually

/opt/starrocks/be/log/*.ERROR.log

). 3. Use a

metrics

stage in Promtail to create a counter:

yaml
    pipeline_stages:
      • match:
          selector: '{job="starrocks_logs"}'
          stages:
            • metrics:
                be_error_log_count:
                  type: Counter
                  description: "Total number of error log lines"
                  source: level
                  config:
                    action: inc

4. Promtail will then export this counter to Prometheus, which you can visualize in Grafana. Summary of Logs vs. Metrics Issue Type | Metric to Use in Grafana | Log File Location Failed SQL Queries |

starrocks_fe_query_err

|

fe/log/fe.audit.log

Data Load Errors |

starrocks_fe_routine_load_error_rows

|

be/storage/error_log/

System/Process Errors | Use Promtail on

be.ERROR.log

|

be/log/be.ERROR.log

For most users, monitoring Query Error Rate (

starrocks_fe_query_err

) is the best way to track "errors" in Grafana. References • administration/management/monitoring/metrics.md • administration/management/BE_configuration.md