@Rocky streaming pipelines (from spark starrocks connector) accumulate too many logs and can I somehow disable it?

The Spark StarRocks connector does not have a specific internal configuration flag to disable logs. Instead, it relies on the standard Log4j (or Log4j2) configuration of your Spark environment. Because streaming pipelines run micro-batches frequently, the

INFO

logs generated by the connector (which report on every Stream Load transaction) can accumulate quickly. To reduce or disable these logs, you need to adjust the log level for the StarRocks connector package. 1. Identify the Logger Package The primary package for the connector is:

com.starrocks.connector.spark

2. Configure via Log4j Properties Add the following line to your

log4j.properties

file (for Spark 2.x/3.x using Log4j 1.x):

properties
log4j.logger.com.starrocks.connector.spark=WARN

If you are using Spark 3.3+ (which uses Log4j 2), add this to your `log4j2.properties`:

properties
logger.starrocks.name = com.starrocks.connector.spark
logger.starrocks.level = warn

3. Configure via

spark-submit

If you cannot modify the cluster's global logging config, you can pass the log level change directly in your

spark-submit

command using `extraJavaOptions`:

bash
spark-submit \
  --conf "spark.driver.extraJavaOptions=-Dlog4j.configuration=log4j.properties" \
  --conf "spark.executor.extraJavaOptions=-Dlog4j.configuration=log4j.properties" \
  ...

(Ensure the

log4j.properties

file containing the

WARN

level is distributed to the executors using

--files

if necessary.) 4. Adjusting the Log Volume in StarRocks (Optional) If you are referring to the audit logs or BE logs inside the StarRocks cluster itself (rather than the Spark driver/executor logs): * Audit Logs: You can filter out internal Stream Load requests from the FE audit log by adjusting

audit_log_modules

in the FE config, though this is usually not recommended as it hides all load history. * BE Logs: You can dynamically change the BE log level to

WARN

if the

<http://be.INFO|be.INFO>

logs are too large:

sql
    ADMIN SET FRONTEND CONFIG ("sys_log_level" = "WARN");

Recommendation: Set the connector's log level to

WARN

on the Spark side. This will silence the "Commit success" and "Begin transaction" messages for every micro-batch while still alerting you if a load actually fails. References • loading/Flink-connector-starrocks.md • administration/management/BE_configuration.md • administration/management/FE_configuration.md • loading/Kafka-connector-starrocks.md • administration/management/logs.md