does anyone know what can cause this error?

Invalid -Xlog option '-Xlog:gc*,safepoint:/tmp/pulsar-logs/pulsar_gc_%p.log:time,uptime,tags:filecount=10,filesize=20M', see error log for details.

from

/nix/store/6zhdlqjlvlv7mgxs33xsfrxr4d25dyc0-openjdk-minimal-jre-17.0.5+8/bin/java

I don't know where

-Xlog:gc*

is from

my whole systemd for Pulsar is

systemd.services.pulsar-bookie = mkIf cfg.bookie.enable {
      description = "Pulsar's Bookkeeper Daemon";
      wantedBy = [ "multi-user.target" ];
      after = [ "network.target" "pulsar-zookeeper.target" ];
      # FIXME: This causes the bookie to restart when the broker changes.
      inherit environment;
      serviceConfig.User = "pulsar-bookie";
      serviceConfig.ExecStartPre = 
       let 
        journalPath = cfg.bookie.settings.journalDirectories;
        ledgerPath = cfg.bookie.settings.ledgerDirectories;
        rangePath = cfg.bookie.settings."storage.range.store.dirs";
       in
       "+" + pkgs.writeScript "pulsar-bookie-prep" 
        ''
        #!${pkgs.runtimeShell}
        set -euo pipefail
        mkdir -p ${journalPath}
        install -d -o pulsar-bookie ${journalPath}
        mkdir -p ${ledgerPath}
        install -d -o pulsar-bookie ${ledgerPath}
        mkdir -p ${rangePath}
        install -d -o pulsar-bookie ${rangePath}
        '';
      script = ''
        # It is non-destructive to re-run the cluster initialization
        ${cfg.package}/bin/pulsar initialize-cluster-metadata \
            --cluster ${cfg.clusterName} \
            --zookeeper ${zkConnectionString} \
            --configuration-store ${zkConnectionString} \
            --web-service-url ${cfg.webServiceUrl} \
            --broker-service-url ${cfg.brokerServiceUrl}
        # However we should only format the bookie data once
        if [ ! -f ${cfg.bookie.dataDir}/has-done-init ]; then
          ${cfg.package}/bin/bookkeeper shell bookieformat -nonInteractive
          touch ${cfg.bookie.dataDir}/has-done-init
        fi
        ${cfg.package}/bin/pulsar bookie
      '';
    };

I mean bookie

looking at the source code, I found 3 places where that option was hard coded: https://github.com/apache/pulsar/blob/master/bin/function-localrunner#L49 https://github.com/apache/pulsar/blob/master/conf/bkenv.sh#L72 https://github.com/apache/pulsar/blob/master/conf/pulsar_env.sh#L74

how did you config it in your test run?

java -version openjdk version “17.0.5” 2022-10-18

same error for

/nix/store/jv4l318w51m21hs2jsz4hf3bh6dhq95r-openjdk-17.0.5+8/lib/openjdk/bin/java

I also see this error

Initialization of output 'file=/tmp/pulsar-logs/pulsar_gc_%p.log' using options 'filecount=10,filesize=20M' failed

which indicates that it could be a file permission issue

is there an option to disable GC logging completely?

but it used to working with Pulsar 2.9.1

You should change permissions on that directory or change the directory that gc logs are written to

I see why. the owner is broker instead of bookie

[root@pulsar3:~]# ls -lh /tmp/pulsar-logs
total 32K
-rw-r--r-- 1 pulsar-broker pulsar-broker 13K Feb 10 18:47 pulsar_gc_408154.log
-rw-r--r-- 1 pulsar-broker pulsar-broker 14K Feb 10 20:37 pulsar_gc_408872.log

but this has been working all the time. there must be a recent change in the bookkeeper logging

I think I've found the cause. https://github.com/apache/pulsar/blob/master/conf/bkenv.sh#L65

is it really a good idea to fall back on

PULSAR_GC_LOG

?

this code block doesn't make sense to me

if [[ -z "$BOOKIE_GC_LOG" ]]; then
  # fallback to PULSAR_GC_LOG if it is set
  BOOKIE_GC_LOG="$PULSAR_GC_LOG"
fi

BOOKIE_GC_LOG_DIR=${BOOKIE_GC_LOG_DIR:-"${PULSAR_GC_LOG_DIR:-"${BOOKIE_LOG_DIR}"}"}

if [[ -z "$BOOKIE_GC_LOG" ]]; then
  if [[ $JAVA_MAJOR_VERSION -gt 8 ]]; then
    BOOKIE_GC_LOG="-Xlog:gc*,safepoint:${BOOKIE_GC_LOG_DIR}/pulsar_bookie_gc_%p.log:time,uptime,tags:filecount=10,filesize=20M"
    if [[ $JAVA_MAJOR_VERSION -ge 17 ]]; then
      # Use async logging on Java 17+ <https://bugs.openjdk.java.net/browse/JDK-8264323>
      BOOKIE_GC_LOG="-Xlog:async ${BOOKIE_GC_LOG}"
    fi
  else
    # Java 8 gc log options
    BOOKIE_GC_LOG="-Xloggc:${BOOKIE_GC_LOG_DIR}/pulsar_bookie_gc_%p.log -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCApplicationStoppedTime -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=20M"
  fi
fi

the second

if [[ -z "$BOOKIE_GC_LOG" ]]; then

block will never run if

PULSAR_GC_LOG

is set, will it?

it may not be a bug, but it is very confusing when doing operations