This message was deleted.
# general
s
This message was deleted.
b
Hello Yu Wei Sung, Thanks, i use the same config, you can see a bookkeeper.conf (part without comments)
Copy code
## DB Ledger storage configuration
dbStorage_writeCacheMaxSizeMb=32
dbStorage_readAheadCacheMaxSizeMb=32
dbStorage_readAheadCacheBatchSize=1000
dbStorage_rocksDB_blockCacheSize=8388608

# Other RocksDB specific tunables
dbStorage_rocksDB_writeBufferSizeMB=8
dbStorage_rocksDB_sstSizeInMB=64
dbStorage_rocksDB_blockSize=65536
dbStorage_rocksDB_bloomFilterBitsPerKey=10
dbStorage_rocksDB_numLevels=-1
dbStorage_rocksDB_numFilesInLevel0=4
dbStorage_rocksDB_maxSizeInLevel1MB=256

# Settings below are used by stream/table service

extraServerComponents=org.apache.bookkeeper.stream.server.StreamStorageLifecycleComponent
storageserver.grpc.port=4181
dlog.bkcEnsembleSize=3
dlog.bkcWriteQuorumSize=2
dlog.bkcAckQuorumSize=2
storage.range.store.dirs=data/bookkeeper/ranges
storage.serve.readonly.tables=false
In fact, the only difference is that I had to add the variable "extraServerComponents=org.apache.bookkeeper.stream.server.StreamStorageLifecycleComponent" because otherwise port 4181 is never open.
Sorry, I didn't specify the correct error, the main error is: (Caused by: java.nio.file.AccessDeniedException: /pulsar/data/bookkeeper/ranges)
Copy code
2023-05-19T13:10:25,997+0000 [io-write-scheduler-OrderedScheduler-0-0] INFO  org.apache.bookkeeper.statelib.impl.journal.AbstractStateStoreWithJournal - Initializing the local state for mvcc store 000000000000000000/000000000000000000/000000000000000000
2023-05-19T13:10:25,999+0000 [io-write-scheduler-OrderedScheduler-0-0] ERROR org.apache.bookkeeper.statelib.impl.kv.RocksdbKVStore - Failed to restore checkpoint: Checkpoint{ID='6e1c6fd3-8992-4bfe-b580-bb0e918332e0', createdAt: 0 null}
org.apache.bookkeeper.statelib.api.exceptions.StateStoreException: Failed to create dir 000000000000000000/000000000000000000/000000000000000000
...
...
Caused by: java.nio.file.AccessDeniedException: /pulsar/data/bookkeeper/ranges
...
..
I will try to add volume /pulsar/data/bookkeeper/ranges
y
access denied. can you shell into bookie pod and check the owner of /pullsar/data?
b
Thanks ! its ok when i add volume /pulsar/data/bookkeeper/ranges
Copy code
$ bin/pulsar-admin functions querystate --tenant public --namespace default --name wordcount --key test --watch
{
  "key": "test",
  "stringValue": "hello pulsar",
  "version": 0
}
Finally, I don't use the datastax chart because they use their own docker images, and they don't have the Pulsar 2.11.0 and 3.0.0 version yet.. however, I was strongly inspired by the chart to succeed in operate stateful functions. Do you know if I can use the official Pulsar 3.0.0 images with the datastax chart?
y
I don’t use helm chart because it is hard to control the version changes.
1
j
Apologies for hijacking this thread, but I'm also having trouble enabling stream/state storage on the bookie in k8s. @Yu Wei Sung - you commented that the bookie config can be included in the values.yml
configData
- would you mind providing an example of the format? I've tried the following (also tried replacing
.
with
_
in the property names). (I'm using a replica count of 1 to test on Minikube)
Copy code
bookkeeper:
  replicaCount: 1 # default 4
  configData:
    extraServerComponents: "org.apache.bookkeeper.stream.server.StreamStorageLifecycleComponent"
    storageserver.grpc.port: "4181"
    dlog.bkcEnsembleSize: "1"
    dlog.bkcWriteQuorumSize: "1"
    dlog.bkcAckQuorumSize: "1"
    storage.range.store.dirs: "data/bookkeeper/ranges"
    storage.serve.readonly.tables: "false"
    <http://storage.cluster.controller.schedule.interval.ms|storage.cluster.controller.schedule.interval.ms>: "30000"
... but only
extraServerComponents
gets picked up. I had what I believe is a similar issue configuring it in Docker because
apply-config-from-env.py
only applies values which already have defaults in
bookkeeper.conf
and the various storage server properties are not in
bookkeeper.conf
. Running in Docker I had to 'add' them to the conf file as follows:
Copy code
command: >
      bash -c "
        bin/apply-config-from-env.py conf/bookkeeper.conf &&
        echo 'storageserver.grpc.port=4181' >> conf/bookkeeper.conf &&
        echo 'dlog.bkcEnsembleSize=1' >> conf/bookkeeper.conf &&
        echo 'dlog.bkcWriteQuorumSize=1' >> conf/bookkeeper.conf &&
        echo 'dlog.bkcAckQuorumSize=1' >> conf/bookkeeper.conf &&
        echo 'storage.range.store.dirs=data/bookkeeper/ranges' >> conf/bookkeeper.conf &&
        echo 'storage.serve.readonly.tables=false' >> conf/bookkeeper.conf &&
        echo '<http://storage.cluster.controller.schedule.interval.ms|storage.cluster.controller.schedule.interval.ms>=30000' >> conf/bookkeeper.conf &&
        exec bin/pulsar bookie
      "
Many thanks (oh, and I'm using Pulsar 3.0.0 Docker images. So weird that Helm chart version 3.0.0 doesn't use image versions 3.0.0?!)
I made progress. This works
Copy code
bookkeeper:
  replicaCount: 1 # default 4
  configData:
    extraServerComponents: "org.apache.bookkeeper.stream.server.StreamStorageLifecycleComponent"
  additionalCommand: > 
    echo 'storageserver.grpc.port=4181' >> conf/bookkeeper.conf && 
    echo 'dlog.bkcEnsembleSize=1' >> conf/bookkeeper.conf && 
    echo 'dlog.bkcWriteQuorumSize=1' >> conf/bookkeeper.conf && 
    echo 'dlog.bkcAckQuorumSize=1' >> conf/bookkeeper.conf && 
    echo 'storage.range.store.dirs=data/bookkeeper/ranges' >> conf/bookkeeper.conf && 
    echo 'storage.serve.readonly.tables=false' >> conf/bookkeeper.conf && 
    echo '<http://storage.cluster.controller.schedule.interval.ms|storage.cluster.controller.schedule.interval.ms>=30000' >> conf/bookkeeper.conf &&
The final
&&
is required, otherwise bookie exec command gets written into
bookkeeper.conf
as well, rather than executed!
🙌 1
1
y
I think you need to add “PULSAR” as prefix to the new variables. like PULSAR_storageserver.grpc.port: “4181"
j
Thanks, I'll give that a try
Can't seem to make that work, or see anything in the
apply-config-from-env
script which handles that pattern, just
PULSAR_PREFIX_
for env variables. (edit: I noticed a
-with-prefix
version in the repo which does in fact add/replace based on a prefix, but the helm chart uses
apply-config-from-env
version)
Now apologies @Bruno - I've got to the same
Copy code
Caused by: java.nio.file.AccessDeniedException: /pulsar/data/bookkeeper/ranges
...
Cannot invoke "org.rocksdb.RocksDB.get(org.rocksdb.ColumnFamilyHandle, byte[])" because "this.db" is null
errors you were seeing. When you say
i add volume /pulsar/data/bookkeeper/ranges
, would you mind explaining how you did that for the bookie in your setup? I'm struggling to work out how to do it, am fairly new to k8s. Found this PR which looked like fix for similar issue, but not working here. Trying to run in Minikube on an M2 Pro Mac, using official Pulsar Helm chart 3.0.0 - overriding to use Pulsar 3.0.0 Docker image. I have
volumes.persistence: false
in case that affects my ability to configure volumes. Thank you.
Since I've so far gone with
additionalCommand
solves everything 😉 I tried adding
Copy code
mkdir /pulsar/data/bookkeeper/ranges &&
    chmod -R g+w /pulsar/data/bookkeeper/ranges &&
to the above, but no joy
🙌 1