Slackbot
05/19/2023, 1:12 PMYu Wei Sung
05/19/2023, 1:18 PMBruno
05/19/2023, 1:50 PM## DB Ledger storage configuration
dbStorage_writeCacheMaxSizeMb=32
dbStorage_readAheadCacheMaxSizeMb=32
dbStorage_readAheadCacheBatchSize=1000
dbStorage_rocksDB_blockCacheSize=8388608
# Other RocksDB specific tunables
dbStorage_rocksDB_writeBufferSizeMB=8
dbStorage_rocksDB_sstSizeInMB=64
dbStorage_rocksDB_blockSize=65536
dbStorage_rocksDB_bloomFilterBitsPerKey=10
dbStorage_rocksDB_numLevels=-1
dbStorage_rocksDB_numFilesInLevel0=4
dbStorage_rocksDB_maxSizeInLevel1MB=256
# Settings below are used by stream/table service
extraServerComponents=org.apache.bookkeeper.stream.server.StreamStorageLifecycleComponent
storageserver.grpc.port=4181
dlog.bkcEnsembleSize=3
dlog.bkcWriteQuorumSize=2
dlog.bkcAckQuorumSize=2
storage.range.store.dirs=data/bookkeeper/ranges
storage.serve.readonly.tables=false
In fact, the only difference is that I had to add the variable "extraServerComponents=org.apache.bookkeeper.stream.server.StreamStorageLifecycleComponent" because otherwise port 4181 is never open.Bruno
05/19/2023, 2:14 PM2023-05-19T13:10:25,997+0000 [io-write-scheduler-OrderedScheduler-0-0] INFO org.apache.bookkeeper.statelib.impl.journal.AbstractStateStoreWithJournal - Initializing the local state for mvcc store 000000000000000000/000000000000000000/000000000000000000
2023-05-19T13:10:25,999+0000 [io-write-scheduler-OrderedScheduler-0-0] ERROR org.apache.bookkeeper.statelib.impl.kv.RocksdbKVStore - Failed to restore checkpoint: Checkpoint{ID='6e1c6fd3-8992-4bfe-b580-bb0e918332e0', createdAt: 0 null}
org.apache.bookkeeper.statelib.api.exceptions.StateStoreException: Failed to create dir 000000000000000000/000000000000000000/000000000000000000
...
...
Caused by: java.nio.file.AccessDeniedException: /pulsar/data/bookkeeper/ranges
...
..
Bruno
05/19/2023, 2:23 PMYu Wei Sung
05/19/2023, 2:23 PMBruno
05/19/2023, 3:21 PM$ bin/pulsar-admin functions querystate --tenant public --namespace default --name wordcount --key test --watch
{
"key": "test",
"stringValue": "hello pulsar",
"version": 0
}
Bruno
05/19/2023, 3:25 PMYu Wei Sung
05/19/2023, 3:26 PMJon P
06/14/2023, 11:18 AMconfigData
- would you mind providing an example of the format? I've tried the following (also tried replacing .
with _
in the property names). (I'm using a replica count of 1 to test on Minikube)
bookkeeper:
replicaCount: 1 # default 4
configData:
extraServerComponents: "org.apache.bookkeeper.stream.server.StreamStorageLifecycleComponent"
storageserver.grpc.port: "4181"
dlog.bkcEnsembleSize: "1"
dlog.bkcWriteQuorumSize: "1"
dlog.bkcAckQuorumSize: "1"
storage.range.store.dirs: "data/bookkeeper/ranges"
storage.serve.readonly.tables: "false"
<http://storage.cluster.controller.schedule.interval.ms|storage.cluster.controller.schedule.interval.ms>: "30000"
... but only extraServerComponents
gets picked up.
I had what I believe is a similar issue configuring it in Docker because apply-config-from-env.py
only applies values which already have defaults in bookkeeper.conf
and the various storage server properties are not in bookkeeper.conf
. Running in Docker I had to 'add' them to the conf file as follows:
command: >
bash -c "
bin/apply-config-from-env.py conf/bookkeeper.conf &&
echo 'storageserver.grpc.port=4181' >> conf/bookkeeper.conf &&
echo 'dlog.bkcEnsembleSize=1' >> conf/bookkeeper.conf &&
echo 'dlog.bkcWriteQuorumSize=1' >> conf/bookkeeper.conf &&
echo 'dlog.bkcAckQuorumSize=1' >> conf/bookkeeper.conf &&
echo 'storage.range.store.dirs=data/bookkeeper/ranges' >> conf/bookkeeper.conf &&
echo 'storage.serve.readonly.tables=false' >> conf/bookkeeper.conf &&
echo '<http://storage.cluster.controller.schedule.interval.ms|storage.cluster.controller.schedule.interval.ms>=30000' >> conf/bookkeeper.conf &&
exec bin/pulsar bookie
"
Many thanks
(oh, and I'm using Pulsar 3.0.0 Docker images. So weird that Helm chart version 3.0.0 doesn't use image versions 3.0.0?!)Jon P
06/14/2023, 2:04 PMbookkeeper:
replicaCount: 1 # default 4
configData:
extraServerComponents: "org.apache.bookkeeper.stream.server.StreamStorageLifecycleComponent"
additionalCommand: >
echo 'storageserver.grpc.port=4181' >> conf/bookkeeper.conf &&
echo 'dlog.bkcEnsembleSize=1' >> conf/bookkeeper.conf &&
echo 'dlog.bkcWriteQuorumSize=1' >> conf/bookkeeper.conf &&
echo 'dlog.bkcAckQuorumSize=1' >> conf/bookkeeper.conf &&
echo 'storage.range.store.dirs=data/bookkeeper/ranges' >> conf/bookkeeper.conf &&
echo 'storage.serve.readonly.tables=false' >> conf/bookkeeper.conf &&
echo '<http://storage.cluster.controller.schedule.interval.ms|storage.cluster.controller.schedule.interval.ms>=30000' >> conf/bookkeeper.conf &&
The final &&
is required, otherwise bookie exec command gets written into bookkeeper.conf
as well, rather than executed!Yu Wei Sung
06/14/2023, 2:06 PMJon P
06/14/2023, 2:11 PMJon P
06/14/2023, 7:05 PMapply-config-from-env
script which handles that pattern, just PULSAR_PREFIX_
for env variables. (edit: I noticed a -with-prefix
version in the repo which does in fact add/replace based on a prefix, but the helm chart uses apply-config-from-env
version)Jon P
06/14/2023, 7:15 PMCaused by: java.nio.file.AccessDeniedException: /pulsar/data/bookkeeper/ranges
...
Cannot invoke "org.rocksdb.RocksDB.get(org.rocksdb.ColumnFamilyHandle, byte[])" because "this.db" is null
errors you were seeing. When you say i add volume /pulsar/data/bookkeeper/ranges
, would you mind explaining how you did that for the bookie in your setup? I'm struggling to work out how to do it, am fairly new to k8s. Found this PR which looked like fix for similar issue, but not working here. Trying to run in Minikube on an M2 Pro Mac, using official Pulsar Helm chart 3.0.0 - overriding to use Pulsar 3.0.0 Docker image. I have volumes.persistence: false
in case that affects my ability to configure volumes. Thank you.Jon P
06/14/2023, 7:59 PMadditionalCommand
solves everything 😉 I tried adding
mkdir /pulsar/data/bookkeeper/ranges &&
chmod -R g+w /pulsar/data/bookkeeper/ranges &&
to the above, but no joy