Getting below error while starting Controller. Any...
# troubleshooting
m
Getting below error while starting Controller. Any idea what could be the problem ?
2025/05/08 18:36:00.589 ERROR [StartServiceManagerCommand] [main] Failed to start a Pinot [CONTROLLER] at 8.054 since launch
java.lang.IllegalArgumentException: Invalid time spec '10' (Valid examples: '3h', '4h30m')
at org.apache.pinot.spi.utils.TimeUtils.convertPeriodToMillis(TimeUtils.java:165) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
I don't see any config with value set to 10 other than below. Which is fine.
Copy code
# grep -inrw "10" /opt/pinot/apache-pinot-1.2.0-bin/conf/
/opt/pinot/apache-pinot-1.2.0-bin/conf/quickstart-log4j2.xml:36:   DefaultRolloverStrategy max="10"/
x
this seems to be issue of controller config, e.g. frequency or delay
The log4j strategy shouldn't matter here
m
still stuck, any idea what I should do here?
x
how did you start controller
what’s the controller config file
m
Here is the config for controller /opt/pinot/apache-pinot-1.2.0-bin/conf/pinot-controller.conf =============================================================== # Pinot Role pinot.service.role=CONTROLLER # Pinot Cluster name pinot.cluster.name=perf-hdppinotperf # Pinot Zookeeper Server pinot.zk.server=perf-pinotperfzk001.zirar.mihr:2181 # Use hostname as Pinot Instance ID other than IP pinot.set.instance.id.to.hostname=true # Pinot Controller Port controller.port=9000 # Pinot Controller VIP Host controller.vip.host=perf-pinotperfctl001.zirar.mihr # Pinot Controller VIP Port controller.vip.port=9000 #HDFS controller.data.dir=hdfs://pinotperf/pinot controller.local.temp.dir=/tmp/pinot pinot.controller.logger.root.dir=/var/log/pinot controller.enable.split.commit=true controller.allow.hlc.tables=false controller.realtime.segment.deepStoreUploadRetryEnabled=true pinot.controller.storage.factory.class.hdfs=org.apache.pinot.plugin.filesystem.HadoopPinotFS pinot.controller.storage.factory.hdfs.hadoop.conf.path=/usr/odp/3.2.3.3-3/hadoop/conf pinot.controller.segment.fetcher.protocols=hdfs pinot.controller.segment.fetcher.hdfs.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher pinot.server.grpc.enable=true Here is the start script for controller : /opt/pinot/apache-pinot-1.2.0-bin/conf/start-controller.sh ============================================================ export JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64 export PINOT_LOG_LOCATION=/var/log/pinot/ export ZOOKEEPER_ADDRESS="perf-pinotperfzk001.zirar.mihr:2181" export SERVER_CONF=/opt/pinot/apache-pinot-1.2.0-bin/conf/pinot-controller.conf export SERVER_CONF_DIR=/opt/pinot/apache-pinot-1.2.0-bin/conf/ export PINOT_DISTRIBUTION_DIR=/opt/pinot/apache-pinot-1.2.0-bin export JAVA_OPTS="-Xms12G -Xmx12G -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -Dplugins.dir=/opt/pinot/apache-pinot-1.2.0-bin/plugins -Dlog4j2.configurationFile=${SERVER_CONF_DIR}/pinot-controller-log4j2.xml" ${PINOT_DISTRIBUTION_DIR}/bin/pinot-admin.sh StartController -zkAddress ${ZOOKEEPER_ADDRESS} -configFileName ${SERVER_CONF} >> ${PINOT_LOG_LOCATION}/pinot.controller.log
This was all working before, this I started to get all of a sudden. Trying to figure what caused it.
x
can you paste full stacktrace?
v
just curious, it shouldnt have 2181 i guess
Copy code
# Pinot Controller VIP Host
controller.vip.host=perf-pinotperfctl001.zirar.mihr:2181
@Mannoj ^^
m
By bad, mistake in copy paste. Edited again, it is without 2181 in config of the server as well.
Copy code
2025/05/12 14:28:02.973 ERROR [StartServiceManagerCommand] [main] Failed to start a Pinot [CONTROLLER] at 41.66 since launch
java.lang.IllegalArgumentException: Invalid time spec '10' (Valid examples: '3h', '4h30m')
	at org.apache.pinot.spi.utils.TimeUtils.convertPeriodToMillis(TimeUtils.java:165) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
	at org.apache.pinot.controller.ControllerConf.convertPeriodToUnit(ControllerConf.java:1059) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
	at org.apache.pinot.controller.ControllerConf.convertPeriodToSeconds(ControllerConf.java:1063) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
	at org.apache.pinot.controller.ControllerConf.lambda$getTaskManagerFrequencyInSeconds$26(ControllerConf.java:747) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
	at java.base/java.util.Optional.map(Optional.java:265) ~[?:?]
	at org.apache.pinot.controller.ControllerConf.getTaskManagerFrequencyInSeconds(ControllerConf.java:747) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
	at org.apache.pinot.controller.helix.core.minion.PinotTaskManager.<init>(PinotTaskManager.java:120) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
	at org.apache.pinot.controller.BaseControllerStarter.setupControllerPeriodicTasks(BaseControllerStarter.java:825) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
	at org.apache.pinot.controller.BaseControllerStarter.setUpPinotController(BaseControllerStarter.java:484) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
	at org.apache.pinot.controller.BaseControllerStarter.start(BaseControllerStarter.java:378) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
	at org.apache.pinot.tools.service.PinotServiceManager.startController(PinotServiceManager.java:118) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
	at org.apache.pinot.tools.service.PinotServiceManager.startRole(PinotServiceManager.java:87) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
	at org.apache.pinot.tools.admin.command.StartServiceManagerCommand.lambda$startBootstrapServices$0(StartServiceManagerCommand.java:252) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
	at org.apache.pinot.tools.admin.command.StartServiceManagerCommand.startPinotService(StartServiceManagerCommand.java:305) [pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
	at org.apache.pinot.tools.admin.command.StartServiceManagerCommand.startBootstrapServices(StartServiceManagerCommand.java:251) [pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
	at org.apache.pinot.tools.admin.command.StartServiceManagerCommand.execute(StartServiceManagerCommand.java:195) [pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
	at org.apache.pinot.tools.admin.command.StartControllerCommand.execute(StartControllerCommand.java:188) [pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
	at org.apache.pinot.tools.Command.call(Command.java:33) [pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
	at org.apache.pinot.tools.Command.call(Command.java:29) [pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
	at picocli.CommandLine.executeUserObject(CommandLine.java:2045) [pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
	at picocli.CommandLine.access$1500(CommandLine.java:148) [pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
	at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2465) [pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
	at picocli.CommandLine$RunLast.handle(CommandLine.java:2457) [pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
	at picocli.CommandLine$RunLast.handle(CommandLine.java:2419) [pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
	at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2277) [pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
	at picocli.CommandLine$RunLast.execute(CommandLine.java:2421) [pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
	at picocli.CommandLine.execute(CommandLine.java:2174) [pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
	at org.apache.pinot.tools.admin.PinotAdministrator.execute(PinotAdministrator.java:173) [pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
	at org.apache.pinot.tools.admin.PinotAdministrator.main(PinotAdministrator.java:204) [pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
Caused by: java.lang.IllegalArgumentException: Invalid format: "10"
	at org.joda.time.format.PeriodFormatter.parseMutablePeriod(PeriodFormatter.java:331) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
	at org.joda.time.format.PeriodFormatter.parsePeriod(PeriodFormatter.java:309) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
	at org.apache.pinot.spi.utils.TimeUtils.convertPeriodToMillis(TimeUtils.java:161) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
	... 28 more
x
can you check if you have this config
controller.task.frequencyPeriod
in your controller conf and make it 10
configure it as 10m
the stacktrace shows this is configured as 10, which the default is -1 so pretty weird.
if cannot figure it out, then just configure it in your controller conf with some valid value
m
I dont have that setting, can I make a new one ? controller.task.frequencyPeriod="10m"
x
you can try
just add this line into your controller config
and please share the full log if possible
m
ack
damn it worked
its up 🙂
hardcoded like this -> controller.task.frequencyPeriod=10m
thanks @Xiang Fu 1. but from where or why it started to pick 10 ? as a default when I don't set any for that variable? 2. how were you able to get to that variable ? did it show anything on the log about that variable?
x
1. I don’t know, default is -1 so it should come from somewhere. Is there a config dump at the container level or can you enter into the container and cat the conf file again ?
2. read the source code and from the log stacktrace, it shows:
getTaskManagerFrequencyInSeconds
m
its a VM based architecture. But yes, whatver I shard above it he only config. file. thanks for that getTaskManagerFrequencyInSeconds will take a look at it.
x
ok, have you changed config or docker image ?
m
yes i added this config. it worked