if it's set to false then as soon as e.g. 'StartBroker' has completed, it does a sys.exit, which triggers the shutdown hook!

Got it... makes sense, thanks for the info!

@Trust Okoroego the trick above works for me, maybe you'll wanna try it yourself too

@Diogo Baeder Thanks. it worked for me too.

Awesome, good to know! 🙂 (^ @Mark Needham)

Hi Team,

while executing batch ingestion job using command - bin/pinot-ingestion-job.sh -jobSpecFile examples/batch/airlineStats/ingestionJobSpec.yaml under pinot-0.9.0 . It gives error - Error: Main method not found in class org.apache.pinot.tools.admin.command.LaunchDataIngestionJobCommand, please define the main method as:   public static void main(String[] args) or a JavaFX application class must extend javafx.application.Application

can you try this one:

bin/pinot-admin.sh LaunchDataIngestionJob -jobSpecFile examples/batch/airlineStats/ingestionJobSpec.yaml

I will take a look this script

this method is actually deprecated and should be replaced with

bin/pinot-admin.sh LaunchDataIngestionJob

I will update the doc

Thanks for bring this up! @Deepak Mishra

it is working . Thanks

Hi Team , I am working on backfill data using spark ingestion job and the segment name should be same which is stored under offline table (which is moved from realtime table using minion) , segment name is like tablename_wxc_18924_18924_0. while pushing backfill data using spark ingestion job , segment name is like tablename _OFFLINE_18924_18924_0 . In spark ingestion YAML file , for segmentNameGeneratorSpec: type :simple , i can’t use type- normalizedDate becuase it gives in different format. Is there any way to remove OFFLINE(tablename _OFFLINE_18924_18924_0) text from segment name

Hi everyone! ❤️ ⭐ ⭐ ⭐ 🔈🔈🔈 📅 👉 NEXT MONDAY (December 13th) 10:00 PST | 13:00 EST | 18:00 UTC We're inviting you all to join us for the 2021 Annual re-cap of Apache Pinot. We will also be opening up the floor for discussion of the future roadmap and would love for you to join the discussion! Please RSVP and attend in the meetup link below. We will be doing it on Zoom, so you will be able to chime in with voice/video or text on your thoughts and feedback. Thank you so much for being part of this movement with us, and we look forward to an awesome 2022!!!! 🤩 https://www.meetup.com/apache-pinot/events/282168447/ ⭐ ⭐ ⭐ ✅ ✅ ✅ Also, please VOTE on what you'd like to see in the future Apache Pinot roadmap! ✅ ✅ ✅ https://poll.ly/#/Pvy1rpp2 (poll closes Friday, December 10th) ❓ Questions, Comments, Concerns, Ideas? Feel free to DM me ❓ 😃

One quick question about instanceAssignmentConfigMap part of my upsert table configuration:

"instanceAssignmentConfigMap": {
    "CONSUMING": {
      "tagPoolConfig": {
        "tag": "Upsert_REALTIME",
        "poolBased": true
      },
      "replicaGroupPartitionConfig": {
        "replicaGroupBased": true,
        "numReplicaGroups": 3,
        "numInstancesPerPartition": 0,
        "numPartitions": 1
      }
    },
    "COMPLETED": {
      "tagPoolConfig": {
        "tag": "Upsert_REALTIME",
        "poolBased": true
      },
      "replicaGroupPartitionConfig": {
        "replicaGroupBased": true,
        "numReplicaGroups": 3,
        "numInstancesPerPartition": 0,
        "numPartitions": 1
      }
    }
  }

Why does the above configuration cause duplicate records in the upsert table? It will become good after changing “numInstancesPerPartition” from 0 to 1?

Hello everyone, i wonder that would setting maxLength property of STRING data types in schema to high values cause extra memory allocation or performance degradation?

Hi team ,facing issues while accessing pinto table in presto getting Query 20211214_044329_00009_tcxbr failed: java.net.SocketTimeoutException: Connect Timeout

Hi Team, @Mayank @Neha Pawar, Need a help. trying to start controller with following command. bin/pinot-admin.sh StartController -configFileName conf/pinot-controller.conf getting null pointer exception. Controller Config:

# Pinot Role
pinot.service.role=CONTROLLER

# Pinot Cluster name
pinot.cluster.name=MAX-Pinot

# Pinot Zookeeper Server
pinot.zk.server=host1:2181

# Use hostname as Pinot Instance ID other than IP
pinot.set.instance.id.to.hostname=true

# Pinot Controller Port
controller.port=9000

# Pinot Controller VIP Host
controller.vip.host=host1

# Pinot Controller VIP Port
controller.vip.port=9000

# Location to store Pinot Segments pushed from clients
controller.data.dir=<hdfs://nameservice1/data/max/poc/hdfs/controller>

controller.task.frequencyPeriod=3600
controller.local.temp.dir=/pinot/data/controller/tmp
controller.enable.split.commit=true
controller.access.protocols.http.port=9000
controller.helix.cluster.name=MAX-Pinot
pinot.controller.segment.fetcher.protocols=file,http,hdfs
pinot.controller.storage.factory.class.hdfs=org.apache.pinot.plugin.filesystem.HadoopPinotFS
pinot.controller.storage.factory.hdfs.hadoop.conf.path=/pinot/hadoop-3.0.0/etc/hadoop
pinot.controller.segment.fetcher.hdfs.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher
pinot.server.grpc.enable=true

Attaching Logs:

I have single node cluster(zk,cont, brok, server), where i configured hdfs as deep storage. so controller segments gets stored in hdfs.... this node got down. i added new node with same hdfs controller segments path... but its not getting downloaded to new node/controller...

please help me

https://docs.pinot.apache.org/basics/getting-started/hdfs-as-deepstorage - used this as ref

2021/12/15 200043.974 INFO [Fetcher] [kf_metrics_vertical__0__0__20211215T0006Z] [Consumer clientId=consumer-515, groupId=] Fetch offset 0 is out of range for partition kf_metrics_topic-0, resetting offset 2021/12/15 200043.974 INFO [Fetcher] [kf_metrics_vertical__0__0__20211215T0006Z] [Consumer clientId=consumer-515, groupId=] Resetting offset for partition kf_metrics_topic-0 to offset 2533.

we have tried changing the table config from smallest to largest it doesn’t make any difference: “stream.kafka.consumer.prop.auto.offset.reset”: “smallest”,

server instances keep getting disconnected with zk. i am not seeing anything in zk logs.

i am not seeing anything with broker or controller either. i am guessing that's because there is nothing to update from there, but i would expect something if all servers are getting disconnected with zk.

hi team