Hi! I'm trying to create an OFFLINE table in Pinot. This is my schema:

{
  "metricFieldSpecs": [],
  "primaryKeyColumns": [
    "device_id"
  ],
  "dimensionFieldSpecs": [
    {
      "name": "country",
      "dataType": "STRING"
    },
    {
      "name": "device_id",
      "dataType": "STRING"
    },
    {
      "name": "device_type",
      "dataType": "STRING"
    },
    {
      "name": "all_segments",
      "dataType": "INT",
      "singleValueField": false
    }
  ],
  "dateTimeFieldSpecs": [],
  "schemaName": "devices"
}

And my table:

{
  "tableName": "devices",
  "tableType": "OFFLINE",
  "tenants": {},
  "segmentsConfig": {
    "replication": "1"
  },
  "tableIndexConfig": {
    "loadMode": "MMAP",
    "createInvertedIndexDuringSegmentGeneration": true,
    "segmentPartitionConfig": {
      "columnPartitionMap": {
        "country": {
          "functionName": "Murmur",
          "numPartitions": 20
        }
      }
    }
  },
  "routing": {
    "segmentPrunerTypes": [
      "partition"
    ]
  },
  "fieldConfigList": [
    {
      "name": "all_segments",
      "indexes": {
        "forward": {
          "disabled": true
        },
        "inverted": {}
      }
    }
  ],
  "metadata": {
    "customConfigs": {}
  }
}

But the table is not added when I check the controller UI, I'm not receiving any errors:

bin/pinot-admin.sh AddTable -schemaFile schema.json -tableConfigFile table.json -controllerHost localhost -controllerPort 9000 -exec
WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance.
ERROR StatusLogger Reconfiguration failed: No configuration found for 'Default' at 'null' in 'null'
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.codehaus.groovy.reflection.CachedClass (file:/opt/pinot/lib/pinot-all-1.1.0-SNAPSHOT-jar-with-dependencies.jar) to method java.lang.Object.finalize()
WARNING: Please consider reporting this to the maintainers of org.codehaus.groovy.reflection.CachedClass
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release

I'm using the latest code from the current

master

branch

Hello Team, we have an id, sorted within each segment(created with time col as transaction_date) and it is unique across segments. is there any best way to stream records in globally sorted order by just streaming records?, since the segments are already order by time and within each segment the id is already sorted..

select * from table where id between 1 and 500000 order by id;

select * from table where id between 500001 and 1000000 order by id;

My understanding is that, at broker side it is being sorted again and causing latency Expectioning something to be done like keyset pagination efficiently. https://sureshdsk.dev/keyset-based-pagination-in-postgres

Hello, Team, Does pinot-client-go provide a method to control the number of connections to Pinot? I am using pinot-client-go to connect to Pinot. When I sent a large number of requests and put a load on it, I received an error from Pinot saying

read: connection reset by peer.

I think that is because it exceeded the simultaneous connection limit of Pinot. For example, when connecting from a Go client to MySQL, sql.DB provides MaxConnection and MaxIdleConnection, allowing control over the simultaneous connection count to the database. Does pinot-client-go provide similar methods?

Hello, team, have a question about add a new field (dimension, string type) into a realtime ingestion table, and reloaded all segments of that table(guide), could see new col in table, but all values are null, checked upstream kafka (data exist), no errors found in log, could you give some guidance of how to debug/fix this? (pinot 0.11)

Hi Team, Are there any restrictions in lookup() on looking up multi-value metadata from the dim table? Slightly modified this lookup join recipe to add a multi-value column called tags. Lookup join fails with an NPE.

Hi Team Looking for some help on enabling TLS in Pinot. I have made the configuration changes as mentioned in this doc https://docs.pinot.apache.org/operators/tutorials/configuring-tls-ssl#tls-only While running the controller, I am getting

java.lang.NullPointerException: trust store password required

Here is the controller configuration

# Pinot Role
pinot.service.role=CONTROLLER

# Pinot Cluster name
pinot.cluster.name=pinot-quickstart

# Pinot Zookeeper Server
pinot.zk.server=localhost:2181

# Use hostname as Pinot Instance ID other than IP
pinot.set.instance.id.to.hostname=true

# Pinot Controller Port
controller.port=9000

# Pinot Controller VIP Host
controller.vip.host=localhost

# Pinot Controller VIP Port
#controller.vip.port=9000

# Location to store Pinot Segments pushed from clients
controller.data.dir=/tmp/pinot/data/controller



# new configuration adding for tls setup
controller.tls.truststore.path=/root/pinot/truststore.jks
controller.tls.truststore.password=xxxxx

controller.access.protocols=https
controller.access.protocols.https.port=9443
controller.broker.protocol=https
controller.vip.protocol=https
controller.vip.port=9443

2023/11/29 23:04:11.071 ERROR [StartServiceManagerCommand] [main] Failed to start a Pinot [CONTROLLER] at 25.451 since launch
java.lang.NullPointerException: trust store password required
	at org.apache.pinot.shaded.com.google.common.base.Preconditions.checkNotNull(Preconditions.java:787) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
	at org.apache.pinot.core.util.ListenerConfigUtil.buildSSLEngineConfigurator(ListenerConfigUtil.java:268) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
	at org.apache.pinot.core.util.ListenerConfigUtil.configureListener(ListenerConfigUtil.java:238) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
	at org.apache.pinot.core.util.ListenerConfigUtil.lambda$buildHttpServer$8(ListenerConfigUtil.java:222) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
	at java.util.ArrayList.forEach(ArrayList.java:1541) ~[?:?]
	at org.apache.pinot.core.util.ListenerConfigUtil.buildHttpServer(ListenerConfigUtil.java:222) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
	at org.apache.pinot.controller.api.ControllerAdminApiApplication.start(ControllerAdminApiApplication.java:79) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
	at org.apache.pinot.controller.BaseControllerStarter.setUpPinotController(BaseControllerStarter.java:492) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
	at org.apache.pinot.controller.BaseControllerStarter.start(BaseControllerStarter.java:328) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
	at org.apache.pinot.tools.service.PinotServiceManager.startController(PinotServiceManager.java:118) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
	at org.apache.pinot.tools.service.PinotServiceManager.startRole(PinotServiceManager.java:87) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
	at org.apache.pinot.tools.admin.command.StartServiceManagerCommand.lambda$startBootstrapServices$0(StartServiceManagerCommand.java:251) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
	at org.apache.pinot.tools.admin.command.StartServiceManagerCommand.startPinotService(StartServiceManagerCommand.java:304) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
	at org.apache.pinot.tools.admin.command.StartServiceManagerCommand.startBootstrapServices(StartServiceManagerCommand.java:250) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
	at org.apache.pinot.tools.admin.command.StartServiceManagerCommand.execute(StartServiceManagerCommand.java:196) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
	at org.apache.pinot.tools.admin.command.StartControllerCommand.execute(StartControllerCommand.java:187) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
	at org.apache.pinot.tools.Command.call(Command.java:33) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
	at org.apache.pinot.tools.Command.call(Command.java:29) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]

Hi, is it possible to turn off upserts on a realtime table? Would this requiring migrating data to a new non-upsert table, or can an existing one be converted?

Hi all, is there a way to tell which segment/server a row comes from? We saw there are duplicates in our data and wanted to know where they come from

Hi! I'm trying to create pinot segments for an offline table from some parquet files in S3 but the container is terminating with exit code 137 every time, even though it has plenty memory available. I've deployed a single pinot controller in an EKS cluster with 16 cores and 128GB of RAM assigned to the pod. The job is reading thousands of parquet files from an S3 bucket (each parquet file is around 150MB): This is the job spec:

executionFrameworkSpec:
  name: 'standalone'
  segmentGenerationJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner'
  segmentTarPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentTarPushJobRunner'

jobType: SegmentCreationAndTarPush
inputDirURI: '<s3://sample-data/pinot-test/>'
includeFileNamePattern: 'glob:**/*.parquet'
outputDirURI: '<s3://sample-data/pinot-segments>'
overwriteOutput: true
segmentCreationJobParallelism: 10

pinotFSSpecs:
  - scheme: s3
    className: org.apache.pinot.plugin.filesystem.S3PinotFS
    configs:
      region: us-east-1

recordReaderSpec:
  dataFormat: 'parquet'
  className: 'org.apache.pinot.plugin.inputformat.parquet.ParquetRecordReader'
  configClassName: 'org.apache.pinot.plugin.inputformat.parquet.ParquetRecordReaderConfig'

tableSpec:
  tableName: 'devices'
pinotClusterSpecs:
  - controllerURI: '<http://localhost:9000>'

Then I run:

bin/pinot-admin.sh LaunchDataIngestionJob -jobSpecFile job.yml

JVM options:

-XX:ActiveProcessorCount=15 -Xms1G -Xmx64G -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -Xlog:gc*:file=/opt/pinot/gc-pinot-controller.log

Issue: Not getting all records after adding partitioning to Realtime table We added partitioning on a realtime table based on a ID column which has alphanumeric values. Now if we made queries containing

where id='something123'

then we don’t get all the data associated with it because it purge the segments (so partitioning is working) but it’s not quering all segments containing data. Specifications: 1. The hash function is Murmur 2. Number of Kafka partition and table matches. 3. I think Kafka also using Murmur but I am not sure. Also, does it matter that Kafka’s partitioning and Pinot’s table partitioning hash function should be the same? A possible solution could be to try out all hash functions but I want to know the cause of it more precisely. Any help?

Hi all! I have a question around observability of Pinot. How do I get the number of queries per second for server. Also, what is the timeline for queries count we get with

pinot_server_queries_Count

metrics?

Hi Guys, Using Pinot 1.0.0 Is there any specific order to start Minion in running Pinot Cluster? Could it be done after zk, controller, broker and all servers are up? I use this command to run:

bin/start-admin.sh StartMinion -zkAddress localhost:2182

process is running but then trying to perform query to ingest data I got error - please see attachment - what I did wrong? Additionally Minion is visible under zookeper browser -> Pinot Cluster -> Configs -> Participants, on the main dashboard minion task manager count is 1

Hi Team. cc. @sunny Peer Download Policy works in version 0.10.0, but fails to complete segment commit in version 1.0.0 We are looking for a solution. https://github.com/apache/pinot/blob/b6bdf6c9686b286a149d2d1aea4a385ee98f3e79/pino[…]ot/controller/helix/core/realtime/SegmentCompletionManager.java

Hi all, I started to see this error message in servers at random times:

ERROR [ServerSegmentCompletionProtocolHandler] [recommended__11__0__20231201T2000Z] Could not send request <http://pinot-controller-0:9000/segmentConsumed?reason=rowLimit&streamPartitionMsgOffset=6553207479&instance=Server_pinot-server-2.pinot-server-headless_8098&offset=-1&name=recommended__11__0__20231201T2000Z&rowCount=5000000&memoryUsedBytes=316670821>
org.apache.pinot.shaded.org.apache.http.conn.HttpHostConnectException: Connect to pinot-controller-0:9000 [pinot-controller-0/10.2.71.220] failed: Connection timed out (Connection timed out)

As far as I figured out, servers are not able to commit the realtime table segment that is full. That’s why servers are not able to consume any more events from the topic. Is it possible to increase the timeout duration for ServerSegmentCompletion ?

Hi All I was trying to use the QuickStart script which comes up with the distribution. I was able to start the cluster and ingest JSON data via a few Kafka topics as well. But when I am trying to use the Groovy scripts in the transformationConfigs of the ingestion Config, it throws me an error saying that Groovy scripts are disabled by default. I added the line

controller.disable.ingestion.groovy=false

to my config/pinot-controller.conf but the issue is still not resolved. Please note, that I am trying to use Groovy scripts via the QuickStart command and not by setting up a manual cluster i.e. starting ZK, Broker and Controller individually. Can someone guide me on what could be the issue here?

Hi All I was trying to use a Groovy function which is supposed to convert XML into JSON. something like :

def xmlSlurper = new groovy.xml.XmlSlurper();

but this seems to be failing giving an error - Invalid transform function. When I was debugging the code, seems like

FunctionEvaluatorFactory.getExpressionEvaluator();

is called and gives an error on my local i.e. unable to resolve class groovy.xml.XmlSlurper. Is it possible to import a libary in the groovy script?

Hi All I am facing another issue now so basically my use-case was to get an XML as the payload and extract the required tags out of it. I was trying to do the same by using Groovy functions like XmlSlurper with the following script :

Groovy({def xmlSlurper = new groovy.xml.XmlSlurper(); def object = xmlSlurper.parseText(payload); return object.dbEvent.@eventId.toString}, payload)

this was supposed to work as I tested this on a simple Groovy compiler. But when I try to have Pinot read this config and form a table, I am getting below error : [screenshot] can someone please advise on what am I missing?

Hi! I've created multiple segments using

jobType: SegmentCreation

and stored them in S3, now I want to upload the segments to Pinot, should I use

SegmentTarPush

or

SegmentMetadataPush

?

Hey all 👋 We have noticed that we have segment downtime for REALTIME tables when the

SegmentRelocator

task is running and moving COMPLETED segments. Is there any configuration that we are missing or is this possibly a bug? Table configuration in 🧵

Hi, I have date time fields as timestamp in parquet files in UTC timezone, after ingestion hours values are change due to time zone where Pinot machines are located. Are to preserve such behavior? Biz requirement is to show date time field in yyyy-MM-dd HHmmss manner. So in source file there is a date lets say 120000 after ingestion it is 070000 due to UTC-5 time zone. "dateTimeFieldSpec": [ { "name": "DateFeild", "dataType": "TIMESTAMP", "format": "1MILLISECONDSSIMPLE_DATE_FORMAT:yyyy-MM-dd HH;mm:ss", "granularity": "1:DAYS" } ]

hi, i have a quick question as i prefer to test backfilling some tables. i plan to use a real-time table only, not a hybrid table, since it is a real-time table that we just want to bootstrap historical values into. it seems that for this use case (no ongoing batch uploads), defining a hybrid table is unnecessary. does batch ingest (using spark in particular, though i think all methods use the same stuff under the hood) handle partitioning for upsert/dedupe realtime tables? i see support for uploading segments to real time tables, including upsert tables, was added (root issue: https://github.com/apache/pinot/issues/8283). learning about the flink connector and the limitations for upsert tables (getting the partition function just right etc), i’m curious if the batch segment creator reads the

segmentPartitionConfig

and does this automatically, if you have to do something manually when creating the input files for the batch, or none of the above. also related i guess, what happens if you add partitions to a kafka topic? presumably all sorts of complexities. anyway that’s more for my curiosity. i’m hoping we’ll stick to regular append-only tables for now but want to make sure i’m covered if we need any upsert tables.

Hi! I've created some segments using the

SegmentCreation

job and stored them in S3, I'm disabling the forward index in my largest MV column to save disk space as I only want to use an inverted index on it. After that I'm uploading the segments to the server with

SegmentMetadataPush

. I'm observing the uncompressed segments in the server are much bigger than the uncompressed data in S3. When comparing the metadata, this is what I get. Segment metadata in S3:

all_segments.dictionary.startOffset = 0
all_segments.dictionary.size = 56400
all_segments.inverted_index.startOffset = 56400
all_segments.inverted_index.size = 40826036
country.dictionary.startOffset = 40882436
country.dictionary.size = 10
country.forward_index.startOffset = 40882446
country.forward_index.size = 16
device_id.dictionary.startOffset = 40882462
device_id.dictionary.size = 476342
device_id.forward_index.startOffset = 41358804
device_id.forward_index.size = 49688
device_type.dictionary.startOffset = 41408492
device_type.dictionary.size = 15
device_type.forward_index.startOffset = 41408507
device_type.forward_index.size = 16

Segment metadata in server:

all_segments.dictionary.startOffset = 0
all_segments.dictionary.size = 56400
all_segments.inverted_index.startOffset = 56400
all_segments.inverted_index.size = 40826036
country.dictionary.startOffset = 40882436
country.dictionary.size = 10
country.forward_index.startOffset = 40882446
country.forward_index.size = 16
device_id.dictionary.startOffset = 40882462
device_id.dictionary.size = 476342
device_id.forward_index.startOffset = 41358804
device_id.forward_index.size = 49688
device_type.dictionary.startOffset = 41408492
device_type.dictionary.size = 15
device_type.forward_index.startOffset = 41408507
device_type.forward_index.size = 16
all_segments.forward_index.startOffset = 41408523
all_segments.forward_index.size = 149779126

The metadata in the server is identical to the metadata in S3, except a forward index is being created for the

all_segments

column (last 2 lines). Why is the server creating the forward index? Table definition in thread. This is with latest Pinot version.

i see below line a lot in the logs. what does it mean?

Instance pinot-controller-0.pinot-controller-headless.pinot.svc.cluster.local_9000 is not leader of cluster pinot-dev due to current session 20392a6522d0024 does not match leader session 304c4bf36fb0004

Hi! I'm trying to sort an array of integers during segment creation to be able to use a "sorted forward index". My config is:

"ingestionConfig": {
    "transformConfigs": [
        {
            "columnName": "all_segments_sorted",
            "transformFunction": "arraysortint(all_segments)"
        }
    ]
}

But when creating the segments for an offline table I'm getting this error:

Caused by: java.lang.ClassCastException: class java.util.HashMap cannot be cast to class java.lang.Number (java.util.HashMap and java.lang.Number are in module java.base of loader 'bootstrap')
	at org.apache.pinot.common.utils.PinotDataType$14.toInt(PinotDataType.java:770) ~[pinot-all-1.1.0-SNAPSHOT-jar-with-dependencies.jar:1.1.0-SNAPSHOT-b25f7cf6c165d9de39d79e86894235fca359adab]
	at org.apache.pinot.common.utils.PinotDataType.anyToInt(PinotDataType.java:1399) ~[pinot-all-1.1.0-SNAPSHOT-jar-with-dependencies.jar:1.1.0-SNAPSHOT-b25f7cf6c165d9de39d79e86894235fca359adab]
	at org.apache.pinot.common.utils.PinotDataType.toPrimitiveIntArray(PinotDataType.java:1009) ~[pinot-all-1.1.0-SNAPSHOT-jar-with-dependencies.jar:1.1.0-SNAPSHOT-b25f7cf6c165d9de39d79e86894235fca359adab]
	at org.apache.pinot.common.utils.PinotDataType$16.convert(PinotDataType.java:846) ~[pinot-all-1.1.0-SNAPSHOT-jar-with-dependencies.jar:1.1.0-SNAPSHOT-b25f7cf6c165d9de39d79e86894235fca359adab]
	at org.apache.pinot.common.utils.PinotDataType$16.convert(PinotDataType.java:843) ~[pinot-all-1.1.0-SNAPSHOT-jar-with-dependencies.jar:1.1.0-SNAPSHOT-b25f7cf6c165d9de39d79e86894235fca359adab]
	at org.apache.pinot.common.function.FunctionInvoker.convertTypes(FunctionInvoker.java:115) ~[pinot-all-1.1.0-SNAPSHOT-jar-with-dependencies.jar:1.1.0-SNAPSHOT-b25f7cf6c165d9de39d79e86894235fca359adab]
	at org.apache.pinot.segment.local.function.InbuiltFunctionEvaluator$FunctionExecutionNode.execute(InbuiltFunctionEvaluator.java:236) ~[pinot-all-1.1.0-SNAPSHOT-jar-with-dependencies.jar:1.1.0-SNAPSHOT-b25f7cf6c165d9de39d79e86894235fca359adab]
	at org.apache.pinot.segment.local.function.InbuiltFunctionEvaluator.evaluate(InbuiltFunctionEvaluator.java:114) ~[pinot-all-1.1.0-SNAPSHOT-jar-with-dependencies.jar:1.1.0-SNAPSHOT-b25f7cf6c165d9de39d79e86894235fca359adab]
	at org.apache.pinot.segment.local.recordtransformer.ExpressionTransformer.transform(ExpressionTransformer.java:123) ~[pinot-all-1.1.0-SNAPSHOT-jar-with-dependencies.jar:1.1.0-SNAPSHOT-b25f7cf6c165d9de39d79e86894235fca359adab]
	... 15 more

The input files are parquet files created by Spark

We created a new offline Pinot table. From the logs it seems that zookeeper restarted during loading of data via spark job. As a result some segments were in a bad state in the UI. I see them ONLINE in the ideal state, and don't see them in the external view in zookeeper browser via the UI. I tried reloading these bad segments, as well as resetting the bad segments. However I see that both reload and reset were skipped in the logs. I also see in the zookeeper browser a message on the server for that specific segment. We also tried deleting the table and re-creating and loading the data again and it still results in the same situation. Creating a new table with a different name works perfectly fine. I am wondering if the message against the server is causing both reload and reset to be skipped? Should I just delete the message? path of the message is

/INSTANCES/Server_pinot-dev-server-12.pinot-dev-server-headless.pinot.svc.cluster.local_8098/MESSAGES/4fdb8c7d-cc74-4744-8889-4bf8b71b1a15

{
  "id": "4fdb8c7d-cc74-4744-8889-4bf8b71b1a15",
  "simpleFields": {
    "CREATE_TIMESTAMP": "1701467131938",
    "ClusterEventName": "IdealStateChange",
    "EXE_SESSION_ID": "102e9911d3d0022",
    "FROM_STATE": "OFFLINE",
    "MSG_ID": "4fdb8c7d-cc74-4744-8889-4bf8b71b1a15",
    "MSG_STATE": "read",
    "MSG_TYPE": "STATE_TRANSITION",
    "PARTITION_NAME": "table_main_OFFLINE_1",
    "READ_TIMESTAMP": "1701467131949",
    "RESOURCE_NAME": "table_main_OFFLINE",
    "RESOURCE_TAG": "table_main_OFFLINE",
    "RETRY_COUNT": "3",
    "SRC_NAME": "pinot-dev-controller-2.pinot-dev-controller-headless.pinot.svc.cluster.local_9000",
    "SRC_SESSION_ID": "304c4bf36fb0004",
    "STATE_MODEL_DEF": "SegmentOnlineOfflineStateModel",
    "STATE_MODEL_FACTORY_NAME": "DEFAULT",
    "TGT_NAME": "Server_pinot-dev-server-12.pinot-dev-server-headless.pinot.svc.cluster.local_8098",
    "TGT_SESSION_ID": "102e9911d3d0022",
    "TO_STATE": "ONLINE"
  },
  "mapFields": {},
  "listFields": {}
}

Insight needed: Pinot UI question by a Typescript novice How do changes to ./js/main.js found at webapp/js/main.js get generated? Looks like main.js is a defined output in webpack.config.js What’s the step to build the UI React and javascript packages in development and to see those changes reflected on the user interface? Thanks

How do i get rid of this log line? seeing a plethora of this one in logs

2023-12-07 14:41:44.401	
Opening socket connection to server zookeeper.pinot.svc.cluster.local/172.20.11.129:2181. Will not attempt to authenticate using SASL (unknown error)

Hi all! We tried upgrading pinot in prod from 0.12.1 to 1.0.0 but the upgrade failed. The servers do not show any log and keep restarting every 3-4 min. Meanwhile the queries simple keep on returning "x segments not found". The resource utilization does not look like a problem. Zookeeper resource utilization does not look like a problem either. On 64gb nodes, pinot server pods barely occupy 7gb and keep restarting. Tried increasing the xms and xmx values but to no avail. Please help shed some light on what would be happening, how to upgrade. We successfully managed to upgrade pinot in our other environments where the clusters were a lot smaller. Simply by running helm upgrade. Thank you!!

Hello everyone. I have an issue on pinot and I'm unable to figure out the issue as to what is causing it. I have installed pinot referring to standard documentation on AWS EKS using helm charts (https://docs.pinot.apache.org/v/release-0.10.0/basics/getting-started/kubernetes-quickstart). It was running fine until couple days ago. The dev team reported that the pinot was not able to consume data from kafka and some of the records were missing as well. They were about 5 million records or so is what the team says. Prior to this they were consuming smaller records and it was all fine. Tried troubleshooting the issue but could not really get any fix for that and I'm stuck at this point. There are about 16 pinot server pods that I have created and

pods pinot-server-7

and

pinot-server-8

are always the ones that gets killed and restarts. Checking the logs on these servers this is what I see below. Any help here would be really beneficial. I'm not a developer (but an infrastructure engineer). So its quite difficult to get into the application level debugging for me.

javax.management.InstanceAlreadyExistsException: kafka.consumer:type=app-info,id=delivery_aggregation_REALTIME-7now-aggregation-0
	at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:436) ~[?:?]
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1855) ~[?:?]
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:955) ~[?:?]
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:890) ~[?:?]
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:320) ~[?:?]
	at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522) ~[?:?]
	at org.apache.pinot.shaded.org.apache.kafka.common.utils.AppInfoParser.registerAppInfo(AppInfoParser.java:64) [pinot-confluent-avro-1.1.0-SNAPSHOT-shaded.jar:1.1.0-SNAPSHOT-d7dc4c5eff6af90bb1e1385276b70e4685137cc0]
	at org.apache.pinot.shaded.org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:814) [pinot-confluent-avro-1.1.0-SNAPSHOT-shaded.jar:1.1.0-SNAPSHOT-d7dc4c5eff6af90bb1e1385276b70e4685137cc0]
	at org.apache.pinot.shaded.org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:665) [pinot-confluent-avro-1.1.0-SNAPSHOT-shaded.jar:1.1.0-SNAPSHOT-d7dc4c5eff6af90bb1e1385276b70e4685137cc0]
	at org.apache.pinot.shaded.org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:646) [pinot-confluent-avro-1.1.0-SNAPSHOT-shaded.jar:1.1.0-SNAPSHOT-d7dc4c5eff6af90bb1e1385276b70e4685137cc0]
	at org.apache.pinot.shaded.org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:626) [pinot-confluent-avro-1.1.0-SNAPSHOT-shaded.jar:1.1.0-SNAPSHOT-d7dc4c5eff6af90bb1e1385276b70e4685137cc0]
	at org.apache.pinot.plugin.stream.kafka20.KafkaPartitionLevelConnectionHandler.createConsumer(KafkaPartitionLevelConnectionHandler.java:84) [pinot-kafka-2.0-1.1.0-SNAPSHOT-shaded.jar:1.1.0-SNAPSHOT-d7dc4c5eff6af90bb1e1385276b70e4685137cc0]
	at org.apache.pinot.plugin.stream.kafka20.KafkaPartitionLevelConnectionHandler.<init>(KafkaPartitionLevelConnectionHandler.java:70) [pinot-kafka-2.0-1.1.0-SNAPSHOT-shaded.jar:1.1.0-SNAPSHOT-d7dc4c5eff6af90bb1e1385276b70e4685137cc0]
	at org.apache.pinot.plugin.stream.kafka20.KafkaStreamMetadataProvider.<init>(KafkaStreamMetadataProvider.java:59) [pinot-kafka-2.0-1.1.0-SNAPSHOT-shaded.jar:1.1.0-SNAPSHOT-d7dc4c5eff6af90bb1e1385276b70e4685137cc0]
	at org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory.createPartitionMetadataProvider(KafkaConsumerFactory.java:35) [pinot-kafka-2.0-1.1.0-SNAPSHOT-shaded.jar:1.1.0-SNAPSHOT-d7dc4c5eff6af90bb1e1385276b70e4685137cc0]
	at org.apache.pinot.core.data.manager.realtime.RealtimeSegmentDataManager.createPartitionMetadataProvider(RealtimeSegmentDataManager.java:1673) [pinot-all-1.1.0-SNAPSHOT-jar-with-dependencies.jar:1.1.0-SNAPSHOT-d7dc4c5eff6af90bb1e1385276b70e4685137cc0]
	at org.apache.pinot.core.data.manager.realtime.RealtimeSegmentDataManager.<init>(RealtimeSegmentDataManager.java:1494) [pinot-all-1.1.0-SNAPSHOT-jar-with-dependencies.jar:1.1.0-SNAPSHOT-d7dc4c5eff6af90bb1e1385276b70e4685137cc0]
	at org.apache.pinot.core.data.manager.realtime.RealtimeTableDataManager.addSegment(RealtimeTableDataManager.java:442) [pinot-all-1.1.0-SNAPSHOT-jar-with-dependencies.jar:1.1.0-SNAPSHOT-d7dc4c5eff6af90bb1e1385276b70e4685137cc0]
	at org.apache.pinot.server.starter.helix.HelixInstanceDataManager.addRealtimeSegment(HelixInstanceDataManager.java:229) [pinot-all-1.1.0-SNAPSHOT-jar-with-dependencies.jar:1.1.0-SNAPSHOT-d7dc4c5eff6af90bb1e1385276b70e4685137cc0]
	at org.apache.pinot.server.starter.helix.SegmentOnlineOfflineStateModelFactory$SegmentOnlineOfflineStateModel.onBecomeConsumingFromOffline(SegmentOnlineOfflineStateModelFactory.java:80) [pinot-all-1.1.0-SNAPSHOT-jar-with-dependencies.jar:1.1.0-SNAPSHOT-d7dc4c5eff6af90bb1e1385276b70e4685137cc0]
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]

Hi, What is the correct syntax for fieldConfigList.? We are using release-1.0.0 and want a sorted inverse index on the column. The following table configuration returns 400 Bad Request with ”null“ as error .

{
  "tableName": "xxx",
  "tableType": "OFFLINE",
  "segmentsConfig": {
    "retentionTimeUnit": "DAYS",
    "retentionTimeValue": "3650",
    "schemaName": "xxx",
    "replication": "3",
    "timeColumnName": "time"
  },
  "fieldConfigList": [
    {
      "name": "location",
      "indexes": {
        "inverted": {}
      }
    }
  ],
  "ingestionConfig": {},
  "tenants": {},
  "tableIndexConfig": {
    "loadMode": "MMAP",
    "nullHandlingEnabled": "true",
    "createInvertedIndexDuringSegmentGeneration": true,
    "sortedColumn": [
      "location"
    ]
  },
  "metadata": {
    "customConfigs": {}
  }
}