https://pinot.apache.org/ logo
Join Slack
Powered by
# getting-started
  • p
    Hello, we are planning to query pinot tables via airflow dags. We found this airflow connector. Is it advised to use the connector or should we use pinotdb? Are there any pros and cons between pinotdb api and sql alchemy?
    m
    n
    • 3
    • 2
  • f
    Hello, can anyone tell how can we connect spark streaming to Apache pinot?
    m
    • 2
    • 3
  • h
    I have a table where eventTime is sent in kafka as unix epoch. What setting I need so the data is displayed and query in IST?
    n
    • 2
    • 1
  • y
    Hello, need to connect pinot with our java/kotlin spring boot application 1. Do we have pooling options while using JDBC
    k
    f
    • 3
    • 4
  • a
    Hi, I have a tricky question. “NAN” values are stored in my realtime type table. My table config is like below. And the value in Kafka message is like {“key”{“price value”“NAN”}}. Is there some way I can use to filter this NAN messages in the ingestion phrase or in the query phrase? “metricFieldSpecs”: [ { “name”: “price_value”, “dataType”: “DOUBLE” } ]
    k
    • 2
    • 1
  • g
    Hi I am trying to understand the size difference between generated segments in deep storage vs segments in local disk. It seems after the ingestion job, segments for this table in s3 is close to 700GB, but the table size reported in pinot is around 4TB in UI (so 2TB for one data copy as we have replication factor 2). I wonder if this is expected? If it is, what is the main reason causing size difference? Is the data compressed in s3 and uncompressed in local?
    n
    n
    • 3
    • 4
  • g
    Hi team, is the output generated by “Reload Status” button in UI shows what are the indexes we have on the table right now? I wonder how do we determine what index it will have for each column by default? Asking because I didn’t specify any index column in table index config so I did not expect to see index for columns, but it seems like some of them are green
    m
    • 2
    • 2
  • a
    Hi, I have a question about time-related columns in the default StarTree index configuration. The documentation says:
    Here we assume that time columns will be included in most queries as the range filter column and/or the group by column, so for better performance, we always include them as the last elements in the dimensionsSplitOrder
    How does putting time columns as the last elements improve performance? Is it related to the number of nodes that need to be processed to solve a given query?
    n
    j
    • 3
    • 3
  • c
    Hi. Pinot Team! I am testing an injection job using a 'org.apache.pinot.plugin.inputformat.parquet.ParquetRecordReader'. My problem is that the binary type of parquet file is not converted to the string type of pinot. The data that appears as a query in the pinot is like a hex string. What's the problem? -- query result "resultTable": { "dataSchema": { "columnNames": [ "firstname", "gender", "lastname", "score", "studentid", "subject", "timestampinepoch" ], "columnDataTypes": [ "STRING", "STRING", "STRING", "INT", "INT", "STRING", "LONG" ] }, "rows": [ [ "4e6174616c6965", "46656d616c65", "4a6f6e6573", 3, 109, "4d61746873", 1647980000000 ] ] }, -- parquet file schema parquet-tools schema 6a4e9212ba501d90-c3a971300000000_1596454343_data.0.parq message schema { optional int32 studentid; optional binary firstname; optional binary lastname; optional binary gender; optional binary subject; optional int32 score; optional int64 timestampinepoch; } -- injestion job log read value: {"studentid": 109, "firstname": "Natalie", "lastname": "Jones", "gender": "Female", "subject": "Maths", "score": 3, "timestampinepoch": 1647980000000} Start building IndexCreator! Finished records indexing in IndexCreator! FileName set to metadata.properties Base path set to /tmp/pinot-6a2e3b81-8eda-40c9-9a53-0d9cc03c85fd/output/tmp-8b8f9c6b-6a22-41d9-a16d-3eefe3d75d81 Finished segment seal! Converting segment: /tmp/pinot-6a2e3b81-8eda-40c9-9a53-0d9cc03c85fd/output/batch_2022-03-22_2022-03-22 to v3 format FileName set to metadata.properties -- table { "tableName": "transcript", "tableType": "OFFLINE", "segmentsConfig": { "schemaName": "transcript", "replication": 3, "timeColumnName": "timestampinepoch", "timeType": "MILLISECONDS" }, "tenants": { "broker":"DefaultTenant", "server":"DefaultTenant" }, "tableIndexConfig": { "loadMode": "MMAP" }, "ingestionConfig": { "batchIngestionConfig": { "segmentIngestionType": "APPEND", "segmentPushFrequency": "DAILY" } }, "metadata": {} } -- schema { "schemaName": "transcript", "dimensionFieldSpecs": [ { "name": "studentid", "dataType": "INT" }, { "name": "firstname", "dataType": "STRING" }, { "name": "lastname", "dataType": "STRING" }, { "name": "gender", "dataType": "STRING" }, { "name": "subject", "dataType": "STRING" } ], "metricFieldSpecs": [ { "name": "score", "dataType": "INT" } ], "dateTimeFieldSpecs": [{ "name": "timestampinepoch", "dataType": "LONG", "format" : "1MILLISECONDSEPOCH", "granularity": "1:MILLISECONDS" } ] }
    k
    • 2
    • 7
  • b
    Hello, I'm just getting started and wondering if there is any documentation on how to use the pulsar plugin for stream ingestion?
    m
    k
    m
    • 4
    • 4
  • h
    One question - select count(*) from table where pipeline=‘TRANSACTIONAL’ and eventTime > ‘2022-05-04 221253.791’ -> this is giving count 10-20K But if i use following to check the plan: explain plan for select count(*) from table where pipeline=‘TRANSACTIONAL’ and eventTime > ‘2022-05-04 221253.791’
    m
    m
    +3
    • 6
    • 21
  • h
    Why it does not show the plan details
  • m
    Limit does not apply to aggr without group by. Also 0 and 1 are operator id. So still unclear on the question
  • m
    But yes @Amrish Lal for what’s the purpose of showing limit 10
  • g
    Hi team, I would like to understand if Pinot has certain query caching/warm up mechanism behind the scene? Asking because I noticed that the first run of a query is always the slowest, for example when I run a count group by query against a table for first time it takes 3000ms, but if I run it again in next couple minutes, the same query consistently taking less than 100ms.
    m
    • 2
    • 8
  • a
    Hi, I’m using Pinot upsert mode for one table and already stored some days’ data. If I change primaryKeyColumns config in schema, such as adding or deleting some filed, do I need to delete and upload this schema and table config to make this change take effect?
    j
    k
    • 3
    • 3
  • n
    Hi all, Im evaluating Pinot for realtime analytics for our feature on mobile app. The total record is about 50-100K transactions/day. Is it good to adopt Pinot, Im afraid of too much engineering on it.
    k
    m
    • 3
    • 9
  • l
    Could anyone help me with how to set pinot data in ThirdEye configuration? https://thirdeye.readthedocs.io/en/latest/index.html It's showing the example configuration with MySQL, but my data is Pinot data. I'm trying to set csv data(I've already installed Pinot docker and added this csv as dataset/table) in thirdeye dashboard. How can I change the configurations for this? Thanks in advance.
    m
    s
    • 3
    • 4
  • f
    Hi all,
  • f
    I am looking for a spark connector for writing the data to pinot. I saw on github, that write support will be availible in the future. Do you have any news about it or tips how to deal it? Thank you in advance !
    k
    • 2
    • 2
  • r
    Hello Team, I'm new to Apache Pinot I'm have setup my Apache pinot cluster in my local laptop using KinD and Helm My question is: What is the best way to automate the upload of new schema, table and job (realtime & batch ingestion) files to pinot ?
    m
    • 2
    • 1
  • g
    hey! my team an I are researching how to implement ML and analytics into our pipeline! It could end up being a bottleneck... what would be goods ideas for scaling, placement, and formatting Apache pinot for ML tasks?
    m
    x
    • 3
    • 2
  • f
    Hi little getting started question as I need more preprocesing on my fields. I’m facing a data error where I need to replace a few empty string to default date let say 2999. I’m ingestion the majority of my data using JSONPATH and I can’t directly use groovy as my cols names contains “.” anyone have face this kind of issue ? By empty I mean the field exist in my json so it’s not interpreted as null
    m
    a
    • 3
    • 4
  • k
    Hi all new here to the pinot community and I am starting to play with ingesting data. I have a data set in parquet that is type STRING and ARRAY<STRING>... the documentation is a bit confusing as to handle this. My question based on the docs is when I define a schema does it mean that there is no definition for ARRAY and I should define it as a string and depend on the table config to transform the array into a json type field and use a json index? My goal is to do multiple filters based on elements in the array, in this situation would it be better to exploded the records before pushing it to pinot?
    k
    • 2
    • 1
  • f
    Hi all, I want to explore the pinot spark connector. Do we have docs on the connector? Thanks
  • m
    Hello team, Just started exploring Pinot and I am new to the community here. I am evaluating Pinot for a schema change and I had a couple of questions. Was wondering if any one has thoughts on these: 1. Array of JSON objects or single JSON with array of objects? Which is faster, smaller or easier to query in a Pinot table? 2. If I have a bunch of fields which will not change frequently, does it make sense to have them in a JSON object OR as columns in a PINOT table in terms of query performance?
    m
    j
    • 3
    • 4
  • m
    Hello team, I am trying to bring up a pinot cluster on a gcp ubuntu 22.04 machine. I am able to bring up a docker container with all of pinot running but when I try to use the docker-compose here, I keep getting zookeeper cleanup exceptions.
    Copy code
    manual-pinot-server | 2022/06/17 13:46:34.432 INFO [StartServiceManagerCommand] [main] Started Pinot [SERVICE_MANAGER] instance [ServiceManager_12b62b9ab52c_-1] at 0.296s since launch
manual-pinot-server | 2022/06/17 13:46:34.434 INFO [StartServiceManagerCommand] [Start a Pinot [SERVER]] Starting a Pinot [SERVER] at 0.298s since launch
manual-pinot-controller | 2022/06/17 13:47:02.739 ERROR [StartServiceManagerCommand] [main] Failed to start a Pinot [CONTROLLER] at 30.356 since launch
manual-pinot-controller | java.lang.NullPointerException: null
manual-pinot-controller | 	at org.apache.helix.manager.zk.client.ZkConnectionManager.cleanupInactiveWatchers(ZkConnectionManager.java:112) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb]
manual-pinot-controller | 	at org.apache.helix.manager.zk.client.ZkConnectionManager.close(ZkConnectionManager.java:95) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb]
manual-pinot-controller | 	at org.apache.helix.manager.zk.client.ZkConnectionManager.close(ZkConnectionManager.java:91) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb]
manual-pinot-controller | 	at org.apache.helix.manager.zk.zookeeper.ZkClient.connect(ZkClient.java:1620) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb]
manual-pinot-controller | 	at org.apache.helix.manager.zk.zookeeper.ZkClient.<init>(ZkClient.java:186) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb]
manual-pinot-controller | 	at org.apache.helix.manager.zk.ZkClient.<init>(ZkClient.java:87) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb]
manual-pinot-controller | 	at org.apache.helix.manager.zk.client.ZkConnectionManager.<init>(ZkConnectionManager.java:41) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb]
manual-pinot-controller | 	at org.apache.helix.manager.zk.client.SharedZkClientFactory.getOrCreateZkConnectionNamanger(SharedZkClientFactory.java:60) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb]
manual-pinot-controller | 	at org.apache.helix.manager.zk.client.SharedZkClientFactory.buildZkClient(SharedZkClientFactory.java:40) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb]
manual-pinot-controller | 	at org.apache.pinot.common.utils.ServiceStartableUtils.applyClusterConfig(ServiceStartableUtils.java:54) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb]
manual-pinot-controller | 	at org.apache.pinot.controller.BaseControllerStarter.init(BaseControllerStarter.java:162) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb]
manual-pinot-controller | 	at org.apache.pinot.tools.service.PinotServiceManager.startController(PinotServiceManager.java:121) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb]
manual-pinot-controller | 	at org.apache.pinot.tools.service.PinotServiceManager.startRole(PinotServiceManager.java:91) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb]
manual-pinot-controller | 	at org.apache.pinot.tools.admin.command.StartServiceManagerCommand.lambda$startBootstrapServices$0(StartServiceManagerCommand.java:249) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb]
manual-pinot-controller | 	at org.apache.pinot.tools.admin.command.StartServiceManagerCommand.startPinotService(StartServiceManagerCommand.java:302) [pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb]
manual-pinot-controller | 	at org.apache.pinot.tools.admin.command.StartServiceManagerCommand.startBootstrapServices(StartServiceManagerCommand.java:248) [pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb]
manual-pinot-controller | 	at org.apache.pinot.tools.admin.command.StartServiceManagerCommand.execute(StartServiceManagerCommand.java:195) [pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb]
manual-pinot-controller | 	at org.apache.pinot.tools.admin.command.StartControllerCommand.execute(StartControllerCommand.java:152) [pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb]
manual-pinot-controller | 	at org.apache.pinot.tools.Command.call(Command.java:33) [pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb]
manual-pinot-controller | 	at org.apache.pinot.tools.Command.call(Command.java:29) [pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb]
manual-pinot-controller | 	at picocli.CommandLine.executeUserObject(CommandLine.java:1953) [pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb]
manual-pinot-controller | 	at picocli.CommandLine.access$1300(CommandLine.java:145) [pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb]
manual-pinot-controller | 	at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2352) [pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb]
manual-pinot-controller | 	at picocli.CommandLine$RunLast.handle(CommandLine.java:2346) [pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb]
manual-pinot-controller | 	at picocli.CommandLine$RunLast.handle(CommandLine.java:2311) [pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb]
manual-pinot-controller | 	at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2179) [pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb]
manual-pinot-controller | 	at picocli.CommandLine.execute(CommandLine.java:2078) [pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb]
manual-pinot-controller | 	at org.apache.pinot.tools.admin.PinotAdministrator.execute(PinotAdministrator.java:161) [pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb]
manual-pinot-controller | 	at org.apache.pinot.tools.admin.PinotAdministrator.main(PinotAdministrator.java:192) [pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb]
    Any thoughts on what could be going on here? Just trying to get started here.
    k
    • 2
    • 4
  • m
    Copy code
    manual-pinot-controller exited with code 0
manual-pinot-controller | SLF4J: Class path contains multiple SLF4J bindings.
manual-pinot-controller | SLF4J: Found binding in [jar:file:/opt/pinot/lib/pinot-all-0.9.3-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
manual-pinot-controller | SLF4J: Found binding in [jar:file:/opt/pinot/plugins/pinot-input-format/pinot-parquet/pinot-parquet-0.9.3-shaded.jar!/org/slf4j/impl/StaticLoggerBinder.class]
manual-pinot-controller | SLF4J: Found binding in [jar:file:/opt/pinot/plugins/pinot-file-system/pinot-s3/pinot-s3-0.9.3-shaded.jar!/org/slf4j/impl/StaticLoggerBinder.class]
manual-pinot-controller | SLF4J: Found binding in [jar:file:/opt/pinot/plugins/pinot-environment/pinot-azure/pinot-azure-0.9.3-shaded.jar!/org/slf4j/impl/StaticLoggerBinder.class]
manual-pinot-controller | SLF4J: Found binding in [jar:file:/opt/pinot/plugins/pinot-metrics/pinot-dropwizard/pinot-dropwizard-0.9.3-shaded.jar!/org/slf4j/impl/StaticLoggerBinder.class]
manual-pinot-controller | SLF4J: Found binding in [jar:file:/opt/pinot/plugins/pinot-metrics/pinot-yammer/pinot-yammer-0.9.3-shaded.jar!/org/slf4j/impl/StaticLoggerBinder.class]
manual-pinot-controller | SLF4J: See <http://www.slf4j.org/codes.html#multiple_bindings> for an explanation.
manual-pinot-controller | SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
manual-pinot-controller | WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance.
manual-pinot-controller | WARNING: An illegal reflective access operation has occurred
manual-pinot-controller | WARNING: Illegal reflective access by org.codehaus.groovy.reflection.CachedClass (file:/opt/pinot/lib/pinot-all-0.9.3-jar-with-dependencies.jar) to method java.lang.Object.finalize()
manual-pinot-controller | WARNING: Please consider reporting this to the maintainers of org.codehaus.groovy.reflection.CachedClass
manual-pinot-controller | WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
manual-pinot-controller | WARNING: All illegal access operations will be denied in a future release
manual-pinot-controller | 2022/06/17 13:47:04.217 INFO [StartControllerCommand] [main] Executing command: StartController -clusterName PinotCluster -controllerHost null -controllerPort 9000 -dataDir /tmp/data/PinotController -zkAddress manual-zookeeper:7000
manual-pinot-controller | 2022/06/17 13:47:04.226 INFO [StartServiceManagerCommand] [main] Executing command: StartServiceManager -clusterName PinotCluster -zkAddress manual-zookeeper:7000 -port -1 -bootstrapServices []
manual-pinot-controller | 2022/06/17 13:47:04.227 INFO [StartServiceManagerCommand] [main] Starting a Pinot [SERVICE_MANAGER] at 0.302s since launch
manual-pinot-controller | 2022/06/17 13:47:04.229 INFO [StartServiceManagerCommand] [main] Started Pinot [SERVICE_MANAGER] instance [ServiceManager_301ef6dd9be9_-1] at 0.305s since launch
manual-pinot-controller | 2022/06/17 13:47:04.229 INFO [StartServiceManagerCommand] [main] Starting a Pinot [CONTROLLER] at 0.305s since launch
manual-pinot-broker | 2022/06/17 13:47:04.474 ERROR [PinotServiceManager] [Start a Pinot [BROKER]] Failed to initialize Pinot Broker Starter
manual-pinot-broker | java.lang.NullPointerException: null
manual-pinot-broker | 	at org.apache.helix.manager.zk.client.ZkConnectionManager.cleanupInactiveWatchers(ZkConnectionManager.java:112) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb]
manual-pinot-broker | 	at org.apache.helix.manager.zk.client.ZkConnectionManager.close(ZkConnectionManager.java:95) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb]
manual-pinot-broker | 	at org.apache.helix.manager.zk.client.ZkConnectionManager.close(ZkConnectionManager.java:91) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb]
manual-pinot-broker | 	at org.apache.helix.manager.zk.zookeeper.ZkClient.connect(ZkClient.java:1620) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb]
manual-pinot-broker | 	at org.apache.helix.manager.zk.zookeeper.ZkClient.<init>(ZkClient.java:186) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb]
manual-pinot-broker | 	at org.apache.helix.manager.zk.ZkClient.<init>(ZkClient.java:87) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb]
manual-pinot-broker | 	at org.apache.helix.manager.zk.client.ZkConnectionManager.<init>(ZkConnectionManager.java:41) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb]
manual-pinot-broker | 	at org.apache.helix.manager.zk.client.SharedZkClientFactory.getOrCreateZkConnectionNamanger(SharedZkClientFactory.java:60) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb]
manual-pinot-broker | 	at org.apache.helix.manager.zk.client.SharedZkClientFactory.buildZkClient(SharedZkClientFactory.java:40) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb]
manual-pinot-broker | 	at org.apache.pinot.common.utils.ServiceStartableUtils.applyClusterConfig(ServiceStartableUtils.java:54) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb]
manual-pinot-broker | 	at org.apache.pinot.broker.broker.helix.BaseBrokerStarter.init(BaseBrokerStarter.java:112) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb]
manual-pinot-broker | 	at org.apache.pinot.tools.service.PinotServiceManager.startBroker(PinotServiceManager.java:141) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb]
manual-pinot-broker | 	at org.apache.pinot.tools.service.PinotServiceManager.startRole(PinotServiceManager.java:96) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb]
manual-pinot-broker | 	at org.apache.pinot.tools.admin.command.StartServiceManagerCommand$1.lambda$run$0(StartServiceManagerCommand.java:276) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb]
manual-pinot-broker | 	at org.apache.pinot.tools.admin.command.StartServiceManagerCommand.startPinotService(StartServiceManagerCommand.java:302) [pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb]
manual-pinot-broker | 	at org.apache.pinot.tools.admin.command.StartServiceManagerCommand$1.run(StartServiceManagerCommand.java:276) [pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb]
manual-pinot-broker | 2022/06/17 13:47:04.485 ERROR [StartServiceManagerCommand] [Start a Pinot [BROKER]] Failed to start a Pinot [BROKER] at 30.351 since launch
  • p
    Hi all, I've read the documentation around pinot star-tree index (https://docs.pinot.apache.org/basics/indexing/star-tree-index) and was thinking on how to define the set of dimensions in the Dimensions Split Order. We currently have a ES solution that is used for real-time dashboarding and a future need for real-time analytics. I can of course go over the current queries executed on ES and work from there. We can do use needs gathering on the future requirements for the real-time analytics. This will lead to an ordered list of dimensions. Does one now add all these dimensions to the "dimensions split order" and leave it up to pinot to stop the tree splits when the leave contains no more than T number of documents, or does one typically limit the number of dimensions by only adding a certain number of dimensions in the 'dimensions split order'? If the latter is the case, how do you know how many to add, and is there a reason not to add all dimensions to 'dimensions split order'? Also, what happens when you have more than T documents in a leaf node: if one has put all dimensions in 'dimensions split order', I assume there is no further actions that can be taken? Else of course, one could add a dimension to 'dimensions split order'. Is the latter an easy fix or does it require reindexing all documents?
    k
    p
    • 3
    • 9
  • m
    I have a Pinot cluster running in kubernetes under docker desktop. I managed to load a schema and table definition to ingest from Kafka. It loaded the first 100,000 records and stopped. The UI does not appear to have any way to display errors, logs, etc. I have looked at all the pod logs (controller, broker, minion, and 2 servers), but do not see anything obvious. I could use some assistance debugging this. There do not appear to be separate ingest "tasks" that I can identify (like for druid).
    k
    n
    • 3
    • 20
12345...11Latest