Hello, I am facing difficulties in flattening and transforming json records from kafka. This is the structure of the json record

{
  "event_name": "abcd",
  "event_type": "",
  "version": "v1",
  "write_key": "",
  "properties": {
    "status_code": "some_code",
    "status": "some_status",
    "mode": "some_mode"
  },
  "event_timestamp": 1616157914,
  "mode": "live"
}

And my schema looks like this

"mode": "string",
"request_failure": "INT"

Hi team, how do I quote a reserved word like “timestamp” when querying pinot in Presto? I tried using double quotes but it’s not working, any workarounds ?

We scaled up our brokers and saw that for about 15 minutes some brokers were returning "RESOURCE_MISSING_ERROR" - when I looked in the zk browser for INSTANCES/broker-<x>/CURRENTSTATES it appears like the END_TIME values spanned that 15 minute interval. To avoid this should users get their client by contacting zookeeper, i.e. brokers for table?

Hi. Guys I have one realtime with 3 replicate segmengs and 6 server nodes, When I kill one server node the status will like below, What I need to do to reblance those segments to 3 replicasPerPartition, Assuming that server node broken forever

Hello, what happens if one of the pinot server disk with 25% of the segments fails and I restore the data from 1 day old backup that holds 20% of segments? Will the segments from the backup disk is honored or everything has to begin from scratch?

Hello, We have setup a monitoring dashboard on grafana for pinot. We are currently using the image with

latest

tag and we observe that few metrics related to query response time have been coming as

0

. We didn’t change any config on pinot or the grafana query, so wanted to know if there was any breaking change merged recently? Here is a screen shot of the dashboard.

Hi All, I got an error Caused by: java.net.BindException: Address already in use at sun.nio.ch.Net.bind0(Native Method) at sun.nio.ch.Net.bind(Net.java:433) at sun.nio.ch.Net.bind(Net.java:425) at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:220) at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:85) at sun.net.httpserver.ServerImpl.bind(ServerImpl.java:133) at sun.net.httpserver.HttpServerImpl.bind(HttpServerImpl.java:54) at io.prometheus.jmx.shaded.io.prometheus.client.exporter.HTTPServer.<init>(HTTPServer.java:145) at io.prometheus.jmx.shaded.io.prometheus.jmx.JavaAgent.premain(JavaAgent.java:31) ... 6 more FATAL ERROR in native method: processing of -javaagent failed When I deploying controller, broker, server with -javaagent:jmx_prometheus_javaagent-0.12.0.jar=9066 in one vm server, Do you know how to send metrics to prometheus:9066 in one node

When ingesting json data, do the json objects have to be flat? I can't find docs about parsing json object trees, but I see the github event example is a tree

The server hosting our Pinot got rebooted and when we restarted the components, we have no segments, no tables and no schemas. I see in the server logs that it's looking in the correct directories for instance data and instance segment, and I've confirmed there's data in those directories. Pinot server isn't loading anything though?

Hi. I'm new to Apache pinot. I have a use case where I need to ingest data through kafka (externally with SSL certs configured) to pinot which is installed in kubernetes environment. How can i connect exernal kafka with SSL certs configured in pinot on kubernetes environment? It would be great if you forward me to the respective documentation.

Hi All When I am writing big with very high speed 190k/s into pinot I got the following error, where can I check the reason

Hello, I have some issues on a realtime table, my segments are moved to OFFLINE because of a

java.lang.NullPointerException

:

Mar 25 09:50:52 pinot-hosts-3 bash[11610]: 2021/03/25 09:50:52.405 ERROR [LLRealtimeSegmentDataManager_datasource_605b02ec0eb00003003bfc41__0__18__20210325T0855Z] [datasource_605b02ec0eb00003003bfc41__0__18__20210325T0855Z] Exception while in work
Mar 25 09:50:52 pinot-hosts-3 bash[11610]: java.lang.NullPointerException: null
Mar 25 09:50:52 pinot-hosts-3 bash[11610]:         at org.apache.pinot.core.data.manager.realtime.SegmentBuildTimeLeaseExtender.addSegment(SegmentBuildTimeLeaseExtender.java:100) ~[pinot-all-0.7.0-jar-with-dependencies.jar:0.7.0-695ca390a8080409b3b76f250f2315b81b86b362]
Mar 25 09:50:52 pinot-hosts-3 bash[11610]:         at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.buildSegmentForCommit(LLRealtimeSegmentDataManager.java:698) ~[pinot-all-0.7.0-jar-with-dependencies.jar:0.7.0-695ca390a8080409b3b76f250f2315b81b86b362]
Mar 25 09:50:52 pinot-hosts-3 bash[11610]:         at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager$PartitionConsumer.run(LLRealtimeSegmentDataManager.java:610) [pinot-all-0.7.0-jar-with-dependencies.jar:0.7.0-695ca390a8080409b3b76f250f2315b81b86b362]
Mar 25 09:50:52 pinot-hosts-3 bash[11610]:         at java.lang.Thread.run(Thread.java:748) [?:1.8.0_275]

Do you have any idea on why this is happening? Thank you

Hello, we are on 0.6 and we are facing one issue with realtime tables consumption from Kafka. Consumption from kafka stops all of a sudden. There are no errors/exceptions in logs. And from thread dumps, we don’t see the consumption thread. From this I assume we hit some uncaught exception in consumption thread. We are using Kafka low level consumer. At pinot bootstrap, should we register a default uncaught exception handler which just logs them with full stack trace. This helps in debugging several unhandled corner cases.

Hi, We have a Pinot cluster with around 100 realtime tables. Around 28 tables went into bad state. We have 2 sets of servers (3 each) with different tags (e.g., realtime and offline). Our tables are configured (using the tagOverrideConfig) that once a consuming segment is completed, it is moved immediately to servers with offline tag. On the tables that went to bad state, we noticed that from the UI, it's still showing that the segment is still assigned to the "realtime" server. We do see the segment get completed and get uploaded to the deepstore. Also, we noticed that in zookeeper in pinot -> instances -> server -> messages there are lots of messages. Does that mean that messages are not getting consumed by the server? I assume this is how controller/server communicate thru helix (?).

I use PERCENTILEMV operations to select data , but no response, anyone one know why?

Hi team. I'm facing this issue while connecting external kafka with pinot alone with SSL keys. anyone help me out on this?

What's the data type for

webConferenceId

?

But I can’t use this “select * from realtime_sjc_wmequality_report where webexConferenceId=189852985506937900 limit 1000” to get it

The following query compiles with the calcite compiler but the broker request still uses the pql2 compiler which fails:

select foo from table
where
and case bar when 1,2,3 then 4 when 4,5,6 then 5 when 7 then 6 when 8 then 7 else 17 end = 6
limit 10

Hi all, while reloading status for a table, I get the following error message. How do we resolve this?

I am trying JSON Indexing: I am able to see the data in the Query Console but. And I am fallowing "githubEvents" example in the Pinot Quick Starts applications provided with "incubator-pinot". I run query form query console:

select exp from my_table WHERE JSON_MATCH(colunm1, 'att1=''Lab''')

But I am getting error

[
  {
    "errorCode": 200,
    "message": "QueryExecutionError:\njava.lang.IllegalStateException: Cannot apply JSON_MATCH on column: column1 without json index\n\tat shaded.com.google.common.base.Preconditions.checkState(Preconditions.java:518)\n\tat org.apache.pinot.core.plan.FilterPlanNode.constructPhysicalOperator(FilterPlanNode.java:206)\n\tat org.apache.pinot.core.plan.FilterPlanNode.run(FilterPlanNode.java:80)\n\tat org.apache.pinot.core.plan.DocIdSetPlanNode.run(DocIdSetPlanNode.java:41)\n\tat org.apache.pinot.core.plan.ProjectionPlanNode.run(ProjectionPlanNode.java:52)\n\tat org.apache.pinot.core.plan.TransformPlanNode.run(TransformPlanNode.java:52)\n\tat org.apache.pinot.core.plan.SelectionPlanNode.run(SelectionPlanNode.java:83)\n\tat org.apache.pinot.core.plan.CombinePlanNode.run(CombinePlanNode.java:100)\n\tat org.apache.pinot.core.plan.InstanceResponsePlanNode.run(InstanceResponsePlanNode.java:33)\n\tat org.apache.pinot.core.plan.GlobalPlanImplV0.execute(GlobalPlanImplV0.java:45)\n\tat org.apache.pinot.core.query.executor.ServerQueryExecutorV1Impl.processQuery(ServerQueryExecutorV1Impl.java:294)\n\tat org.apache.pinot.core.query.executor.ServerQueryExecutorV1Impl.processQuery(ServerQueryExecutorV1Impl.java:215)\n\tat org.apache.pinot.core.query.executor.QueryExecutor.processQuery(QueryExecutor.java:60)\n\tat org.apache.pinot.core.query.scheduler.QueryScheduler.processQueryAndSerialize(QueryScheduler.java:157)"
  }
]

My Table config appears like this:

.
.
 "tableIndexConfig": {
    "loadMode": "MMAP",
    
 },
    "jsonIndexColumns": [
      "colunm1",
      "column2",
      "column3"
    ],
 
  "metadata": {
    "customConfigs": {}
  }
}

I fallowed quick start project, there, it declared like below way, and I tried that to but this time not even see data in my table in query console,

.
.
 "tableIndexConfig": {
    "loadMode": "MMAP",
    "jsonIndexColumns": [
      "colunm1",
      "column2",
      "column3"
    ]
 },
  "metadata": {
    "customConfigs": {}
  }
}

Schema:

.
.
.
"dimensionFieldSpecs": [
  {
    "name": "colunm1",
    "dataType": "STRING",
    "maxLength": 2147483647
  },
  {
    "name": "colunm2",
    "dataType": "STRING",
    "maxLength": 2147483647
  },

  {
    "name": "colunm3",
    "dataType": "STRING",
    "maxLength": 2147483647
  },
.
.
.

Need your help please 😇😇

you have typo in the field name

From my perspective it has to have something to do with the Schema Registry client because the kafka consumer is able to actually retrieve data from kafka, the step failing is the validation of the avro schema against the schema registry, but I am very sure it is picking the right certs for that schema registry

Hello 👋 What could be the reasons for an external view to be missing ? (Specifically for a REALTIME table) Thanks

Hi, we added a column to a schema via PUT the schema api (it's in the kafka topic) but still do not see data, i.e. the select list does not contain that column. Do we need to restart the servers or do some other api call?

Hello 👋 Can you confirm Pinot Controller uses

/opt/pinot/conf/pinot-controller-log4j2.xml

for logging configuration ? (without override from JAVA_OPTS) Thanks !

Hi All, In documentation, it saying Pinot can not support joins in queries. is there any alternative to achieve that. I have to implement derived stats. which will query on multiple pinot tables(schema) and get the data.

Has anyone been successful with using Superset and Pinot, when following the Superset manual installation steps at https://superset.apache.org/docs/installation/installing-superset-from-scratch? I finally got Superset running after some manual fixup, but when trying to connect Pinot as a database I get

Can't load plugin: sqlalchemy.dialects:http

, which makes me think my install is borked. And yes, I know Docker is the preferred solution here 🙂

Hi Community, I am fairly new to pinot. Does the star tree index help when we have aggregate queries with time range (and exclusion) filters ?

Hello , What will happen to the in realtime in-memory consuming segment if a server is restarted? Do Pino start recreating the segment again after restart or I lose the data?