Hi Folks, is there a direct way( or a query) to check if a subset of values exist in a pinot multi value column? eg. select * from a where test_tags in (‘a’, ‘b’) I want both (‘a’,‘b’) to exist in the column. test_tags can contain more values as well but, (‘a’, ‘b’) should be a subset.

Hey, I am trying to ingest some data from s3. After running the ingestion job, the status of all the segments show

GOOD

, but all the columns have

NULL

values and the time column has

1662356536

(looks like time of ingestion?). Adding more details in the thread.

Hi, is there something similar to element_at (https://prestodb.io/docs/current/functions/array.html) operation in Pinot? I want to find the last element in a multi-value column field.

Hey team, quick question. Could I in theory create a table with id_sets already generated, and then look it up that way? Instead of generating it on the fly?

I am writing the avro->pinot schema converter. But I am not sure how to convert an avro complex map type like into pinot's:

{
  "name": "optAdditional",
  "type": {
    "type": "map",
    "values": "string"
  },
  "default": {},
  "doc": "collection of miscellaneous logged strings"
}

In

transcript-table-offline.json

, what is

MMAP

load mode vs.

HEAP

load mode?

{
  "tableName": "transcript",
  "segmentsConfig": {
    "timeColumnName": "timestampInEpoch",
    "timeType": "MILLISECONDS",
    "replication": "1",
    "schemaName": "transcript"
  },
  "tableIndexConfig": {
    "invertedIndexColumns": [],
    "loadMode": "MMAP"
  },
  "tenants": {
    "broker": "DefaultTenant",
    "server": "DefaultTenant"
  },
  "tableType": "OFFLINE",
  "metadata": {}
}

I have a

long

column called

dirTs

for a record's timestamp. How can I convert this

long

data type into

timestamp

data type? Which transformation function can I use?

"ingestionConfig":{
      "transformConfigs": [
        {
          "columnName": "dirTs",
          "transformFunction": "toTimestamp(\"dirTs\")"
        }
      ]
  },

Is there a function called

toTimestamp

?

Got two segments created for a new table

director

. But they are not

ONLINE

. How can I make them online?

Trouble Running Server in IDE. Anyone have any input on the following? Error is

Failed to initialize PinotMetricsFactory. Please check if any pinot-metrics related jar is actually added to the classpath

Following this example: https://docs.pinot.apache.org/basics/getting-started/running-pinot-locally Successfully built the project using JDK11 Started all components including the server through admin scripts everything works as expected. Then started the following via the admin scripts Zookeeper Controller Broker Attempting to start the server component in IntelliJ using a run configuration as shown here: https://docs.pinot.apache.org/basics/getting-started/running-pinot-locally#start-pinot-component-in-debug-mode-with-intellij Details on IntelliJ Build version: IntelliJ IDEA 2021.1.1 Build #IU-211.7142.45 April 30, 2021 Java version: 11.0.10+9-b1341.41x86_64 metrics-core-2.2.0.jar can be found under External Libraries The issue is seemingly when we attempt to scan for annotated classes. I have evaluated that code fragment from PinotReflectionUtils. It does not return any found classes. see screen capture below

new Reflections(new ConfigurationBuilder().setUrls(ClasspathHelper.forPackage(packageName))
            .filterInputsBy(new FilterBuilder.Include(regexPattern)))

image.png

Now, I see the status of table segments are

ONLINE

, But when I go to Query Console and click on the table

director

, I get HTTP 500 error Below is from controller's log

2022/11/29 03:30:45.857 ERROR [PinotQueryResource] [grizzly-http-server-57] Caught exception in sendQueryRaw
java.io.IOException: Failed : HTTP error code : 500
        at org.apache.pinot.controller.api.resources.PinotQueryResource.sendPostRaw(PinotQueryResource.java:306) ~[pinot-all-0.11.0.0-jar-with-dependencies.jar:0.11.0.0-cc1ceb04dc39a1d76eee11b480e34370c6344424]
        at org.apache.pinot.controller.api.resources.PinotQueryResource.sendRequestRaw(PinotQueryResource.java:344) ~[pinot-all-0.11.0.0-jar-with-dependencies.jar:0.11.0.0-cc1ceb04dc39a1d76eee11b480e34370c6344424]
        at org.apache.pinot.controller.api.resources.PinotQueryResource.sendRequestToBroker(PinotQueryResource.java:243) ~[pinot-all-0.11.0.0-jar-with-dependencies.jar:0.11.0.0-cc1ceb04dc39a1d76eee11b480e34370c6344424]
        at org.apache.pinot.controller.api.resources.PinotQueryResource.getQueryResponse(PinotQueryResource.java:214) ~[pinot-all-0.11.0.0-jar-with-dependencies.jar:0.11.0.0-cc1ceb04dc39a1d76eee11b480e34370c6344424]
        at org.apache.pinot.controller.api.resources.PinotQueryResource.executeSqlQuery(PinotQueryResource.java:139) ~[pinot-all-0.11.0.0-jar-with-dependencies.jar:0.11.0.0-cc1ceb04dc39a1d76eee11b480e34370c6344424]
        at org.apache.pinot.controller.api.resources.PinotQueryResource.handlePostSql(PinotQueryResource.java:103) ~[pinot-all-0.11.0.0-jar-with-dependencies.jar:0.11.0.0-cc1ceb04dc39a1d76eee11b480e34370c6344424]
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
        at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
        at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
        at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52) ~[pinot-all-0.11.0.0-jar-with-dependencies.jar:0.11.0.0-cc1ceb04dc39a1d76eee11b480e34370c6344424]
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:124) ~[pinot-all-0.11.0.0-jar-with-dependencies.jar:0.11.0.0-cc1ceb04dc39a1d76eee11b480e34370c6344424]
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:167) ~[pinot-all-0.11.0.0-jar-with-dependencies.jar:0.11.0.0-cc1ceb04dc39a1d76eee11b480e34370c6344424]
        at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:219) ~[pinot-all-0.11.0.0-jar-with-dependencies.jar:0.11.0.0-cc1ceb04dc39a1d76eee11b480e34370c6344424]
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:79) ~[pinot-all-0.11.0.0-jar-with-dependencies.jar:0.11.0.0-cc1ceb04dc39a1d76eee11b480e34370c6344424]

what is the difference between minion and minion stateless? is the stateless one preferred?

[SOLVED]Is it possible to have same schema and table name for offline and realtime tables? I have an OFFLINE table in Pinot and Im trying to have the same schema and table name for realtime table but when I run addTable command I am getting

{
  "code": 400,
  "error": "TableConfigs: linkpage already exists. Use PUT to update existing config"
}

Is there some thing I am missing out. I have used

"tableType": "REALTIME"

in table config.

I have been pushing data to kinesis and the push is successful, and pinot was showing the data as expected but suddenly it seems to have stopped ingesting the data. I have set the flush threshold time to 1 hr and I cannot see the data written to deep store either ( pinot has been consuming for more than an hour). Any idea how i can debug the issue?

Is it possible to have two different tables subscribing to the same broker / topic at the same time? I am getting

2022/11/29 15:07:01.989 INFO [AddTableCommand] [main] {"code":500,"error":"<http://org.apache.pinot.shaded.org|org.apache.pinot.shaded.org>.apache.kafka.common.KafkaException: Failed to construct kafka consumer"}

when setting up the tables, but not finding any error logs in the pinot server or controller

Hi Team Using pinot helm chart to run pinot cluster and trying to load data more than 100Million records and getting No space left on device and tried to increase size of controller pv to 6gb and still getting this error . ERROR: WARN [SegmentPushUtils] [Executor task launch worker for task 5] Caught temporary exception while pushing table: rcem_map_dly segment: rcem_map_dly_OFFLINE_1 to http://10.20.20.205:9011, will retry org.apache.pinot.common.exception.HttpErrorStatusException: Got error status code: 500 (Internal Server Error) with reason: "Exception while uploading segment: No space left on device" while sending request: http://10.20.20.205:9011/v2/segments?tableName=rcem_map_dly to controller: pinot-controller-0.pinot-controller-headless.pinot.svc.cluster.local, version: Unknown

Hi Pinot Team! I am using the newly released feature

GAPFILL

and it works perfectly in most cases. However, I would like to aggregate the result of a

GAPFILL

by a column that isn't the time column. When I try this, I get the following error message :

[...]
java.io.IOException: Failed : HTTP error code : 500. Root Cause: No Group By timebucket.
[...]

Is there a workaround? Why is it impossible to group by columns that aren't the time bucket?

Hi strange case on backfilling data. I’ve added a derived column on my schema and then on the table. Then force commit my table and reload segments to get my backfilled but I keep getting empty value instead of filled vals. Any ideas ?

We've seen some situations where force-commit can delay completing; reads don't provide data immediately. How long are you waiting?

Seeing a lot of

2022/11/27 23:57:09.794 ERROR [JobDispatcher] [HelixController-pipeline-task-PinotCluster-(9d89f973_TASK)] No available instance found for job!

in my logs (0.10.0 version). Could this be related to the

RealtimeToOfflineSegmentsTask

?

Hi everyone, having trouble running locally. Using JDK 11.0.17 Pinot 0.11.0 Tried following the instructions on the website for running locally, but when attempting to run quick start the shell just briefly pops up, not even leaving me a chance to read the error message. Desperate for a solution

Hi Team ,Trying to connect pinot through JDBC-CLIENT facing java.sql.SQLException: Failed to connect to url : jdbc:pinot://host:port. using pinot 0.11.0 version. pinot is running and tried to do curl ,it is working can any one help to fix this?

Hi , I could not find info about the defaultNullValue for complex data type in pinot ? Does pinot support defaultNullValue for complex types ? If so what would be the default value as in we have for other data types ?

Hi all, I’m trying a very basic SQL query - but I am getting a weird error that states the error code is

450

, does anyone have any idea how I can get more context?

ProcessingException(errorCode:450, message:InternalError:
org.apache.pinot.sql.parsers.SqlCompilationException: Caught exception while parsing query: select * from Pageviews  ORDER BY time DESC
	at org.apache.pinot.sql.parsers.CalciteSqlParser.compileToSqlNodeAndOptions(CalciteSqlParser.java:136)
	at org.apache.pinot.controller.api.resources.PinotQueryResource.executeSqlQuery(PinotQueryResource.java:135)
	at org.apache.pinot.controller.api.resources.PinotQueryResource.handlePostSql(PinotQueryResource.java:103)
	at jdk.internal.reflect.GeneratedMethodAccessor263.invoke(Unknown Source)
...

Hi team, does Pinot support group by top n query? Example query, query a,b,c from table group by f1,f2,f3. How can I get max n rows for each combination of f1 and f2? (edited)

We received the following error from one of our realtime servers today (see in thread). Any thoughts?

For batch ingestion, Pinot will create one segment from each input file. And there is a minion task that can reduce the number of segments by merging. I wonder if there is any reason or problem (other than

not implemented yet

) that Pinot cannot create one segment from multiple input files (says, limited by the total size of the files)?

We've been receiving "segments unavailable" messages in a realtime table recently, details in thread, any idea how they could be happening?

Hello, does anyone have experience with monitoring Pinot on Grafana with Prometheus? I am using

prometheus-community/kube-prometheus-stack

but the CPU usage panels is showing no data since I am metric container_cpu_`user`_seconds_total , but I have container_cpu_`usage`_seconds_total Is there a way to enable the missing user metric?