Apache Pinot #troubleshooting

Join Slack

Neha Pawar

08/26/2020, 2:23 AM

it works for both

Neha Pawar

08/26/2020, 2:23 AM

have you put epochMinutes in your schema?

Pradeep

08/26/2020, 2:28 AM

yeah it’s part of the schema

Pradeep

08/26/2020, 2:29 AM

let me try with the latest code

Pradeep

08/26/2020, 2:34 AM

Actually with the latest code it worked, I was using slightly older version from master

👍 1

Mustafa

08/26/2020, 7:57 AM

When I use Hour() function like below.

Copy code

SELECT "timestamp",variant_id,sum(amount) FROM Sales WHERE operator_id = 1 AND campaign_id = 1 GROUP BY Hour("timestamp"), variant_id

I get the following error. It says 'timestamp' should appear in GROUP BY clause. But it already does.

Copy code

ProcessingException(errorCode:150, message:PQLParsingError:
org.apache.pinot.sql.parsers.SqlCompilationException: 'timestamp' should appear in GROUP BY clause.
	at org.apache.pinot.sql.parsers.CalciteSqlParser.validateGroupByClause(CalciteSqlParser.java:177)
	at org.apache.pinot.sql.parsers.CalciteSqlParser.validate(CalciteSqlParser.java:114)
	at org.apache.pinot.sql.parsers.CalciteSqlParser.queryRewrite(CalciteSqlParser.java:364)
	at org.apache.pinot.sql.parsers.CalciteSqlParser.compileCalciteSqlToPinotQuery(CalciteSqlParser.java:338)
	at org.apache.pinot.sql.parsers.CalciteSqlParser.compileToPinotQuery(CalciteSqlParser.java:104)
	at org.apache.pinot.sql.parsers.CalciteSqlCompiler.compileToBrokerRequest(CalciteSqlCompiler.java:33)
	at org.apache.pinot.controller.api.resources.PinotQueryResource.getQueryResponse(PinotQueryResource.java:158)
	at org.apache.pinot.controller.api.resources.PinotQueryResource.handlePostSql(PinotQueryResource.java:131)
	at sun.reflect.GeneratedMethodAccessor32.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:124)
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:167))

Kishore G

08/26/2020, 8:19 AM

It’s not valid sql. You are selecting a column but grouping by a function on that column

Mustafa

08/26/2020, 8:27 AM

When I bring in the sum aggregation, I get the error above.

Elon

08/26/2020, 4:38 PM

Hi, great talk yesterday! Have a question about the new date time field spec. I created a simple test table with the schema below but it does not seem to be bucketing into seconds (i.e. granularity). I inserted a time value every 500 ms to verify and did not see that the

ts

column is bucketed - is there another name for the "bucketed" column? Here is the schema:

Copy code

{
  "schemaName": "myTable2",
  "dimensionFieldSpecs": [
    {
      "name": "col1",
      "dataType": "LONG",
      "defaultNullValue": 0
    }
  ],
  "metricFieldSpecs": [
    {
      "name": "m1",
      "dataType": "LONG"
    }
  ],
  "dateTimeFieldSpecs": [
    {
      "name": "ts",
      "dataType": "LONG",
      "format": "1:MILLISECONDS:EPOCH",
      "granularity": "1:SECONDS"
    }
  ]
}

And here is the data:

Copy code

"resultTable": {
        "dataSchema": {
            "columnDataTypes": ["LONG", "LONG", "LONG"],
            "columnNames": ["col1", "m1", "ts"]
        },
        "rows": [
            [0, 0, 1598431406131],
            [1, 1000, 1598431406634],
            [2, 2000, 1598431407134],
            [3, 3000, 1598431407634],
            [4, 4000, 1598431408134],
            [5, 5000, 1598431408634],
            [6, 6000, 1598431409134],
            [7, 7000, 1598431409634],
            [8, 8000, 1598431410134],
            [9, 9000, 1598431410634]
        ]
    },

Kishore G

08/26/2020, 4:41 PM

you need to use transformFunction

Elon

08/26/2020, 4:46 PM

Thanks! I saw examples in the unit tests, would I be using one of the functions with the

bucketed

suffix or just the regular

toEpochSeconds

transform?

Kishore G

08/26/2020, 4:50 PM

@Neha Pawar ^^

Neha Pawar

08/26/2020, 5:09 PM

if you want to round to the nearest seconds, but still have it in millis, then use

"tsSeconds" : "round(ts, 1000)"

in your transform function

Neha Pawar

08/26/2020, 5:09 PM

if you want to convert it to secondsSinceEpoch, then you can use toEpochSeconds

Neha Pawar

08/26/2020, 5:10 PM

and set that in your table config like this: https://docs.pinot.apache.org/developers/advanced/ingestion-level-transformations#column-transformation

Elon

08/26/2020, 5:31 PM

Thanks!

Elon

08/26/2020, 5:33 PM

Did

streamConfigs

change to

ingestionConfig

in 0.5.0?

Elon

08/26/2020, 5:34 PM

Or is it an additional config?

Neha Pawar

08/26/2020, 5:36 PM

it is additional config

Neha Pawar

08/26/2020, 5:36 PM

correct about granularity

Elon

08/26/2020, 5:39 PM

That really helps, thanks:) Also, great talk yesterday!

Neha Pawar

08/26/2020, 5:39 PM

thank you! 🙂

Tim Chan

09/01/2020, 10:32 PM

hey #C011C9JHN7R i have followed the getting started guide for running pinot in kubernetes. i was able to configure a table to ingest records from a kafka cluster.

Tim Chan

09/01/2020, 10:32 PM

i was able to ingest 125000 records

Tim Chan

09/01/2020, 10:33 PM

when i do a count(*) on the table the count never increases

Tim Chan

09/01/2020, 10:33 PM

i see the following in the controller log

Tim Chan

09/01/2020, 10:33 PM

Copy code

2020/09/01 22:22:31.308 WARN [LLCSegmentCompletionHandlers] [grizzly-http-server-1] Segment file: file:/var/pinot/controller/data/motion/motion__3__0__20200901T2214Z already exists. Replacing it with segment: motion__3__0__20200901T2214Z from instance: Server_pinot-server-1.pinot-server-headless.pinot-quickstart.svc.cluster.local_8098
2020/09/01 22:22:31.311 WARN [LLCSegmentCompletionHandlers] [grizzly-http-server-0] Segment file: file:/var/pinot/controller/data/motion/motion__6__0__20200901T2214Z already exists. Replacing it with segment: motion__6__0__20200901T2214Z from instance: Server_pinot-server-1.pinot-server-headless.pinot-quickstart.svc.cluster.local_8098
2020/09/01 22:22:31.320 ERROR [SegmentCompletionFSM_motion__3__0__20200901T2214Z] [grizzly-http-server-1] Caught exception while committing segment metadata for segment: motion__3__0__20200901T2214Z
java.lang.NullPointerException: null
        at org.apache.pinot.controller.helix.core.realtime.PinotLLCRealtimeSegmentManager.updateCommittingSegmentZKMetadata(PinotLLCRealtimeSegmentManager.java:507) ~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-86a01ff6da71e433a29f26db5c3a586fa57e7ea7]
        at org.apache.pinot.controller.helix.core.realtime.PinotLLCRealtimeSegmentManager.commitSegmentMetadataInternal(PinotLLCRealtimeSegmentManager.java:446) ~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-86a01ff6da71e433a29f26db5c3a586fa57e7ea7]
        at org.apache.pinot.controller.helix.core.realtime.PinotLLCRealtimeSegmentManager.commitSegmentMetadata(PinotLLCRealtimeSegmentManager.java:416) ~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-86a01ff6da71e433a29f26db5c3a586fa57e7ea7]
        at org.apache.pinot.controller.helix.core.realtime.SegmentCompletionManager$SegmentCompletionFSM.commitSegment(SegmentCompletionManager.java:1091) ~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-86a01ff6da71e433a29f26db5c3a586fa57e7ea7]
        at org.apache.pinot.controller.helix.core.realtime.SegmentCompletionManager$SegmentCompletionFSM.segmentCommitEnd(SegmentCompletionManager.java:656) ~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-86a01ff6da71e433a29f26db5c3a586fa57e7ea7]
        at org.apache.pinot.controller.helix.core.realtime.SegmentCompletionManager.segmentCommitEnd(SegmentCompletionManager.java:325) ~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-86a01ff6da71e433a29f26db5c3a586fa57e7ea7]
        at org.apache.pinot.controller.api.resources.LLCSegmentCompletionHandlers.segmentCommit(LLCSegmentCompletionHandlers.java:330) ~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-86a01ff6da71e433a29f26db5c3a586fa57e7ea7]
        at sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source) ~[?:?]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_265]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_265]
        at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52) ~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-86a01ff6da71e433a29f26db5c3a586fa57e7ea7]
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:124) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-86a01ff6da71e433a29f26db5c3a586fa57e7ea7]
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:167) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-86a01ff6da71e433a29f26db5c3a586fa57e7ea7]
        at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:219) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-86a01ff6da71e433a29f26db5c3a586fa57e7ea7]
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:79) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-86a01ff6da71e433a29f26db5c3a586fa57e7ea7]
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:469) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-86a01ff6da71e433a29f26db5c3a586fa57e7ea7]
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:391) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-86a01ff6da71e433a29f26db5c3a586fa57e7ea7]
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:80) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-86a01ff6da71e433a29f26db5c3a586fa57e7ea7]
        at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:253) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-86a01ff6da71e433a29f26db5c3a586fa57e7ea7]
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-86a01ff6da71e433a29f26db5c3a586fa57e7ea7]
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-86a01ff6da71e433a29f26db5c3a586fa57e7ea7]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:292) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-86a01ff6da71e433a29f26db5c3a586fa57e7ea7]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:274) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-86a01ff6da71e433a29f26db5c3a586fa57e7ea7]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:244) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-86a01ff6da71e433a29f26db5c3a586fa57e7ea7]
        at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-86a01ff6da71e433a29f26db5c3a586fa57e7ea7]
        at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:232) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-86a01ff6da71e433a29f26db5c3a586fa57e7ea7]
        at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:679) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-86a01ff6da71e433a29f26db5c3a586fa57e7ea7]
        at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:353) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-86a01ff6da71e433a29f26db5c3a586fa57e7ea7]
        at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:200) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0^Cfa57e7ea7]
        at org.apache.pinot.controller.helix.core.realtime.PinotLLCRealtimeSegmentManager.commitSegmentMetadataInternal(PinotLLCRealtimeSegmentManager.java:446) ~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-86a01ff6da71e433a29f26db5c3a586fa57e7ea7]
        at org.apache.pinot.controller.helix.core.realtime.PinotLLCRealtimeSegmentManager.commitSegmentMetadata(PinotLLCRealtimeSegmentManager.java:416) ~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-86a01ff6da71e433a29f26db5c3a586fa57e7ea7]
        at org.apache.pinot.controller.helix.core.realtime.SegmentCompletionManager$SegmentCompletionFSM.commitSegment(SegmentCompletionManager.java:1091) ~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-86a01ff6da71e433a29f26db5c3a586fa57e7ea7]
        at org.apache.pinot.controller.helix.core.realtime.SegmentCompletionManager$SegmentCompletionFSM.segmentCommitEnd(SegmentCompletionManager.java:656) ~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-86a01ff6da71e433a29f26db5c3a586fa57e7ea7]

Tim Chan

09/01/2020, 10:34 PM

why does it say segment file already exists?

Tim Chan

09/01/2020, 10:35 PM

image.png

Tim Chan

09/01/2020, 10:35 PM

the cluster manager ui says status good or consuming