Apache Pinot #troubleshooting

Join Slack

Laxman Ch

08/17/2020, 4:12 PM

Here is the error

Mayank

08/17/2020, 4:12 PM

Does it look like valid gcs path?

Laxman Ch

08/17/2020, 4:12 PM

Segment metadata for above segment

Copy code

{
  "segment.realtime.endOffset": "410508",
  "segment.time.unit": "MILLISECONDS",
  "segment.start.time": "1597505986487",
  "segment.flush.threshold.size": "13037",
  "segment.realtime.startOffset": "398812",
  "segment.end.time": "1597527578602",
  "segment.total.docs": "11696",
  "segment.table.name": "rawServiceView_REALTIME",
  "segment.realtime.numReplicas": "1",
  "segment.creation.time": "1597506020322",
  "segment.realtime.download.url": "<http://pinot-controller-1.pinot-controller.traceable.svc.cluster.local:9000/segments/rawServiceView/rawServiceView__3__31__20200815T1540Z>",
  "segment.name": "rawServiceView__3__31__20200815T1540Z",
  "segment.index.version": "v3",
  "custom.map": null,
  "segment.flush.threshold.time": null,
  "segment.type": "REALTIME",
  "segment.crc": "2161937309",
  "segment.realtime.status": "DONE"
}

Mayank

08/17/2020, 4:13 PM

Hmm, it should be gcs path

Laxman Ch

08/17/2020, 4:13 PM

No @Mayank. Its http path.

Laxman Ch

08/17/2020, 4:13 PM

Even current segments that getting prepared its http path only…

Mayank

08/17/2020, 4:14 PM

Then how do you know you are using gcs?

Laxman Ch

08/17/2020, 4:14 PM

We see data in gcs

Laxman Ch

08/17/2020, 4:15 PM

Is this related to this ticket? I see this is still open https://github.com/apache/incubator-pinot/issues/5630

Mayank

08/17/2020, 4:15 PM

I thought I saw a fix from @Kartik Khare for S3 on this?

Kishore G

08/17/2020, 4:22 PM

Ting fixed it

Laxman Ch

08/17/2020, 4:23 PM

But issue is still open. Must be a duplicate. https://github.com/apache/incubator-pinot/issues/5630

Kishore G

08/17/2020, 4:24 PM

i think he fixed it as part of another PR

Kishore G

08/17/2020, 4:24 PM

@Ting Chen ^^

Laxman Ch

08/17/2020, 4:26 PM

Also, what happens to the metadata for the segments when gcs was not enabled? Do we have a tool to fix the existing metadata?

Kishore G

08/17/2020, 4:29 PM

no, but its a good idea to write one, should be simple. Do you want to give it a shot, we can guide you

Laxman Ch

08/17/2020, 4:46 PM

Sure Kishore. I’m already trying it for a segment by manually changing the segment metadata in zookeeper using zk-cli.

Laxman Ch

08/17/2020, 4:47 PM

Meanwhile, Can you please point me to the PR or version where this is fixed?

Laxman Ch

08/17/2020, 4:47 PM

We are currently on 0.3.0 version

Kishore G

08/17/2020, 4:49 PM

the fix should be available in 0.5.0 (going out this week)

Elon

08/17/2020, 7:39 PM

We had ha controllers (3 of them) in pinot 0.3.0 and 0.4.0 and this worked for us.

Laxman Ch

08/17/2020, 9:32 PM

Segment download flow from gcs doesn’t work Here is the ticket: https://github.com/apache/incubator-pinot/issues/5884

Christian Acuna

08/18/2020, 12:05 AM

I’m running into an issue with the Broker in an ERROR state for Offline tables that are backfilled using the

SegmentCreationAndTarPush

Spark Ingestion job. When I query the hybrid table for the min of the table it only returns the min of the realtime tables and not the OFFLINE segments. When I query

localhost:9500/segments/TABLE_NAME

it show both OFFLINE and ONLINE segments. I’m looking for guidance on where to look to start debugging OFFLINE tables in

"ERROR"

an error state from the broker’s perspective. We are using S3FileSystem.

Copy code

"sys_oops_OFFLINE": {
      "CURRENT_STATE": "ERROR",
      "END_TIME": "1597271220022",
      "INFO": "",
      "PREVIOUS_STATE": "OFFLINE",
      "START_TIME": "1597271219800",
      "TRIGGERED_BY": "pinot-controller-0.pinot-controller-headless.pinot.svc.cluster.local_9000"
    },
"sys_reboot_OFFLINE": {
      "CURRENT_STATE": "ERROR",
      "END_TIME": "1597271219935",
      "INFO": "",
      "PREVIOUS_STATE": "OFFLINE",
      "START_TIME": "1597271219798",
      "TRIGGERED_BY": "pinot-controller-0.pinot-controller-headless.pinot.svc.cluster.local_9000"
    }

Kishore G

08/18/2020, 12:06 AM

What’s in ERROR state?

Christian Acuna

08/18/2020, 12:07 AM

image.png

Christian Acuna

08/18/2020, 12:09 AM

When I look at

ERRORS

within the Broker Instance it’s an empty

{}

Kishore G

08/18/2020, 12:10 AM

You should see something in that broker log

Christian Acuna

08/18/2020, 12:26 AM

Copy code

[HelixTaskExecutor-message_handle_thread] Caught exception while processing transition from OFFLINE to │
│ java.lang.IllegalStateException: Failed to find schema for table: DnsForwarderServiceStatus_OFFLINE                                                                                 │
│     at shaded.com.google.common.base.Preconditions.checkState(Preconditions.java:518) ~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-0fc081157c88734af0433c3cd │
│     at org.apache.pinot.broker.routing.timeboundary.TimeBoundaryManager.<init>(TimeBoundaryManager.java:70) ~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-0fc │
│     at org.apache.pinot.broker.routing.RoutingManager.buildRouting(RoutingManager.java:332) ~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-0fc081157c88734af04 │
│     at org.apache.pinot.broker.broker.helix.BrokerResourceOnlineOfflineStateModelFactory$BrokerResourceOnlineOfflineStateModel.onBecomeOnlineFromOffline(BrokerResourceOnlineOfflin │
│     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_262]                                                                                                   │
│     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_262]                                                                                 │
│     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_262]                                                                         │
│     at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_262]                                                                                                              │
│     at org.apache.helix.messaging.handling.HelixStateTransitionHandler.invoke(HelixStateTransitionHandler.java:404) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPS │
│     at org.apache.helix.messaging.handling.HelixStateTransitionHandler.handleMessage(HelixStateTransitionHandler.java:331) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5. │
│     at org.apache.helix.messaging.handling.HelixTask.call(HelixTask.java:97) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-0fc081157c88734af0433c3cd30f310474d │
│     at org.apache.helix.messaging.handling.HelixTask.call(HelixTask.java:49) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-0fc081157c88734af0433c3cd30f310474d │
│     at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_262]                                                                                                       │
│     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_262]                                                                                │
│     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_262]                                                                                │
│     at java.lang.Thread.run(Thread.java:748) [?:1.8.0_262]

Christian Acuna

08/18/2020, 12:26 AM

Still getting the same error after restarting, the schema is the same as the realtime table schema

Kishore G

08/18/2020, 12:26 AM

Do you have a hen for the table?