Channels
#troubleshooting
Title
e
Elon
08/21/2020, 7:34 PMWe can upload a flamegraph. @Onha Choe is the brave person that is working with the pinot logging controller. Anyone familiar with this that we can work with?
Here is the flamegraph
message has been deleted
@Onha Choe has some jstacks and other context as well.
message has been deleted
o
Onha Choe
08/21/2020, 7:36 PM^ Seems like controller is responsible for grabbing segments and compress/upload to object store(GCS) via pinotFS. Not sure if it is necessary for this to be going through the controller, or can this be uploaded directly from servers
e
Elon
08/21/2020, 10:22 PMLooks like there is a call to do this on the server
Also, it seems like LLCSegmentCompletionHandler.segmentCommit() should respond asynchronously so that the threads don't wait on io
@Onha Choe - want to try this?
Copy code
listener.getTransport().setIOStrategy(WorkerThreadIOStrategy.getInstance());n
Neha Pawar
08/23/2020, 12:51 AM
@Ting Chen ^^
t
Ting Chen
08/25/2020, 6:36 PMIn the deep store by pass (see the cwiki for design), servers can directly upload segments to deepstore.
o
Onha Choe
08/28/2020, 12:54 AM@Ting Chen I’ve looked at the cwiki, though non of the PR were POCs and never got merged? What is the status on it?
t
Ting Chen
08/28/2020, 1:01 AMhttps://github.com/apache/incubator-pinot/pulls?q=is%3Apr+author%3Achenboat+is%3Aclosed those PR marked as deepstore by-pass are releated.
👍 1
There is a last PR for integration test and bug fix waiting for approval now : https://github.com/apache/incubator-pinot/pull/5857
n
Neha Pawar
08/30/2020, 8:02 PM
FYI @Kishore G
@Ting Chen is this part already implemented:
In the deep store by pass (see the cwiki for design), servers can directly upload segments to deepstore.whats the configuration to achieve this (servers uploading to deep store) ?
t
Ting Chen
08/31/2020, 5:21 PM#5314 implements server direct upload to the deep store.
there are two new config to enable that:
1. pinot.server.instance.segment.store.uri for deepstore uri
2. a table config segmentsValidationAndRetentionConfig.setPeerSegmentDownloadScheme()
I will add them to the doc.