Hello,
Pinot docs related to deep-storage in K8s ...
# generalp
Pedro Silva
05/03/2021, 1:35 PMHello,
Pinot docs related to deep-storage in K8s seem to be broken: https://docs.pinot.apache.org/operators/tutorials/deployment-pinot-on-kubernetes#deep-storage, can anyone point to the right resource?
m
Mayank
05/03/2021, 3:04 PM
Mayank
05/03/2021, 3:06 PM
p
Pedro Silva
05/03/2021, 3:38 PMThe last link is for file import, is that relevant?
Pedro Silva
05/03/2021, 4:25 PMI've configured the controller & server to connect to ADFS. I get the following exception:
Copy code
2021/05/03 16:20:48.309 ERROR [StartServiceManagerCommand] [Start a Pinot [SERVER]] Failed to start a Pinot [SERVER] at 1.844 since launch
java.lang.RuntimeException: com.azure.storage.file.datalake.models.DataLakeStorageException: Status code 400, "<?xml version="1.0" encoding="utf-8"?>
<Error><Code>OutOfRangeInput</Code><Message>One of the request inputs is out of range.
RequestId:45faebbd-b01e-0064-2438-40f291000000
Time:2021-05-03T16:20:48.1516232Z</Message></Error>"
at org.apache.pinot.spi.filesystem.PinotFSFactory.register(PinotFSFactory.java:58) ~[pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
at org.apache.pinot.spi.filesystem.PinotFSFactory.init(PinotFSFactory.java:74) ~[pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
at org.apache.pinot.server.starter.helix.SegmentFetcherAndLoader.<init>(SegmentFetcherAndLoader.java:71) ~[pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
at org.apache.pinot.server.starter.helix.HelixServerStarter.start(HelixServerStarter.java:324) ~[pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
at org.apache.pinot.tools.service.PinotServiceManager.startServer(PinotServiceManager.java:150) ~[pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
at org.apache.pinot.tools.service.PinotServiceManager.startRole(PinotServiceManager.java:95) ~[pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
at org.apache.pinot.tools.admin.command.StartServiceManagerCommand$1.lambda$run$0(StartServiceManagerCommand.java:260) ~[pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
at org.apache.pinot.tools.admin.command.StartServiceManagerCommand.startPinotService(StartServiceManagerCommand.java:286) [pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
at org.apache.pinot.tools.admin.command.StartServiceManagerCommand.access$000(StartServiceManagerCommand.java:57) [pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
at org.apache.pinot.tools.admin.command.StartServiceManagerCommand$1.run(StartServiceManagerCommand.java:260) [pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
Caused by: com.azure.storage.file.datalake.models.DataLakeStorageException: Status code 400, "<?xml version="1.0" encoding="utf-8"?>m
Mayank
05/03/2021, 7:47 PM
@User ^^
r
Ravi Teja Kanumula
05/03/2021, 9:29 PMI haven’t seen this error before.. But will spend some time on it..
m
Mayank
05/03/2021, 9:30 PM
Also, @User are you using ADLS gen1 or gen2?
p
Pedro Silva
05/04/2021, 6:41 AMGen2
r
Ravi Teja Kanumula
05/09/2021, 1:20 AMDo you still have issues with this ? Can you supply the configuration that you are using.. (without the secrets ofcourse)
Ravi Teja Kanumula
05/09/2021, 1:21 AMalso the version you are using..
p
Pedro Silva
05/09/2021, 5:26 PMHello Ravi, we have solved the issue through trial and error.
The issue was finding:
• What format an ADL connection string took. For future reference it's:
adl://<container-name>@<azure-storage-account>/<path within the container wherein to store information>m
Mayank
05/09/2021, 5:52 PM
@User @User Would really appreciate if this can be documented here: https://docs.pinot.apache.org/basics/data-import/pinot-file-system/import-from-adls-azure