https://pinot.apache.org/ logo
#general
Title
# general
p

Pedro Silva

05/03/2021, 1:35 PM
Hello, Pinot docs related to deep-storage in K8s seem to be broken: https://docs.pinot.apache.org/operators/tutorials/deployment-pinot-on-kubernetes#deep-storage, can anyone point to the right resource?
p

Pedro Silva

05/03/2021, 3:38 PM
The last link is for file import, is that relevant?
I've configured the controller & server to connect to ADFS. I get the following exception:
Copy code
2021/05/03 16:20:48.309 ERROR [StartServiceManagerCommand] [Start a Pinot [SERVER]] Failed to start a Pinot [SERVER] at 1.844 since launch
java.lang.RuntimeException: com.azure.storage.file.datalake.models.DataLakeStorageException: Status code 400, "<?xml version="1.0" encoding="utf-8"?>
<Error><Code>OutOfRangeInput</Code><Message>One of the request inputs is out of range.
RequestId:45faebbd-b01e-0064-2438-40f291000000
Time:2021-05-03T16:20:48.1516232Z</Message></Error>"
	at org.apache.pinot.spi.filesystem.PinotFSFactory.register(PinotFSFactory.java:58) ~[pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
	at org.apache.pinot.spi.filesystem.PinotFSFactory.init(PinotFSFactory.java:74) ~[pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
	at org.apache.pinot.server.starter.helix.SegmentFetcherAndLoader.<init>(SegmentFetcherAndLoader.java:71) ~[pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
	at org.apache.pinot.server.starter.helix.HelixServerStarter.start(HelixServerStarter.java:324) ~[pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
	at org.apache.pinot.tools.service.PinotServiceManager.startServer(PinotServiceManager.java:150) ~[pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
	at org.apache.pinot.tools.service.PinotServiceManager.startRole(PinotServiceManager.java:95) ~[pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
	at org.apache.pinot.tools.admin.command.StartServiceManagerCommand$1.lambda$run$0(StartServiceManagerCommand.java:260) ~[pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
	at org.apache.pinot.tools.admin.command.StartServiceManagerCommand.startPinotService(StartServiceManagerCommand.java:286) [pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
	at org.apache.pinot.tools.admin.command.StartServiceManagerCommand.access$000(StartServiceManagerCommand.java:57) [pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
	at org.apache.pinot.tools.admin.command.StartServiceManagerCommand$1.run(StartServiceManagerCommand.java:260) [pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
Caused by: com.azure.storage.file.datalake.models.DataLakeStorageException: Status code 400, "<?xml version="1.0" encoding="utf-8"?>
Does this ring any bells?
m

Mayank

05/03/2021, 7:47 PM
@User ^^
r

Ravi Teja Kanumula

05/03/2021, 9:29 PM
I haven’t seen this error before.. But will spend some time on it..
m

Mayank

05/03/2021, 9:30 PM
Also, @User are you using ADLS gen1 or gen2?
p

Pedro Silva

05/04/2021, 6:41 AM
Gen2
r

Ravi Teja Kanumula

05/09/2021, 1:20 AM
Do you still have issues with this ? Can you supply the configuration that you are using.. (without the secrets ofcourse)
also the version you are using..
p

Pedro Silva

05/09/2021, 5:26 PM
Hello Ravi, we have solved the issue through trial and error. The issue was finding: • What format an ADL connection string took. For future reference it's:
adl://<container-name>@<azure-storage-account>/<path within the container wherein to store information>
• `pinot.controller.storage.factory.adl.fileSystemName`property had to be defined, this seems specific to ADLS.
m

Mayank

05/09/2021, 5:52 PM
@User @User Would really appreciate if this can be documented here: https://docs.pinot.apache.org/basics/data-import/pinot-file-system/import-from-adls-azure