https://pinot.apache.org/ logo
Join Slack
Powered by
what exactly we need to do to restore controller s...
# troubleshooting
s
what exactly we need to do to restore controller segments to new pinot cluster?
m
do you see that the segments are being saved into HDFS?
s
yes it got saved to hdfs, i checked it
Screenshot from 2021-12-15 17-37-01.png
FYI
m
ok cool, so that bit's working.
the segments wouldn't be downloaded from hdfs --> controller though, rather they get downloaded to the servers
s
basically my req is i have 1 node cluster,(assuming zk is in separate node).. and configured hdfs as deep storage... if that pinot cluster goes down(controller, broker, server), then i need to install new one node cluster with same pinot hdfs controller segments path...
m
oh I see so all on the same node
s
yes
what exactly the procedure i need to follow in case of restoring controller data to new 1 node cluster...
m
do you see any errors in the logs (
logs/pinot-all.log
) when you install the new cluster?
s
NO
everything is proper in new cluster...
m
just with no data!
s
yes exctly
m
let me have a look at the code.. What you've done seems like it should work
s
ok
I just added table, with no data hoping data will get downloaded from hdfs deep storage once controller is started in new cluster...but no...
m
how did you load the data in?
I'm thinking that you could use an ingestion job that just downloads the segments from HDFS to that table
Copy code
executionFrameworkSpec:
  name: 'standalone'
  segmentTarPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentTarPushJobRunner'
jobType: SegmentTarPush
outputDirURI: '/path/to/hdfs'
pinotFSSpecs:
  - scheme: hdfs
    className: org.apache.pinot.plugin.filesystem.HadoopPinotFS
tableSpec:
  tableName: 'airlineStats'
pinotClusterSpecs:
  - controllerURI: '<controller-addresss>'
s
i dont need whole segment here, i dont maintain an segments locally, thats why we pushed to hdfs deep storage
will check above thing
m
the hdfs deep storage is the persistent storage, but the Pinot Servers will still have a copy of the segment data for serving queries
but yeh in my example I'm storing deep store on the controller
just for simplicity
s
yes
Caused by: java.lang.IllegalArgumentException: Wrong FS: hdfs://akram/pinotcommon/segments/airlineStats, expected: file:/// at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:647) at org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:82) at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:601) at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:819) at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:596) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:421) at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1424) at org.apache.pinot.plugin.filesystem.HadoopPinotFS.listFiles(HadoopPinotFS.java:146) at org.apache.pinot.plugin.ingestion.batch.standalone.SegmentTarPushJobRunner.run(SegmentTarPushJobRunner.java:76) at org.apache.pinot.spi.ingestion.batch.IngestionJobLauncher.kickoffIngestionJob(IngestionJobL
scheme - i gave hdfs fyi
m
what does the whole file look like? Exclude any specifics about your en
env
maybe put scheme to 
file
s
how scheme-file will work, as deep storage is hdfs, it should be hdfs, right?
segmentTarPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentTarPushJobRunner' jobType: SegmentTarPush outputDirURI: '/akram/pinotcommon/segments/airlineStats' scheme: hdfs className: org.apache.pinot.plugin.filesystem.HadoopPinotFS
after specifying hadoop.conf.path - hadoop/hdfs/hadoop/etc/hadoop/ it is working
but Start pushing segments: []... to locations: [org.apache.pinot.spi.ingestion.batch.spec.PinotClusterSpec@75769ab0] for table airlineStats
empty segments there
m
are all the segments empty?
s
nothing got created
i mean nothing downloaded
tart pushing segments: []... to locations: [org.apache.pinot.spi.ingestion.batch.spec.PinotClusterSpec@75769ab0] for table airlineStats
m
oh I see
s
got above print after finishing
m
hmm ok. We can try changing the 
segmentTarPushJobRunnerClassName
to 
org.apache.pinot.plugin.ingestion.batch.hadoop.HadoopSegmentTarPushJobRunner
s
one sec
still same problem
empty segments created
m
it's not finding the segments in HDFS - is that path correct?
trying to figure out why it's not reading the segments
s
yes
m
and the files in HDFS all have the 
.tar.gz
suffix?
s
no
m
oh
s

https://apache-pinot.slack.com/files/U0208LD4LLA/F02QRBKM62H/screenshot_from_2021-12-15_17-37-01.pngâ–ľ

m
ok now I'm confused then! The files in the deep store should have the 
.tar.gz
extension.
s
i didnt create this, when i configured controller with hdfs deepstore, it automatically created there
without .gz format
m
what do you have this set to: 
pinot.server.instance.segmentTarDir
?
it's weird b/c the files you shared there look like what normally goes on local storage even though it isn't
s
pinot.server.instance.dataDir=/path/in/local/filesystem/for/pinot/data/server/index pinot.server.instance.segmentTarDir=/path/in/local/filesystem/for/pinot/data/server/segment
i have not configured
m
ok
s
because i dont want to maintain locally
if i configure above 2 properties, data will get duplicated again
controller - segments in deepstore server - in local only 2 copies i am maintaining
m
I didn't know that you can configure Pinot to not have a local copy of the data. I thought it needed to have a local copy to serve queries.
in the deep store you should have files with the 
tar.gz
suffix. I don't really understand why that isn't the case
s
if data got deleted from one machine and restarted controller/server then its restored from hdfs
problem is with another machine
@Xiang Fu
m
If I understand correctly, you created a brand new cluster and table and just pointed the controller data dir to hdfs? If so, that won’t automatically push data to newly created table
You need to explicitly repush that data
s
in case, if i have same old zookeeper and i am running controller,broker,server newly... in that case how can i restore data from hdfs?
@Mayank
basically my req is i have 1 node cluster,(assuming zk is in separate node).. and configured hdfs as deep storage... if that pinot cluster goes down(controller, broker, server), then i need to install new one node cluster with same pinot hdfs controller segments path... where segments can be restored using this. what should be done or what is the procedure for attaining this... we have 10's of TB's of data... @Mayank
m
You want to run Pinot in HA mode (replication of 3 recommended) in production.
2 Views
Open in Slack
PreviousNext
Powered by