Ok so an update on using the minions to ingest, af...
# troubleshootingn
Nick Bowles
02/19/2021, 9:35 PMOk so an update on using the minions to ingest, after small changes I see this in the logs. The tar.gz file exists in the bucket, but it looks like it tries to push anyways to the path
/segments/blah.tar.gzx
Xiang Fu
02/19/2021, 9:41 PM
looks like this is the push job type issue. Default push is segment tar push
Xiang Fu
02/19/2021, 9:41 PM
Copy code
Trying to push Pinot segment with push mode tar from <gs://bucket/segments/mytable_OFFLINE_REDACTED_10.tar.gz>Xiang Fu
02/19/2021, 9:41 PM
we should change it to metadata push or uri push, let me double check
n
Nick Bowles
02/19/2021, 9:43 PMWhere is it trying to push to? I don’t see any changes to what’s in the deep storage bucket configured on the controller so not sure.
Nick Bowles
02/19/2021, 9:43 PMand thanks!
x
Xiang Fu
02/19/2021, 9:44 PM
can you add a config to ingestion job:
Copy code
push.mode=metadataXiang Fu
02/19/2021, 9:44 PM
in parallel to input.uri, etc
n
Nick Bowles
02/19/2021, 9:45 PMWhen it’s set to metadata, what is the behavior vs the default of segment tar push?
Nick Bowles
02/19/2021, 9:45 PMI would think the default would push the tar to the output bucket I have set on the controller
x
Xiang Fu
02/19/2021, 9:46 PM
metadata/uri mode will not try to save segment in controller
Xiang Fu
02/19/2021, 9:46 PM
segment tar push expects a controller deep store to save all the tar file
Xiang Fu
02/19/2021, 9:47 PM
I still need to investigate why tar push mode failure
n
Nick Bowles
02/19/2021, 9:49 PMDo I maybe need to define that deep store somewhere in the batchConfigMap or the minion?
Nick Bowles
02/19/2021, 9:49 PMThanks for clarifying
x
Xiang Fu
02/19/2021, 10:04 PM
I think if your minion is started with deepstore credential then should be fine
Xiang Fu
02/19/2021, 10:04 PM
otherwise, the batchConfig should contain the credentials
Xiang Fu
02/19/2021, 10:05 PM
uri push is lightweight on client(minion) side, controller will download segment and extract segment metadata
Xiang Fu
02/19/2021, 10:06 PM
metadata push is light weight on controller side, client(minion) will download the segment based on uri and extract metadata, then upload metadata to controller, no controller deep store download involved
n
Nick Bowles
02/19/2021, 10:09 PMThis is great info. Thank you so much I really appreciate it.
x
Xiang Fu
02/19/2021, 10:11 PM
You are welcome! I’m also going to add this into pinot docs
n
Nick Bowles
02/19/2021, 10:19 PMI can try and add that this weekend or at least get some of the writing out of the way.
x
Xiang Fu
02/19/2021, 10:22 PM
sounds good! Many thanks!