https://pinot.apache.org/ logo
Join Slack
Powered by
Question about split commit: we enabled the config...
# troubleshooting
e
Question about split commit: we enabled the config on the controller, server and table (i.e. 
peerSegmentDownloadScheme
= 'http'), and see 
"isSplitCommitType":true
log messages in the server but noticed the controller still contains those segments in the temp directory 
/var/pinot/controller/data/untarredFileTemp
- does that indicate we don't have something configured incorrectly?
I do see messages in the server like this:
Copy code
pinotServer.2021-12-17.1.log.gz:2021/12/17 02:49:20.239 INFO [PinotFSSegmentUploader] [XXX_v3__2__236__20211217T0235Z] Successfully upload segment XXX_v3__2__236__20211217T0235Z to <gs://XXX/XXX//XXX_v3/XXX_v3__2__236__20211217T0235Z7be8b1e0-64f5-4c55-a814-5de96d5c37b6>.
Why would the controller also be downloading the segment and untarring it?
m
I've been trying to understand this bit of the code and I reckon that this line creates the segment committer: https://github.com/apache/pinot/blob/master/pinot-core/src/main/java/org/apache/pi[…]ot/core/data/manager/realtime/LLRealtimeSegmentDataManager.java That then takes you over here: https://github.com/apache/pinot/blob/master/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/SegmentCommitterFactory.java#L64 And if I'm understanding it correctly, if you only set split commit, it will still use the 
Server2ControllerSegmentUploader
, which means that the tar.gz version of a segment is sent to the Pinot Controller, which then sends is to the deep store. I think the other parameter that you need to set in your table config is 
peerSegmentDownloadScheme
- is that actually what you set and you had a typo above?
e
Hi, thanks for sharing this info, much appreciated! I made a typo in the message, verified that it's 
peerSegmentDownloadScheme
.
m
hmmm so in that case it does seem like it shouldn't be having segments on the controller
e
Yep, and I verified that the PinotFSSegmentUploader is uploading the segment, from the log message above, but also continue to observe segments from those same tables in the untarred dir on the controller.
It looks like it could be from the LLCSegmentCompletionHandlers -either 
segmentCommit
or 
segmentCommitEndWithMetadata
- it doesn't appear to upload the segment but definitely untars it. One of those methods gets triggered - I'll enable more logging on the controller to investigate.
m
e
Yep, those seem to be the only 2 methods that call it - we're using pinot-0.8.0.
m
PinotSegmentUploadDownload is the other place
ah ok
is it only meta data in that folder?
or is it the full segment?
i.e. 
metadata.properties
and 
creation.meta
files
e
I see the full segment getting untarred
If I reenable info logging on the controller I could check for those log messages in the method you mentioned: 
uploadSegment
It seems based on the fact that it's the full segment that it's likely the PinotSegmentUploadDownload method is being called, right?
m
yeh although the location of where it's unpacking is weird given it's the full segment
as the path you mentioned only seems to be used by metadata 🤔
e
Yep:)
m
I'm looking at the code in master though
let me go back to 0.8
👍 1
e
Thanks for helping with this
m
is this config set to anything? 
controller.local.temp.dir
e
Yes, it's set to a local ssd mount.
m
but not 
/var/pinot/controller/data/untarredFileTemp
?
just wondering as that's where the controller would unpack a segmetn
from what I can tell
e
There are 3 directories that I see under that directory:
Copy code
fileDownloadTemp  fileUploadTemp    untarredFileTemp
But I only see files in the untarredFileTemp directory
We do have some tables which don't set the peer download scheme, so files appear and quickly disappear in the 
fileUploadTemp
also
m
I think for the non peer download ones, the way it works is: 
(server) -[:sends_segment]-> (controller) --> [:sends_segment] --> (deepstore)
for the peer download it should be skipping that middle step (and it seems like it does from your server logs)
but you also have some peer download segments permanently staying in that dir?
e
No, they just appear temporarily and disappear
I see segments stay around the untarred directory for a longer period but those eventually disappear as well.
m
@Mayank will probably know what's going on. I can't really see why the peer downloaded segments should go there at all
e
Sounds good, really appreciate all the help!
I just added back info logging, will update with findings.
Happy friday:)
m
cool! I wanna update the deep store page explaining how all the interactions work once I understand it!
🙏 2
e
That would be great
Open in Slack
PreviousNext
Powered by