Hello, I am getting this error on one of the realt...
# troubleshootingt
Tanmay Movva
12/02/2020, 4:31 AMHello, I am getting this error on one of the realtime tables
Copy code
ERROR [LLRealtimeSegmentDataManager_spanEventView__0__15__20201201T0448Z] [spanEventView__0__15__20201201T0448Z] Could not build segment
java.lang.IllegalStateException: Cannot create output dir: /var/pinot/server/data/index/spanEventView_REALTIME/_tmp/tmp-spanEventView__0__15__20201201T0448Z-160688315943Tanmay Movva
12/02/2020, 4:37 AMEven when pinot is not able to ingest and the lag is increasing, the table and segment status on UI is
GOODTanmay Movva
12/02/2020, 4:39 AMbtw, the state of this segment is
CONSUMINGk
Kishore G
12/02/2020, 4:53 AM
Any things else in the log?
t
Tanmay Movva
12/02/2020, 4:54 AMNo info logs were not enabled. Only error was the above one.
But I checked at the path it is trying to create a directory. There already exists a directory in the
_tmpk
Kishore G
12/02/2020, 4:55 AM
Are any other segments getting created?
t
Tanmay Movva
12/02/2020, 4:55 AMYes. Only this table was affected.
k
Kishore G
12/02/2020, 4:56 AM
I am guessing you have enough disk
t
Tanmay Movva
12/02/2020, 4:57 AMYes.
x
Xiang Fu
12/02/2020, 4:58 AM
is this directory local or on remote?
t
Tanmay Movva
12/02/2020, 4:59 AMThis directory is on the volume attached to the pod. So local.
x
Xiang Fu
12/02/2020, 4:59 AM
your volume is local disk or remote like ebs?
t
Tanmay Movva
12/02/2020, 4:59 AMebs volume.
x
Xiang Fu
12/02/2020, 5:00 AM
all are segment persist failed on that volume?
Xiang Fu
12/02/2020, 5:01 AM
can you try to create a file on that volume
Xiang Fu
12/02/2020, 5:01 AM
try to access it through the pinot server container ?
t
Tanmay Movva
12/02/2020, 5:02 AMWas able to do that. Remaining realtime segments are still able to ingest and serve.
x
Xiang Fu
12/02/2020, 5:04 AM
hmm
Xiang Fu
12/02/2020, 5:05 AM
is there any log inside the pinot server container
Xiang Fu
12/02/2020, 5:05 AM
there is a file pinotServer.log
t
Tanmay Movva
12/02/2020, 6:04 AMI deleted and recreated the table and now everything is running fine. Not sure what was the issue.
@Xiang Fu I will check the pinoServer.log and share if I find anything.
Tanmay Movva
12/02/2020, 6:07 AMOne more thought, when the freshness metric for a table is increasing for any reason(server unresponsive, error connecting to kafka, etc,.) Shouldn’t the status of table be “not GOOD”?
When I checked the table and segment status on UI when facing this issue all status were
GOODx
Xiang Fu
12/02/2020, 6:10 AM
agreed
Xiang Fu
12/02/2020, 6:10 AM
if you can provide more info, then we can fix this
Xiang Fu
12/02/2020, 6:10 AM
also need to distinguish this with the scenario like kafka upstream has no data coming
Xiang Fu
12/02/2020, 6:11 AM
in certain cases, we need to define what is “good” status
t
Tanmay Movva
12/02/2020, 6:13 AMalso need to distinguish this with the scenario like kafka upstream has no data comingIn that scenario, comparing last consumed offset in pinot and latest available offset in kafka should help, yes? Also I am assuming the current freshness metric is measured from current time. If the data is not available in kafka, then this would keep on increasing.