Hi Team Pinot server went down when checked it fai...
# troubleshooting
b
Hi Team Pinot server went down when checked it failed with following error in screenshot. Is it because of memory issue ? Will i loss data if i restart the server with following command ?
Copy code
bin/pinot-admin.sh StartServer -configFileName conf/pinot-server.conf
x
do you have pinot server logs file
b
Above was the server log which was seen when i logged in But now I'm unable to login into EC2 instance which runs the Pinot. Hence cant access latest log file. But I'm able to see pinot dashboard up and running. can i get any insight from the dashboard ?
b
That error message means JVM crashed on a memory issue. It's not possible to say what exactly without the crash log file.
b
Thanks for responding, Actually i got connected with AWS support team and got to know its because of not having enough space in EBS volume. But I do have follow up question regarding the same I have enabled deep store , so all committed segment would have a backup in S3. Is there way to delete older segment so limited data will only get cached in EBS volume and rest can be found in S3 bucket. Does this approach work?
y
Nope, segments can't be queried directly from deep store. What you're referring to would only be possible with StarTree's tiered storage, see https://docs.startree.ai/corecapabilities/manage-data/set-up-tiered-storage/architecture.
b
I understand this feature is only available in StarTree cloud. But for now is there DIY process to make it happen Just have a idea let me know if is feasible to make it happen. I will have cron job which delete older segments which are cached in local. I already have a Deepstore setup which act as a backup so if segment is not cached in local it will fetch from S3 deepstore. let me know if it sounds possible ?
x
in that case, you can try to mount s3 to your local using POSIX so pinot server might treat s3 as local file system. Then you will have local fs tier and s3 tier Note, I haven't tested or tried this approach and not sure if supported s3 posix apis are sufficient for Pinot. But you may give a try https://docs.aws.amazon.com/AmazonS3/latest/userguide/mountpoint.html
b
Appreciate your idea, let me check on this.