This message was deleted.
# ask-for-help
s
This message was deleted.
👀 1
c
I think that depends on what’s causing the 500 internal server error. Could you share the error message in the logs?
j
Ah, interesting. It’s a Pydantic root validation error in this case.
The reason I ask is because my service takes a significant amount of time to start up and so I’d like to know when the service will be automatically restarting
c
I see, I’d recommend try-catch this error in the API service code and handle the exception. It sounds like an error with input validation?
j
Thanks Chaoyu, yes I am working on a fix for this particular issue, but I’m wondering about unforeseen errors in the future. I do have a testing framework in place, but there will probably be the occasional error that slips through. It sounds like that it depends on what is causing the 500 — you’re saying that not all 500s will restart the service?
@Chaoyu it also restarted after getting this error:
Copy code
Traceback (most recent call last): File "/home/giants/.local/lib/python3.7/site-packages/tornado/ioloop.py", line 921, in _run val = self.callback() File "/home/giants/.local/lib/python3.7/site-packages/circus/util.py", line 1039, in wrapper % arbiter._exclusive_running_command) circus.exc.ConflictError: arbiter is already running arbiter_stop command
I’ve always gotten this error, and when I tried to see how to fix it, it seems there’s not a good fix for it. It’s less about fixing this specific error and more about the fact that the service restarts at certain times and then while it’s restarting, my request lags.
I’m not sure if it’s related but I have (purposefully) cpu-throttling set to false.
Also, when I say my service restarts, I’m referring to the service.py script being run again. I know this is happening because some of the time consuming tasks which I have logs for are showing up in the logs. It just got re-run for no apparent reason (even after only a handful of 200 requests). So my question is: for what reasons / when does service.py get run?
@Chaoyu would it be possible to get 30 minutes on the calendar with you or someone else on your team? I wonder if my question is just misguided to begin with, and maybe if I walk you through the code that something else will jump out. Monday, Tuesday, or Wednesday would be particularly good days for me. 🙏
j
Hello @Jack Norman. BentoML Eng here.
Copy code
circus.exc.ConflictError: arbiter is already running arbiter_stop command
This is a known and harmless messaging bug which will display every time the server stopped. It shall not be the cause here.
I've received the service code, but it seems I don't have the models so I can not reproduce it. Would you mind sharing a exported bento? You can get it by calling
Copy code
bentoml export <your_bento_name>