https://docs.all-hands.dev/ logo
Join Slack
Powered by
also more importantly: I did not get any indicatio...
# proj-agent-canvas
w
also more importantly: I did not get any indication I was out of credits until I refreshed the page
@millions-afternoon-47582 this problems seems to come up constantly. I think it's been fixed 3-4 times. Do we need to rethink the logic here? do we need better tests?
m
Hello @wooden-umbrella-84098, Unfortunately, the issue has not been fully resolved yet, and the flash still appears after 500ms. After you informed me to look into it, I created a ticket to track the issue. However, as I am currently supporting multiple projects, I haven’t yet had the opportunity to work on a fix for this. I will make it a priority and ensure that the fix is available and included in the next release. I apologize for the delay and any inconvenience this may have caused. Thank you very much! thankyou
w
👍 so long as you're tracking it I'm happy!
thankyou 1
thanks for the update
thankyou 1
c
Sorry to intervene on the run. I don't see an issue on GitHub. OK, I should note here: in my understanding, the billing error has been sent to the FE as an 
AgentErrorEvent
, which I think maybe it's not the right thing to do, conceptually. • 
AgentErrorEvent
is just a transitory LLM blunder that the agent just fixes next step • application-level errors should not reuse it. They are application-level errors. As long as we use one for the other, the result is error-prone and bugs keep creeping up everywhere. Please note that these 3-4 times that the billing error returned, are linked to 3-4 times (and still not fully solved) of the complementary issue: the huge red ERROR when the LLM makes a blunder, and the huge toast when the LLM makes a blunder, none of which should happen, I think
I think maybe we should ideally have 2 very distinct concepts: • agent errors (that the app-server / enterprise / FE should almost not care about it at all) • application-level errors
@millions-afternoon-47582 If you don't mind, could you perhaps let me know when we have a fix, I'd love to know that the architecture level stuff here is fixed?
m
Hello @calm-artist-7206, Thank you for your message. Please give me a few minutes, as I am currently working on some ongoing tasks. I will respond to your messages shortly. Thank you very much! thankyou
🙏 1
Hello @calm-artist-7206, I apologize for the delay in my response. At the moment, it seems that the 'Run out of credits' billing error is not being sent as an AgentErrorEvent. Here’s an outline of the current error flow in the SDK (software-agent-sdk): 1. Origin: When the user exceeds their budget, the LiteLLM proxy returns a BadRequestError with the message "ExceededBudget." 2. SDK LLM Layer: The BadRequestError is mapped to a LLMBadRequestError. There is no specific handling for budget or credit-related errors, and they are treated as a general bad request. 3. agent.step(): The error is propagated up because the LLMBadRequestError is not caught, as the current flow only catches FunctionCallValidationError and LLMContextWindowExceedError. 4. local_conversation.run(): In this layer, a generic Exception handler catches the error and emits a ConversationErrorEvent. This is the correct application-level error type, and The docstring on ConversationErrorEvent explicitly says: "Conversation-level failure that is NOT sent back to the LLM... intended for client applications (e.g., UIs) to present a top-level error state." 5. Frontend: The isDisplayableErrorEvent check matches the ConversationErrorEvent, and isBudgetOrCreditError() checks if the error message contains "budget" or "credit," displaying the appropriate internationalization (i18n) message. With regard to the issue Robert mentioned, it appears to be entirely front-end related, specifically concerning the way we handle the removeErrorMessage functionality on the front-end. Thank you very much! thankyou
c
That sounds just right to me! Except that, on FE side, we are looking for this error inside 
AgentErrorEvent
, because that's where FE expects the billing error, it seems. As you clearly proved here, it can never be there. 😅 I see that the FE code does:
Copy code
// Track credit limit reached if AgentErrorEvent has budget-related error
          if (isAgentErrorEvent(event)) {
            trackError({
              message: event.error,
              ...
            // Use friendly i18n message for budget/credit errors instead of raw error
            if (isBudgetOrCreditError(event.error)) {
              setErrorMessage(I18nKey.STATUS$ERROR_LLM_OUT_OF_CREDITS);
              trackCreditLimitReached({
                conversationId: conversationId || "unknown",
              });
            } else {
              setErrorMessage(event.error);
It'll never find it. Can we remove this code, so it stops confusing us? Also, please note, it seems the 
setErrorMessage
in that 
else
block makes the red banner appear? For all 
AgentErrorEvent
, but not billing, which seems... upside down. 😅
m
Yes, I noticed the same thing while researching to answer your question. I will re-review that part while addressing the issue. Thank you very much! thankyou
thanks on a stick 1
c
I just see another time when there’s a banner and a toast, which I’m pretty sure were meant for billing/conversation errors, and they happen instead for 
AgentErrorEvent
It’s going to be awesome to get rid of this confusion, thank you for looking into this!
❤️ 1
thankyou 1
2 Views
Open in Slack
PreviousNext
Powered by