Hey guys, I am using python for my livekit agent and i am trying to let the agent launch a headless browser, share screen and do the actions what user's asks for, so far so good every thing works like navigating to a website and basic interactions like clicking buttons and scrolling a page.
Right now i want my agent to be contextually aware as well, to do that i use the content extracted from the page, but it losses the ability to judge the layout which is very important for my use case, i can capture the screenshot but i am not sure how to feed that to the llm. Should i upload it somewhere and the pass the link idk.
I'd appreciate advice if somebody implemented something like this 🤝