Hey guys, I am using python for my livekit agent a...
# agents
e
Hey guys, I am using python for my livekit agent and i am trying to let the agent launch a headless browser, share screen and do the actions what user's asks for, so far so good every thing works like navigating to a website and basic interactions like clicking buttons and scrolling a page. Right now i want my agent to be contextually aware as well, to do that i use the content extracted from the page, but it losses the ability to judge the layout which is very important for my use case, i can capture the screenshot but i am not sure how to feed that to the llm. Should i upload it somewhere and the pass the link idk. I'd appreciate advice if somebody implemented something like this 🤝
l
You could use a model that supports vision, for example we have this demo: https://github.com/livekit-examples/python-agents-examples/tree/main/complex-agents/vision that uses grok-2-vision. You can pair this with our https://github.com/livekit-examples/agent-starter-react to share your screen with the agent and it will be able to describe what it is seeing. I swear I remember also seeing a video from somebody accessing the video tracks in one of their demos... ah here it is:

https://www.youtube.com/watch?v=_0PcGMxw_Hs▾

e
Thanks, but my usecase is a bit different, instead of users sharing the screen and asking the agent what they see, instead my agent launches a headless browser, shares its screen and do stuff on behalf of the agent, i just discovered it how to do it few hours back. But thanks for your response.
🙌 1