I am playing with the vision agent. Is it possible to connect one participant’s voice and another participant’s video to the vision agent?

Hi Lindskog Work, ⚠️ This channel will be deleted soon ⚠️ Please use our new community at https://community.livekit.io for technical discussions and to post technical questions. The Slack community is not going away, but we are migrating technical questions to our new forum. Please see full details here.

Yes thats possible, you only expose the specific tracks that you want to the agent to see

I was thinking of using room_options, maybe it is not the right way? await session.start( room=ctx.room, agent=Assistant(), room_options=room_io.RoomOptions( participant_identity=“identity”, ), )

no you'll have to deal with the individual tracks to make it work

I am a beginner, can you give me some tips please?

can you share your usecase

I have an ingest that provides video. I want to connect the video to the Gemini agent but I want my voice to control the Gemini agent so I can ask questions about the content of the ingest.

Ok, I think there are helpers already to do this, try setting

audio_input

to true in the RoomOptions

Sorry, this was not clear to me. I hope you can give a bit more details. I start the session like this with

audio_input=True

.

await session.start(

room=ctx.room,

agent=Assistant(),

room_options=room_io.RoomOptions(

audio_input=True,

video_input=True,

participant_identity=VOICE_PARTICIPANT_IDENTITY,

)

)

Then when the

ingress

participant shows up, I set it as this.

session._room_io.video_input.set_participant(ingress)

Though, I couldn’t get it work. The ingress is “`no_transcoding`” and the source is H264. Maybe that is the reason or my logic of setting up sources is wrong.

The session configuration looks fine. You don't need the

session

call - the participant identity can be set to a static string (as part of the access token). Ideally this should work - what happens when you start the agent

I got this working. The problem was H264. The video participant was H264 with WHIP ingestion. I converted that to VP8. Doesn’t agent SDK support H264?

it does, it could be related to WHIP transcoding: https://docs.livekit.io/transport/media/ingress-egress/ingress/transcode/#enabling-transcoding-for-whip-sessions

transcoding is off on the endpoint though. I can verify it with egress that it is h264.

ok

For the reference, this is how I connected it all.

await ctx.connect()

await session.start(

room=ctx.room,

agent=Assistant(),

room_options=room_io.RoomOptions(

audio_input=True,

video_input=True,            participant_identity=VOICE_PARTICIPANT_IDENTITY,

)

)

# Wait for both participants to be present in the room.

await ctx.wait_for_participant(identity=VOICE_PARTICIPANT_IDENTITY)

ingress = await ctx.wait_for_participant(

identity=INGRESS_PARTICIPANT_IDENTITY,

kind=rtc.ParticipantKind.PARTICIPANT_KIND_INGRESS,

)

if session._room_io and session._room_io._init_atask:

await session._room_io._init_atask

# Override only the video input to the ingress. Audio remains on the voice participant.

if session._room_io and session._room_io.video_input:

session._room_io.video_input.set_participant(ingress)

<http://logger.info|logger.info>(

"video input linked to ingress, audio input linked to voice participant",

extra={"ingress": ingress.identity, "voice": VOICE_PARTICIPANT_IDENTITY},

)

await session.generate_reply()