https://livekit.io logo
Join Slack
Powered by
I am currently experimenting with `video-processor...
# sdk-web
w
I am currently experimenting with 
video-processor.js
to implement virtual backgrounds. I am trying to ensure that participant video tracks and background images are mirrored correctly. At the moment, local users are correctly shown a mirrored view of themselves, but when a virtual background is applied, the background is also mirrored, which is not the intended behavior. If I pre-mirror the background image to compensate, remote participants then see an incorrectly mirrored result. How can I configure the video processing so that the local self-view is mirrored while the background and the video sent to other participants remain correctly oriented? I am using React.
r
This is expected and intended behavior. It is common that the local view of the person is mirrored as well as the background while the remote view is not mirrored.
w
Yeah exactly, but this becomes a problem when the background contains text or similar since that is also going to be mirrored for the local user. Is there a way to fix this problem somehow? Or am I missing something here? Currently we mirror the local user with css and tailwind
r
I think you maybe able to do it in CSS for the local participant view
(we are pretty far outside my area of expertise)
Maybe something like this:
Copy code
// Local self-view (mirrored for display)  
<div style={{ transform: 'scaleX(-1)' }}>  
  <VideoTrack trackRef={localTrackRef} mirror={false} />  
</div>  
  
// Remote views (no mirroring)  
<VideoTrack trackRef={remoteTrackRef} mirror={false} />
w
Hmm 🤔 do you think it's just gonna mirror only the video track and not the background?
We do something similar today but the css mirrors the whole DOM/element and not only the video track. You haven't heard of this problem from other users?
r
it is intended behavior
I've not heard of it being a problem for someone but not sure if it has come up before.
w
But when you say intended behavior do you mean it's expected to have a mirrored background for the local user?
r
yes. Try doing it in Google Meet for instance
Local is mirrored
w
But even for backgrounds? That sounds super odd to me? But I will of course check this out in meets
r
That is how Video local view works commonly across video conf platforms
I am not sure if it is 
super odd
or not. But it is the norm
w
Hmm 🤔 very interesting to say the least! But then from your guys perspective and with livekit there isn't a way of having the local video track mirrored and background non-mirrored?
r
I don't know for sure. I expect there is a way. But I am very outside of my area of expertise
w
No don't worry about it. I just need to sorta priority this with my team but if this is the norm, we are good with it. But I will investigate because of curiosity 😅
r
What is the issue with unmirror the entire view lcoally?
w
Not a problem really
Other than users expect to see themselves mirrord
nodding 1
r
ok. The video is composite so it is hard to just undo one and not effect the remote
w
But from a technical perspective, the mirroring logic should only happen locally for the local user right?
Like when we try to render the UI
r
I just captured this from this google meet https://meet.google.com/ojn-qiap-zrw. The LK logo is backward from my view but others see it correctly. Same as if you were looking in a mirror which I believe is what humans have come to expect. Same if you look at a live feed of your camera and a "real" background
w
Ooooh so interesting. So right now when you tilt your head right it actually goes left for you?
r
no it goes in the expected direction like if you were looking in a mirror.
IMG_7470.MOV
Same in LiveKit meet
IMG_7471.MOV
You will have to talk to whoever created real mirrors about why they did not un-mirror the background 🤣
w
Beautifully tested, thank you for taking your time with this. Hahaha but couldn't you agree that from a ux perspective you would have the same "background-experience" as remote users? I wonder how many users have reported that "my virtual background is mirrored, do my peers have the same experience?"
Or maybe this could be just me haha
r
I don't really agree. I think we expect things a certain way. If you point over your shoulder at something you expect it to be there and not on the opposite side of reality
But I understand why you don't like mirrored text locally...
w
True but I guess the concept of real vs virtual is the question here? Because I agree that in the real world it makes sense for everything to be mirrored. But since this is virtual I don't feel the same thing
But it's gonna be fine, if Google does it, it's not a problem
r
That is perfectly acceptable way to feel
w
Again I was just curious if this has been discussed or if more people have had this "issue"
r
My suggestion is this from a non confusing UX perspective. I would give your user the option to mirror/un-mirror but that will apply to the entire local video feed.
It was a fun discussion. It is good to dig into these things sometimes
w
Yeah, I am guessing we have the option of mirror everything or nothing
And google mirrors everything
nodding 1
r
Made me think about why I believe something 🙂
Yea, figure out what works best for your usecase. At the end of the day you can do what you want since you have all the source there.
w
Alright, thank you for your time and help! I will bring this along to the team
👍🏼 1
r
I wish I knew more about that layer but I am not exaclty sure where you start if you want to solve this
w
I was wondering if we could do more with the processors but it's just gonna generate one "picture" and can't see the difference between background and video track
And we don't really want to send mirrored images at all because remote users are never going to want that
💯 1
Awesome, again thank you for your help and quick answer
👍🏼 1
a
Sorry for jumping in, but this is fun topic... I was building video conferencing from the scratch and having yourself un-mirrored is an EXTREMELY weird experience. You are simply not used to seeing yourself in real time as others see you. It's fine on video, but Real time, crazy ... you have to try it 😊
😀 1
w
Haha yeah we have had this conversation within our team as well and when this happens in a video call everything just feels broken 😅
2 Views
Open in Slack
PreviousNext
Powered by