https://livekit.io logo
Join Slack
Powered by
This message was deleted.
# helpdesk
s
This message was deleted.
i
Also, if it matters, I am using Unity version 2021.3.21f1 with the Universal Rendering Pipeline (URP).
And also, the "washed out" issue may not be exclusive to Firefox - it might be app-wide regardless of the web browser used. But I am not sure yet since I haven't seen it work in Chrome quite yet.
I'm still very early in my implementation, so there's a chance I may not be doing something right. However I would expect to see something in Chrome if it's working in Firefox.
d
For 1. the colors to the left appear to be much brighter. are you screen sharing this from another LiveKit client? if so, which one? For 2. that's rather unexpected. do you have any logs in the console that doesn't look right? i.e. any errors or warnings.
i
For #1, everything is done in a Unity WebGL build. One Unity WebGL app is screen sharing to another app instance. To me this looks like something could be wrong with the "color space" of the video texture in Unity (or the "format"), but I am not sure as I don't have a full understanding of the internals. For #2 I don't see anything suspect. There are no errors or warnings in the console log.
I put in a console log for when the video texture is updated. The console log reports the resolution of the video texture and its format. When screen share begins, looks like a video texture is being created 3 times in a row (Chrome):
In Firefox (which works), looks like it's being created 2 times in a row, and maybe these warnings might mean something?
In this test I am sharing my entire screen (a 1920x1080 monitor).
d
this is helpful. thanks! we'll look through and see if we can figure out the Chrome issue. To be clear, you are screen sharing from FireFox, and it's showing up as blank in Chrome as the viewer?
i
Thank you, I really appreciate it! Specifically I am screen sharing from Chrome (using very similar code from the example documentation and code samples), and the video shows up as a black screen in other receiving app instances also on Chrome. The video works in Firefox. Audio works fine on both.
👍 1
And @boundless-energy-78552 if there's anything specific that I can do to help please let me know. Happy to help any way I can! I'm sure we can get to the bottom of it. 🙂
❤️ 1
b
Hey @important-megabyte-62876, do you keep a strong reference to the HTMLVideoElement on the C# side?
Can you try to add a console.log()/breakpoint here?
i
Hey @boundless-energy-78552, I added a console log where you mentioned (and also double checked the app build to make sure the console log is actually there) but I don't see it in the browser console. 🤔 I don't believe I'm keeping a strong reference to the HTMLVideoElement in my C# code? I am subscribing to 
video.VideoReceived
via a lambda function and only using the texture. Here is my C# code for when a track is subscribed:
Copy code
private void OnTrackSubscribed(RemoteTrack track, RemoteTrackPublication publication, RemoteParticipant participant)
{
	if (track.Kind == TrackKind.Video)
	{
		var video = track.Attach() as HTMLVideoElement;

		// VideoReceived is called every time the video resolution changes
		video.VideoReceived += tex =>
		{
			OnGotNewScreenShareVideoTexture(tex);
		};
	}
	else if (track.Kind == TrackKind.Audio)
	{
		track.Attach();
		Debug.Log("Got new audio track!");
	}
}

// This function creates a new material with the video texture, then applies that material onto a target renderer.
private void OnGotNewScreenShareVideoTexture(Texture2D screenShareVideoTexture)
{
	debugTargetRenderer.material = MaterialUtils.CreateNewMaterialUnlitMipMapBias(screenShareVideoTexture);
	Debug.Log("Got new screen share video texture! ("+ screenShareVideoTexture.width + "x" + screenShareVideoTexture.height + " Format: " + screenShareVideoTexture.format.ToString() + ")");
}
b
Which operating system are you on?
Gonna try to reproduce the issue
i
Windows 10
👍 1
Sounds good, let me know if there's anything else I can help with!
🙏 1
My full test conditions are Windows 10, Unity 2021.3.21f1 + Universal Rendering Pipeline (URP). Testing 3 instances of the same WebGL build at the same time (hosting the app on an Amazon AWS S3 storage bucket). Two of the instances are running Chrome, and one instance is running Firefox. I initiate the screen share from one of the Chrome instances. Then for the two other instances, I see a black screen on the receiving Chrome instance, and the receiving Firefox instance I do see the screen share.
Hi @boundless-energy-78552, were you able to reproduce the issue?
b
Hey I just tested using 2 chromes instances and I can’t reproduce the issue. I used the example app inside the sdk repo. I’m using Unity 2021.3.23f1. Can you try to run this demo and see if you have the issue? For the difference between the color space, can you try to change the second boolean of this line to 
false
. The colors seem normal for me but I maybe need a way to change the color space depending on some conditions. I’m using my camera instead of screenshares but there is no difference on how I handle the texture creation on the receiving side. On the Chrome console of the video receiver, do you see any WebGL errors or warnings? Maybe something related to texImage2d
I tried on my Macbook, I will retry to reproduce the issue on my Windows machine tomorrow
i
Hey @boundless-energy-78552, thank you so much for the update! I will do as you mention and investigate further via the example app, then report back. Could you tell me which line to change in 
HTMLVideoElement.cs
, specifically? Looks like your link didn't point me to it. I do not see any errors or warning in the browser console on Chrome for the receiver app - only a black screen. My app differs from the example app in that I'm: 1. Creating a new unlit material (duplicating it from an existing "template" unlit material) 2. Applying the video texture to the material 3. Applying the material to the renderer of a 3D "screen" object in my scene (not using a 
Raw Image
or Unity's canvas system at all). So I would recommend trying this when attempting to reproduce the issue. Please let me know what you find out on your Windows machine! If you still aren't able to reproduce the issue after testing on Windows via the method I described above, then I should create and send over a project with the reproducible issue for you to use in order to look into it further.
d
@important-megabyte-62876 I think Theo meant this line, where it's creating a texture.. the parameter indicates
Copy code
linear	Is texture using linear color space?
➕ 1
🙏 1
i
Hey @boundless-energy-78552, the example app also has the same "black screen" issue on Google Chrome. Here is a screenshot - Chrome is on the left side (black screen) and Firefox is on the right side (working):
I would recommend trying to reproduce on Windows 10, using Unity 2021.3.21f1
Please let me know if the same thing happens during your testing. If you are still unable to reproduce the issue on your end given those conditions, I could then create and send over a project with the reproducible issue for you. I'm hopeful we can get to the bottom of it together - this is currently the main issue barring us from using LiveKit since a vast majority of our users are on Chrome.
@boundless-energy-78552 I have just confirmed with another colleague that this issue is also reproducible on their system. They are running Windows 11 and get a black screen for all video feeds on Chrome, and it works on Firefox.
b
Perfect, just trying it right now
i
Created via Unity 2021.3.21f1
b
Ok just reproduced
with your link
👍 1
i
Excellent
b
Ok this is nice, I’m going to debug, thank you 🙂
So the issue seems to be only on Windows
i
Great! Please keep me updated about your findings. I am happy to help any way I can to get to the bottom of it!
🙌 1
❤️ 1
🙏 2
b
Hey, when building the project locally, I can’t reproduce the issue. This is weird.. But I think I know what happens, I created a new branch on the repo called 
theo/blackvideo
(link) It seems that the video does not start automatically. Can you try it?
i
I will try this out...
@boundless-energy-78552 to test I simply just need to modify the 
livekit-bridge.jslib
per your link and rebuild?
b
Yes you can do that
👍 1
I confirm that when I use your link and retrieve the corresponding HTMLVideoElement, just calling 
video.play()
fixes the issue
i
Hey @boundless-energy-78552, I'm happy to report that the code change to 
livekit-bridge.jslib
that you provided appears to fix the issue 🙂 Left side is Chrome, right side is Firefox - both working now:
🙌 1
This also got screen share to work in my test scene with the screen being a 3D model in world space.
And the video preview now starts right away in Chrome, whereas it did not prior to this change.
b
Ok nice, I’m going to merge it to main, did you get a chance to test the difference between color spaces when editing this line?
i
I have not tested that yet - I will try this out next!
👌 1
Thank you so much for being attentive to these issues - our team really appreciates it! 🙂 I will report back my findings once finished testing this out.
❤️ 1
🙏 1
Hey @boundless-energy-78552, I've gone ahead and changed the second boolean to false in line 80 of 
HTMLVideoElement.cs
. Unfortunately the result appears to be the exact same regardless of whether or not the boolean is set to true or false. Here's an image showing the "washed out" look without any changes:
And here is an image with the boolean changed to false:
I will direct message you a hosted app build that you can try out to observe this yourself...
Hey @boundless-energy-78552 I have just sent you a DM for a build that you can try out to see the issue. Here's the test pattern image I'm using to verify correct colors also:
Hey @boundless-energy-78552 just wanted to check in. Are you able to observe the same "washed out" look that we are seeing, via the build I sent you?
b
Hey @important-megabyte-62876 I haven’t looked yet, I’ll check that in a bit
i
Ok, thanks! 🙂
@boundless-energy-78552 Here is one more example of the washed-out look, with an actual video instead of a test pattern:
b
Oh I just answered you in DM 🙂
i
Oh okay, great!
b
So yes, I just reproduced the issue, I’ll fix that 😄
🙏 1
i
Thank you so much!
b
I have just discovered the cause of the issue, I will create a fix once I am on my Windows computer. Thank you very much for reporting your issues 🙏
🥇 1
🙌 1
i
I'm so happy to hear that!
@boundless-energy-78552 as long as you're updating things, looks like the Init() method in CopyPaste.cs might need to get wrapped into a compiler directive (or maybe even that whole script's code) so it doesn't run in the Unity Editor. I get this error when entering playmode without having any LiveKit logic in the scene:
b
Hey, I just fixed the color space on this PR. Can you give it a try? I’ll create another PR for the CopyPaste.cs 🙂
CopyPaste PR
i
Thank you @boundless-energy-78552! I will take a look as soon as I'm able today and let you know the result!
🙏 1
Hey @boundless-energy-78552, good news and bad news. The good news is that this does fix the problem with the "washed out" look - the colors look correct after applying your PR. The bad news is that for some reason this causes a massive drop in performance. On a high-performance Windows PC the framerate drops from a smooth 60 fps down to around 12 fps when I begin screen sharing. I'll send you another build that I just made via DM so that you can compare the updated build against the previous build I sent you in terms of performance.
I would unfortunately not recommend pushing this update to main until we've discovered the cause of the drop in performance due to the color space update. 😕
Actually, I have run into a very similar (perhaps identical) issue with a different plugin called Video Player WebGL. This plugin also has a "washed out" look to the video color - same as the issue with LiveKit Unity WebGL. Awhile back, I emailed the developer about it and they said to change a line in their code dealing with the color space from: 
GLctx.texImage2D(GLctx.TEXTURE_2D, 0, GLctx.RGBA, GLctx.RGBA, GLctx.UNSIGNED_BYTE, video);
to 
GLctx.texImage2D(GLctx.TEXTURE_2D, 0, GLctx.SRGB8_ALPHA8, GLctx.RGBA, GLctx.UNSIGNED_BYTE, video);
which also resulted in a large drop in performance. This appears to be the exact same change that you made in 
livekit-bridge.jslib
and resulting in the same exact performance issue. I don't believe a solve was ever figured out for the "Video Player WebGL" plugin and therefore the above change was never implemented in that plugin, but also the issue simply wasn't investigated further so I don't know the cause.
But it would appear that changing the color format from 
RGBA
to 
SRGB8_ALPHA8
in the function 
GLctx.SRGB8_ALPHA8
is somehow causing a very bad drop in frame rate of the app.
Here is an image where you can see the frames-per-second in the upper right corner. When I am screen sharing (top image), the color now looks correct but the FPS drops to 12 FPS. When I stop screen sharing (lower image) the FPS goes back up to 60 fps. This issue was not present prior to changing 
RGBA
to 
SRGB8_ALPHA8
:
d
that's a great find Matt! I think the issue might be that 
SRGB8_ALPHA8
isn't hardware accelerated. I wonder if there's a way to avoid that and stick to RGBA space to begin with?
b
I just revert the PR, yes it seems like Chrome is sending the RGBA buffer to the CPU to do the SRGB to RGB conversion and then reuploading it to the GPU. This is tricky, I’ll investigate further, thanks for the analysis
i
Thank you so much for investigating - please keep me updated and if there's anything you'd like me to test please let me know! 🙂
🙏 1
b
@important-megabyte-62876 Inside the BuildSettings, can you try these settings? Make sure to remove the patch I made for the colorspaces (main branch on gh atm)
Uh, webgl 1.0 is already removed on the screenshot. Also note that webgl 1.0 is deprecated, so I don’t think this is a big deal
i
These are my current settings that have been in builds thus far (they match your screenshot above):
b
oh you already had these settings?
i
Yes
b
mmh, can you try switching to gamma (just for me to understand how Unity works with linear textures)
i
Sure, you just need me to share a build with you that uses gamma color space?
b
Yes, it can be great 🙏 Or if ever, just check if the colors are now normal on the screenshares
i
Ok, working on it... The Unity project has to re-import all materials/shaders in order to change the color space and our project is rather large, so it's taking awhile. 😅 I cannot permanently change our project to gamma of course, because we've done our materials and lighting using the linear color space workflow (which I believe is the default and recommended option in Unity) but I can certainly temporarily change it for you, for test purposes.
🙏 1
@boundless-energy-78552 I've sent you another app build to try with the Gamma color space. With the color space set to Gamma in the Unity project settings, the color of the screen share video looks correct:
I've now reverted my own Unity project back to Linear color space because we need to use Linear (in general Gamma is now probably considered a "legacy" mode), but hopefully this helps!
If you need me to switch the color space mode again for test purposes, that should be more easy now. Unity appears to cache the imported shaders/materials, so there isn't another long import after the first time switching it.
Hey @boundless-energy-78552, I have discovered an in-Unity workaround to convert the texture to the correct colorspace via a shader:
This relies on Unity's Shadergraph system and the Colorspace Conversion Node.
b
Hey, did you use a math.Pow?
Yes, this is a perfectly valid solution, but it is still a bug in our SDK because we should follow the project settings.
I’ll still fix that in the future, glad that you found a workaround 🙌
i
Here is what my shadergraph looks like:
🎉 1
👌 1
But yes I agree, it should be adjusted correctly before it reaches Unity.
➕ 1
Because most users probably wouldn't know to do this and would wonder why it looks washed out.
But, I am glad to have a temporary workaround!
b
Yes, I’m thinking about adding this workaround on the README atm.
I’ll try to look further after publishing the NativeSDK 🙂
i
Ok, sounds good! I hope an "efficient" way can be found to convert the colorspace! I don't know the inner-workings but I feel like there has got to be some way to do it...
b
Yes, I’m thinking about creating two textures on the GPU. Using a Blit command should be enough and this is really fast
i
Ok, please let me know if you'd like me to test this out!! 🙂
❤️ 1
b
Thanks! 🙏
👍 1
i
@boundless-energy-78552 One more thing to put on your radar in regards to improving the quality of the video texture: Mipmaps. Mipmaps are very important when rendering video in 3D world space, otherwise the video texture tends to look very aliased when standing further away. I know that it is possible to do auto-generated mipmaps because the developer of the Video Player WebGL plugin I mentioned earlier did this to his plugin, which appears to rely on very similar technology as LiveKit's Unity WebGL SDK. We use mipmaps and trilinear filtering across the board for materials in our Unity WebGL project. Would it be possible for you to add mipmap capability to LiveKit's video texture in Unity? It would really help screenshare (or any virtual video screen powered by LiveKit) look nicer.
➕ 1
b
Oh yes, this is already planned for the Native SDK. (Along with adaptive stream where the server only send a lower resolution if the video is rendered far away)
i
Fantastic!
Open in Slack
PreviousNext
Powered by