This message was deleted.

Hey there! We do have a Unity SDK, but I'm not sure if it would work with Hololens at the moment. We do want to make it support Hololens at some point though. We do not use Unity WebRTC, but instead build against libwebrtc directly.

Hi David. I need to know how to send the

AddTrackRequest

for audio and video to the Livekit signaler correctly. My unity project has implemented webtick (https://github.com/chef-studios-inc/webtick-unity) for establishing the WebRTC connection, but it only supports Livekit datachannels. I tried to request sending to the signaler websocket these messages:

public async Task SendVideoTrackRequest() {
            var req = new LiveKit.Proto.AddTrackRequest
            {
                Source = LiveKit.Proto.TrackSource.Camera,
                Type = LiveKit.Proto.TrackType.Video
            };

            await websocket.Send(req.ToByteArray());
}

public async Task SendAudioRequestTrack() {
            var req = new LiveKit.Proto.AddTrackRequest
            {
                Source = LiveKit.Proto.TrackSource.Microphone,
                Type = LiveKit.Proto.TrackType.Audio,
            };

            await websocket.Send(req.ToByteArray());
}

But when it sends them, the web socket closes the connection and prints a

On Close: Abnormal

ah I see. this is @powerful-garage-56223’s library.. Any advise here Neil?

hey @adventurous-family-21979 - so yes this webtick repo was adding livekit as a transport for DOTS netcode. But as you have found, it makes a connection to livekit and established peer connections for data channels. It’s difficult to say exactly what’s going wrong here. I’ve implemented this in the past in an old repo. Here’s what I’ve got in an RTCEngine.cs. Should get you going in the right direction but this code is over 2 years old

public async Task<string> PublishVideoTrack(VideoStreamTrack track, int bitrate)
        {
            // Make sure we're in a good ICEConnectionState before we add the track
            while (localPeerConnection.IceConnectionState != RTCIceConnectionState.Connected && localPeerConnection.IceConnectionState != RTCIceConnectionState.Completed)
            {
                await Task.Delay(100);
            }

            Debug.LogFormat("publisher - publishing track with client id: {0}", track.Id);

            // Create the request for the server
            var req = new LivekitProto.AddTrackRequest();
            req.Type = LivekitProto.TrackType.Video;
            req.Name = "";
            req.Cid = track.Id;
            signalClient.SendAddTrack(req);

            //TODO: implement some timeout error handling
            // var startTime = Time.realtimeSinceStartup;

            Debug.LogFormat("publisher - waiting for publish response for client_id: {0}", track.Id);
            while (!clientIDserverIDWaitingMap.ContainsKey(track.Id))
            {
                await Task.Delay(50);
            }

            var serverID = clientIDserverIDWaitingMap[track.Id];
            clientIDserverIDWaitingMap.Remove(track.Id);
            Debug.LogFormat("publisher - got for publish response for client_id: {0}, server_id: {1}", track.Id, serverID);

            //Debug.LogFormat("publisher - waiting for encoder to be initialized");
            //while (!track.IsEncoderInitialized)
            //{
            //    await Task.Delay(50);
            //}
            //Debug.LogFormat("publisher - encoder is initialized");

            var transceiver = localPeerConnection.AddTransceiver(track);
            transceiver.Direction = RTCRtpTransceiverDirection.SendOnly;

            while(true)
            {
                var parameters = transceiver.Sender.GetParameters(); 
                if(parameters.encodings.Length > 0)
                {
                    parameters.encodings[0].maxBitrate = (ulong?)bitrate;
                    parameters.encodings[0].minBitrate = (ulong?)bitrate;
                    parameters.encodings[0].maxFramerate = 30;
                    transceiver.Sender.SetParameters(parameters);
                    break;
                }
                Debug.Log("Waiting for encodings");
                await Task.Delay(100);
            }

            return serverID;
        }

Hi, thanks for this snippet. I tested it in my Unity Project with a self-hosted Livekit Server and I got this error:

2023-09-08T09:30:19.184+0200    ERROR   livekit service/rtcservice.go:376       error reading from websocket    {"room": "my-first-room", "roomID": "RM_djMqZsfs8LUu", "participant": "user1", "pID": "PA_9Ud3PLM7LyzL", "remote": false, "connID": "CO_WWvyxUHBxFju", "error": "proto: cannot parse invalid wire-format data"}
<http://github.com/livekit/livekit-server/pkg/service.(*RTCService).ServeHTTP|github.com/livekit/livekit-server/pkg/service.(*RTCService).ServeHTTP>
        /home/runner/go/pkg/mod/github.com/livekit/livekit-server@v1.4.5/pkg/service/rtcservice.go:376
net/http.(*ServeMux).ServeHTTP
        /opt/hostedtoolcache/go/1.21.0/x64/src/net/http/server.go:2514
<http://github.com/urfave/negroni/v3.(*Negroni).UseHandler.Wrap.func1|github.com/urfave/negroni/v3.(*Negroni).UseHandler.Wrap.func1>
        /home/runner/go/pkg/mod/github.com/urfave/negroni/v3@v3.0.0/negroni.go:59
<http://github.com/urfave/negroni/v3.HandlerFunc.ServeHTTP|github.com/urfave/negroni/v3.HandlerFunc.ServeHTTP>
        /home/runner/go/pkg/mod/github.com/urfave/negroni/v3@v3.0.0/negroni.go:33
<http://github.com/urfave/negroni/v3.middleware.ServeHTTP|github.com/urfave/negroni/v3.middleware.ServeHTTP>
        /home/runner/go/pkg/mod/github.com/urfave/negroni/v3@v3.0.0/negroni.go:51
net/http.HandlerFunc.ServeHTTP
        /opt/hostedtoolcache/go/1.21.0/x64/src/net/http/server.go:2136
<http://github.com/livekit/livekit-server/pkg/service.(*APIKeyAuthMiddleware).ServeHTTP|github.com/livekit/livekit-server/pkg/service.(*APIKeyAuthMiddleware).ServeHTTP>
        /home/runner/go/pkg/mod/github.com/livekit/livekit-server@v1.4.5/pkg/service/auth.go:98
<http://github.com/urfave/negroni/v3.middleware.ServeHTTP|github.com/urfave/negroni/v3.middleware.ServeHTTP>
        /home/runner/go/pkg/mod/github.com/urfave/negroni/v3@v3.0.0/negroni.go:51
<http://github.com/rs/cors.(*Cors).ServeHTTP|github.com/rs/cors.(*Cors).ServeHTTP>
        /home/runner/go/pkg/mod/github.com/rs/cors@v1.9.0/cors.go:271
<http://github.com/urfave/negroni/v3.middleware.ServeHTTP|github.com/urfave/negroni/v3.middleware.ServeHTTP>
        /home/runner/go/pkg/mod/github.com/urfave/negroni/v3@v3.0.0/negroni.go:51
<http://github.com/urfave/negroni/v3.(*Recovery).ServeHTTP|github.com/urfave/negroni/v3.(*Recovery).ServeHTTP>
        /home/runner/go/pkg/mod/github.com/urfave/negroni/v3@v3.0.0/recovery.go:210
<http://github.com/urfave/negroni/v3.middleware.ServeHTTP|github.com/urfave/negroni/v3.middleware.ServeHTTP>
        /home/runner/go/pkg/mod/github.com/urfave/negroni/v3@v3.0.0/negroni.go:51
<http://github.com/urfave/negroni/v3.(*Negroni).ServeHTTP|github.com/urfave/negroni/v3.(*Negroni).ServeHTTP>
        /home/runner/go/pkg/mod/github.com/urfave/negroni/v3@v3.0.0/negroni.go:111
net/http.serverHandler.ServeHTTP
        /opt/hostedtoolcache/go/1.21.0/x64/src/net/http/server.go:2938
net/http.(*conn).serve
        /opt/hostedtoolcache/go/1.21.0/x64/src/net/http/server.go:2009
2023-09-08T09:30:19.184+0200    INFO    livekit service/rtcservice.go:261       finishing WS connection {"room": "my-first-room", "roomID": "RM_djMqZsfs8LUu", "participant": "user1", "pID": "PA_9Ud3PLM7LyzL", "remote": false, "connID": "CO_WWvyxUHBxFju"}

Even if I update Livekit.Proto classes I get this error from Livekit server

Well we’re getting somewhere! To confirm, everything connects fine and you just get this when you try adding the video track?

I try to add video track after ice state changes to Completed/Connected

Ok well at least we know at least some proto messages are working for it to get to that state. Can you show how you’re sending the payload?

I send the payload using

websocket.Send(req.ToByteArray())

. SendVideoTrackRequest is SignalerClient method

public async Task SendVideoTrackRequest(VideoStreamTrack track)
{
    var req = new LiveKit.Proto.AddTrackRequest
    {
        Source = LiveKit.Proto.TrackSource.Camera,
        Type = LiveKit.Proto.TrackType.Video,
        Name = "",
        Cid = track.Id,
    };

    await websocket.Send(req.ToByteArray());
}

hrmm that looks reasonable to me. If I get some free cycles I can look at this but otherwise I recommend looking at the javascript client sdk and seeing how that works

I resolved the problem by sending LiveKit.Proto.SignalRequest containing the request:

public async Task SendVideoTrackRequest(VideoStreamTrack track)
        {
            var trackReq = new LiveKit.Proto.AddTrackRequest
            {
                Source = LiveKit.Proto.TrackSource.Camera,
                Type = LiveKit.Proto.TrackType.Video,
                Name = "",
                Width = (uint)WebRTCSettings.StreamSize.x,
                Height = (uint)WebRTCSettings.StreamSize.y,
                Cid = track.Id,
            };


            await SendAddRequestTrack(trackReq);

        }

        public async Task SendAddRequestTrack(LiveKit.Proto.AddTrackRequest request)
        {
            var req = new LiveKit.Proto.SignalRequest();
            req.AddTrack = request;

            await websocket.Send(req.ToByteArray());
        }

I received the

TrackPublishedResponse

correct one but when I sent the offer in Unity the received track was stopped and after 20/30 seconds the websocket was closed anomaly. I tried on Livekit Cloud and sometimes the Unity peer tracks are published but after 30 seconds Unity streams stopped and its peer left the room.