https://livekit.io logo
Join Slack
Powered by
# helpdesk
  • w
    for the last 10 minutes maybe
  • w
    even the room receives messages when the ws initializes the connection
  • w
    Copy code
    {"level":"info","ts":1611709670.8599367,"caller":"service/rtcservice.go:92","msg":"new client connected","room":"RM_8eTmZn4xSVsWaKu8ByDC3B","roomName":"abc","name":"WzEsICJwdWJsaWMiLCAidXNlciIsICIweGM5ZTk0NjQ3NDg4NTJFQWZDMDlFOGM1MkEwZTA5MjQwQkFmYUMyNUMiXQ=="}
{"level":"info","ts":1611709708.037398,"caller":"service/rtcservice.go:110","msg":"WS connection closed","participant":"WzEsICJwdWJsaWMiLCAidXNlciIsICIweEZGQzgwYmQyQTQxM2YzN0UxMjVEMzlDMjgxQ2M4NUI4OGRjZWJGMjAiXQ=="}
{"level":"info","ts":1611709709.083549,"caller":"rtc/room.go:90","msg":"new participant joined","id":"PA_Zzwsc5e6FFib832G6JMgfj","name":"WzEsICJwdWJsaWMiLCAidXNlciIsICIweEZGQzgwYmQyQTQxM2YzN0UxMjVEMzlDMjgxQ2M4NUI4OGRjZWJGMjAiXQ==","roomId":"RM_pXmT5VY2q2vqChAQ3tDLym"}
{"level":"info","ts":1611709709.0880227,"caller":"service/rtcservice.go:92","msg":"new client connected","room":"RM_pXmT5VY2q2vqChAQ3tDLym","roomName":"8fb5bc1e-1918-4240-9e05-b1c01420152c","name":"WzEsICJwdWJsaWMiLCAidXNlciIsICIweEZGQzgwYmQyQTQxM2YzN0UxMjVEMzlDMjgxQ2M4NUI4OGRjZWJGMjAiXQ=="}
  • w
    restarting the server fixed the hangs
  • d
    do you mean 
    multi-node
    above? that there's a lack of response from server to clients?
  • w
    yes
  • d
    cool, that's helpful. likely a lock somewhere.. will look
  • w
    sounds good
  • w
    here's a local run
  • s
    This message was deleted.
    d
    w
    • 3
    • 3
  • w
    you know it does correlate with me publishing video
  • w
    thats odd, running locally it recovers from the panic
  • w
    Copy code
    livekit_server     | {"level":"info","ts":1611711120.595618,"caller":"rtc/mediatrack.go:275","msg":"keyframe required, sending PLI","srcParticipant":"PA_iUw8knfEK3H7jpaQRpRMnV"}
livekit_server     | {"level":"error","ts":1611711123.828131,"caller":"runtime/chan.go:275","msg":"recovered panic","error":"send on closed channel","stacktrace":"runtime.chansend\n\t/usr/local/go/src/runtime/chan.go:275\nruntime.chansend1\n\t/usr/local/go/src/runtime/chan.go:143\<http://ngithub.com/livekit/livekit-server/pkg/routing.(*MessageChannel).WriteMessage\n\t/workspace/pkg/routing/messagechannel.go:23\ngithub.com/livekit/livekit-server/pkg/rtc.(*ParticipantImpl).SendParticipantUpdate\n\t/workspace/pkg/rtc/participant.go:379\ngithub.com/livekit/livekit-server/pkg/rtc.(*Room).broadcastParticipantState\n\t/workspace/pkg/rtc/room.go:202\ngithub.com/livekit/livekit-server/pkg/rtc.(*Room).Join.func1\n\t/workspace/pkg/rtc/room.go:65\ngithub.com/livekit/livekit-server/pkg/rtc.(*ParticipantImpl).updateState.func1\n\t/workspace/pkg/rtc/participant.go:485%22}|ngithub.com/livekit/livekit-server/pkg/routing.(*MessageChannel).WriteMessage\n\t/workspace/pkg/routing/messagechannel.go:23\ngithub.com/livekit/livekit-server/pkg/rtc.(*ParticipantImpl).SendParticipantUpdate\n\t/workspace/pkg/rtc/participant.go:379\ngithub.com/livekit/livekit-server/pkg/rtc.(*Room).broadcastParticipantState\n\t/workspace/pkg/rtc/room.go:202\ngithub.com/livekit/livekit-server/pkg/rtc.(*Room).Join.func1\n\t/workspace/pkg/rtc/room.go:65\ngithub.com/livekit/livekit-server/pkg/rtc.(*ParticipantImpl).updateState.func1\n\t/workspace/pkg/rtc/participant.go:485"}>
  • d
    yeah some write after closures are recovered, others aren't. pushing a fix for this shortly
  • d
    fix pushed
  • d
    I'm able to repro the multi-node issues too, some timing issue
  • w
    sweet I'm testing your latest push
  • w
    still seeing similar issues, were the stacktraces helpful?
  • d
    yes, they were.. could you share the latest if possible?
  • w
    yes one moment
  • w
    Copy code
    livekit_server     | {"level":"info","ts":1611712079.806331,"caller":"rtc/room.go:90","msg":"new participant joined","id":"PA_QVv7SjVzR6rCDVgM8Eqyi4","name":"WzEsICJwdWJsaWMiLCAidXNlciIsICIweEZGQzgwYmQyQTQxM2YzN0UxMjVEMzlDMjgxQ2M4NUI4OGRjZWJGMjAiXQ==","roomId":"RM_VGcKnqAtvPoq5bYNKuNKkW"}
livekit_server     | {"level":"info","ts":1611712079.807056,"caller":"service/rtcservice.go:92","msg":"new client connected","room":"RM_VGcKnqAtvPoq5bYNKuNKkW","roomName":"8fb5bc1e-1918-4240-9e05-b1c01420152c","name":"WzEsICJwdWJsaWMiLCAidXNlciIsICIweEZGQzgwYmQyQTQxM2YzN0UxMjVEMzlDMjgxQ2M4NUI4OGRjZWJGMjAiXQ=="}
livekit_server     | {"level":"info","ts":1611712135.2656472,"caller":"rtc/room.go:90","msg":"new participant joined","id":"PA_T7AJ2HbPqYEZwd7iFf5hqL","name":"WzEsICJwdWJsaWMiLCAidXNlciIsICIweGM5ZTk0NjQ3NDg4NTJFQWZDMDlFOGM1MkEwZTA5MjQwQkFmYUMyNUMiXQ==","roomId":"RM_VGcKnqAtvPoq5bYNKuNKkW"}
livekit_server     | {"level":"info","ts":1611712135.266127,"caller":"service/rtcservice.go:92","msg":"new client connected","room":"RM_VGcKnqAtvPoq5bYNKuNKkW","roomName":"8fb5bc1e-1918-4240-9e05-b1c01420152c","name":"WzEsICJwdWJsaWMiLCAidXNlciIsICIweGM5ZTk0NjQ3NDg4NTJFQWZDMDlFOGM1MkEwZTA5MjQwQkFmYUMyNUMiXQ=="}
livekit_server     | {"level":"info","ts":1611712154.951387,"caller":"service/rtcservice.go:110","msg":"WS connection closed","participant":"WzEsICJwdWJsaWMiLCAidXNlciIsICIweEZGQzgwYmQyQTQxM2YzN0UxMjVEMzlDMjgxQ2M4NUI4OGRjZWJGMjAiXQ=="}
livekit_server     | {"level":"info","ts":1611712155.471571,"caller":"service/rtcservice.go:110","msg":"WS connection closed","participant":"WzEsICJwdWJsaWMiLCAidXNlciIsICIweGM5ZTk0NjQ3NDg4NTJFQWZDMDlFOGM1MkEwZTA5MjQwQkFmYUMyNUMiXQ=="}
livekit_server     | {"level":"info","ts":1611712157.26515,"caller":"rtc/room.go:90","msg":"new participant joined","id":"PA_2qLdsX9crmccYbA7c2TKAW","name":"WzEsICJwdWJsaWMiLCAidXNlciIsICIweEZGQzgwYmQyQTQxM2YzN0UxMjVEMzlDMjgxQ2M4NUI4OGRjZWJGMjAiXQ==","roomId":"RM_VGcKnqAtvPoq5bYNKuNKkW"}
livekit_server     | {"level":"info","ts":1611712157.2654898,"caller":"service/rtcservice.go:92","msg":"new client connected","room":"RM_VGcKnqAtvPoq5bYNKuNKkW","roomName":"8fb5bc1e-1918-4240-9e05-b1c01420152c","name":"WzEsICJwdWJsaWMiLCAidXNlciIsICIweEZGQzgwYmQyQTQxM2YzN0UxMjVEMzlDMjgxQ2M4NUI4OGRjZWJGMjAiXQ=="}
livekit_server     | {"level":"info","ts":1611712157.5914612,"caller":"rtc/room.go:90","msg":"new participant joined","id":"PA_wMyJ27GFx5cPib93P7KaRg","name":"WzEsICJwdWJsaWMiLCAidXNlciIsICIweGM5ZTk0NjQ3NDg4NTJFQWZDMDlFOGM1MkEwZTA5MjQwQkFmYUMyNUMiXQ==","roomId":"RM_VGcKnqAtvPoq5bYNKuNKkW"}
livekit_server     | {"level":"info","ts":1611712157.5917299,"caller":"service/rtcservice.go:92","msg":"new client connected","room":"RM_VGcKnqAtvPoq5bYNKuNKkW","roomName":"8fb5bc1e-1918-4240-9e05-b1c01420152c","name":"WzEsICJwdWJsaWMiLCAidXNlciIsICIweGM5ZTk0NjQ3NDg4NTJFQWZDMDlFOGM1MkEwZTA5MjQwQkFmYUMyNUMiXQ=="}
livekit_server     | {"level":"error","ts":1611712160.096131,"caller":"runtime/chan.go:275","msg":"recovered panic","error":"send on closed channel","stacktrace":"runtime.chansend\n\t/usr/local/go/src/runtime/chan.go:275\nruntime.chansend1\n\t/usr/local/go/src/runtime/chan.go:143\<http://ngithub.com/livekit/livekit-server/pkg/routing.(*MessageChannel).WriteMessage\n\t/workspace/pkg/routing/messagechannel.go:28\ngithub.com/livekit/livekit-server/pkg/rtc.(*ParticipantImpl).SendParticipantUpdate\n\t/workspace/pkg/rtc/participant.go:379\ngithub.com/livekit/livekit-server/pkg/rtc.(*Room).broadcastParticipantState\n\t/workspace/pkg/rtc/room.go:202\ngithub.com/livekit/livekit-server/pkg/rtc.(*Room).Join.func1\n\t/workspace/pkg/rtc/room.go:65\ngithub.com/livekit/livekit-server/pkg/rtc.(*ParticipantImpl).updateState.func1\n\t/workspace/pkg/rtc/participant.go:489%22}|ngithub.com/livekit/livekit-server/pkg/routing.(*MessageChannel).WriteMessage\n\t/workspace/pkg/routing/messagechannel.go:28\ngithub.com/livekit/livekit-server/pkg/rtc.(*ParticipantImpl).SendParticipantUpdate\n\t/workspace/pkg/rtc/participant.go:379\ngithub.com/livekit/livekit-server/pkg/rtc.(*Room).broadcastParticipantState\n\t/workspace/pkg/rtc/room.go:202\ngithub.com/livekit/livekit-server/pkg/rtc.(*Room).Join.func1\n\t/workspace/pkg/rtc/room.go:65\ngithub.com/livekit/livekit-server/pkg/rtc.(*ParticipantImpl).updateState.func1\n\t/workspace/pkg/rtc/participant.go:489"}>
livekit_server     | {"level":"error","ts":1611712160.0963788,"caller":"runtime/chan.go:275","msg":"recovered panic","error":"send on closed channel","stacktrace":"runtime.chansend\n\t/usr/local/go/src/runtime/chan.go:275\nruntime.chansend1\n\t/usr/local/go/src/runtime/chan.go:143\<http://ngithub.com/livekit/livekit-server/pkg/routing.(*MessageChannel).WriteMessage\n\t/workspace/pkg/routing/messagechannel.go:28\ngithub.com/livekit/livekit-server/pkg/rtc.(*ParticipantImpl).SendParticipantUpdate\n\t/workspace/pkg/rtc/participant.go:379\ngithub.com/livekit/livekit-server/pkg/rtc.(*Room).broadcastParticipantState\n\t/workspace/pkg/rtc/room.go:202\ngithub.com/livekit/livekit-server/pkg/rtc.(*Room).Join.func1\n\t/workspace/pkg/rtc/room.go:65\ngithub.com/livekit/livekit-server/pkg/rtc.(*ParticipantImpl).updateState.func1\n\t/workspace/pkg/rtc/participant.go:489%22}|ngithub.com/livekit/livekit-server/pkg/routing.(*MessageChannel).WriteMessage\n\t/workspace/pkg/routing/messagechannel.go:28\ngithub.com/livekit/livekit-server/pkg/rtc.(*ParticipantImpl).SendParticipantUpdate\n\t/workspace/pkg/rtc/participant.go:379\ngithub.com/livekit/livekit-server/pkg/rtc.(*Room).broadcastParticipantState\n\t/workspace/pkg/rtc/room.go:202\ngithub.com/livekit/livekit-server/pkg/rtc.(*Room).Join.func1\n\t/workspace/pkg/rtc/room.go:65\ngithub.com/livekit/livekit-server/pkg/rtc.(*ParticipantImpl).updateState.func1\n\t/workspace/pkg/rtc/participant.go:489"}>
livekit_server     | {"level":"info","ts":1611712163.90857,"caller":"service/rtcservice.go:110","msg":"WS connection closed","participant":"WzEsICJwdWJsaWMiLCAidXNlciIsICIweEZGQzgwYmQyQTQxM2YzN0UxMjVEMzlDMjgxQ2M4NUI4OGRjZWJGMjAiXQ=="}
livekit_server     | {"level":"info","ts":1611712165.709913,"caller":"rtc/room.go:90","msg":"new participant joined","id":"PA_czXeytWkbCtWM22p8N3Nga","name":"WzEsICJwdWJsaWMiLCAidXNlciIsICIweEZGQzgwYmQyQTQxM2YzN0UxMjVEMzlDMjgxQ2M4NUI4OGRjZWJGMjAiXQ==","roomId":"RM_VGcKnqAtvPoq5bYNKuNKkW"}
livekit_server     | {"level":"info","ts":1611712165.710581,"caller":"service/rtcservice.go:92","msg":"new client connected","room":"RM_VGcKnqAtvPoq5bYNKuNKkW","roomName":"8fb5bc1e-1918-4240-9e05-b1c01420152c","name":"WzEsICJwdWJsaWMiLCAidXNlciIsICIweEZGQzgwYmQyQTQxM2YzN0UxMjVEMzlDMjgxQ2M4NUI4OGRjZWJGMjAiXQ=="}
livekit_server     | {"level":"info","ts":1611712171.636658,"caller":"service/rtcservice.go:110","msg":"WS connection closed","participant":"WzEsICJwdWJsaWMiLCAidXNlciIsICIweGM5ZTk0NjQ3NDg4NTJFQWZDMDlFOGM1MkEwZTA5MjQwQkFmYUMyNUMiXQ=="}
livekit_server     | {"level":"info","ts":1611712226.8868108,"caller":"service/rtcservice.go:110","msg":"WS connection closed","participant":"WzEsICJwdWJsaWMiLCAidXNlciIsICIweEZGQzgwYmQyQTQxM2YzN0UxMjVEMzlDMjgxQ2M4NUI4OGRjZWJGMjAiXQ=="}
livekit_server     | panic: send on closed channel
livekit_server     |
livekit_server     | goroutine 385 [running]:
livekit_server     | <http://github.com/livekit/livekit-server/pkg/rtc.NewReceiver.func1(0xc0007fc1d0|github.com/livekit/livekit-server/pkg/rtc.NewReceiver.func1(0xc0007fc1d0>, 0x1, 0x1)
livekit_server     |    /workspace/pkg/rtc/receiver.go:38 +0x67
livekit_server     | <http://github.com/pion/ion-sfu/pkg/buffer.(*Buffer).calc(0xc000558000|github.com/pion/ion-sfu/pkg/buffer.(*Buffer).calc(0xc000558000>, 0xc00051c000, 0x5e, 0x2000, 0x165df3b456348920)
livekit_server     |    /go/pkg/mod/github.com/pion/ion-sfu@v1.7.8/pkg/buffer/buffer.go:288 +0x2a7
livekit_server     | <http://github.com/pion/ion-sfu/pkg/buffer.(*Buffer).Write(0xc000558000|github.com/pion/ion-sfu/pkg/buffer.(*Buffer).Write(0xc000558000>, 0xc00051c000, 0x5e, 0x2000, 0x0, 0x0, 0x0)
livekit_server     |    /go/pkg/mod/github.com/pion/ion-sfu@v1.7.8/pkg/buffer/buffer.go:192 +0x11d
livekit_server     | <http://github.com/pion/srtp/v2.(*ReadStreamSRTP).write(0xc000078040|github.com/pion/srtp/v2.(*ReadStreamSRTP).write(0xc000078040>, 0xc00051c000, 0x5e, 0x2000, 0xc00051c000, 0x68, 0x2000)
livekit_server     |    /go/pkg/mod/github.com/pion/srtp/v2@v2.0.1/stream_srtp.go:64 +0x56
livekit_server     | <http://github.com/pion/srtp/v2.(*SessionSRTP).decrypt(0xc0001a7b80|github.com/pion/srtp/v2.(*SessionSRTP).decrypt(0xc0001a7b80>, 0xc00051c000, 0x68, 0x2000, 0x68, 0x0)
livekit_server     |    /go/pkg/mod/github.com/pion/srtp/v2@v2.0.1/session_srtp.go:165 +0x1a8
livekit_server     | <http://github.com/pion/srtp/v2.(*session).start.func1(0xc0001a7b80|github.com/pion/srtp/v2.(*session).start.func1(0xc0001a7b80>, 0xc0000677c0, 0xe83f40, 0xc0001a7b80)
livekit_server     |    /go/pkg/mod/github.com/pion/srtp/v2@v2.0.1/session.go:141 +0x12d
livekit_server     | created by <http://github.com/pion/srtp/v2.(*session).start|github.com/pion/srtp/v2.(*session).start>
livekit_server     |    /go/pkg/mod/github.com/pion/srtp/v2@v2.0.1/session.go:120 +0x226
  • d
    thanks! this helps, a diff panic šŸ™‚
  • w
    sure thing
  • c
    btw @wonderful-manchester-12533 mentioned that the server panics pretty regularly. for me (i think) it happens with new people joining the room
  • d
    yeah. some race conditions. aiming to fix them tonight
  • c
    sweet lmk šŸ™‚
  • d
    ok, this should be improved a bit. pushed v0.2.3 that fixes multi-node routing, as well as gets rid of a few of the panics when clients connect/exit
  • c
    nice
  • c
    iā€™l check it out shortly
12345...91Latest