https://livekit.io logo
Join Slack
Powered by
This message was deleted.
# helpdesk
s
This message was deleted.
d
The request seems fine.. what issues are you having?
n
hey thanks a lot for the reply, the code doesnt give any error but the segemented files are not being saved. On the other hand I am receiving all the webhooks even the egress_strated webhook. But when i use roomCompositeEgress instead of trackCompositeEgress then my files are being saved on my GCS. Any help or any hint to where to lookout to solve this will be really appreciated.
d
Are you using Cloud? or self hosting? if the latter, you'd want to take a look at the logs for clues
n
its self hosted, and i will take some more tries to get it fixed and if not fixed I will come to you with logs. But still thanks a lot for such a quick response.
hey @dry-elephant-14928 one more thing please is it ok to receive "participant_joined" webhook after receiving "egress_started" webhook ?
here are the logs for roomComposite and trackComposite with same configs but trackComposite sessms to have some problem
roomComposit_successfull.txttrackComposite_unsuccessfull.txt
Copy code
2023-09-06T07:27:56.625Z        INFO    egress  redis/redis.go:113      connecting to redis     {"nodeID": "NE_BmsUMFynL58w", "clusterID": "", "simple": true, "addr": "10.174.73.179:6379"}
2023-09-06T07:27:56.627Z        INFO    egress  stats/monitor.go:149    cpu available: 4.000000 max cost: 4.000000      {"nodeID": "NE_BmsUMFynL58w", "clusterID": ""}
2023-09-06T07:30:37.656Z        INFO    egress  service/service.go:115  request received        {"nodeID": "NE_BmsUMFynL58w", "clusterID": "", "egressID": "EG_NKdCBdmtVoee"}
2023-09-06T07:30:37.657Z        INFO    egress  service/service.go:124  request validated       {"nodeID": "NE_BmsUMFynL58w", "clusterID": "", "egressID": "EG_NKdCBdmtVoee", "requestType": "track_composite", "outputType": "file, stream, segments", "room": "E267_room", "request": {"TrackComposite":{"room_name":"E267_room","audio_track_id":"TR_AMBkKsDK8H22mg","video_track_id":"TR_VCLeLKBs2dDtfo","Output":null,"Options":null,"file_outputs":[{"file_type":1,"filepath":"E267_file/E267.mp4","Output":{"Gcp":{"credentials":"{credentials}","bucket":"flutin-hls-gcs-test"}}}],"stream_outputs":[{"protocol":1,"urls":["<rtmp://x.rtmp.youtube.com/live2/{9td...068}>"]}],"segment_outputs":[{"protocol":1,"filename_prefix":"E267_hls/segement","playlist_name":"playlist.m3u8","segment_duration":6,"Output":{"Gcp":{"credentials":"{credentials}","bucket":"flutin-hls-gcs-test"}}}]}}}
2023-09-06T07:30:37.678Z        INFO    egress  redis/redis.go:113      connecting to redis     {"nodeID": "NE_BmsUMFynL58w", "handlerID": "EGH_VgT3BqjbScde", "clusterID": "", "egressID": "EG_NKdCBdmtVoee", "simple": true, "addr": "10.174.73.179:6379"}
0:00:00.001907496    23 0x7fbeac019e90 ERROR     GST_PLUGIN_LOADING gstpluginloader.c:442:gst_plugin_loader_try_helper: Spawning gst-plugin-scanner helper failed: Failed to close file descriptor for child process (Operation not permitted)

(egress:23): GStreamer-WARNING **: 07:30:37.685: External plugin loader failed. This most likely means that the plugin loader helper binary was not found or could not be run. You might need to set the GST_PLUGIN_SCANNER environment variable if your setup is unusual. This should normally not be required though.
2023-09-06T07:30:37.757Z        INFO    logger/logger.go:152    received offer for subscriber   {"nodeID": "NE_BmsUMFynL58w", "handlerID": "EGH_VgT3BqjbScde", "clusterID": "", "egressID": "EG_NKdCBdmtVoee"}
2023-09-06T07:30:37.765Z        INFO    logger/logger.go:152    ICE connected   {"nodeID": "NE_BmsUMFynL58w", "handlerID": "EGH_VgT3BqjbScde", "clusterID": "", "egressID": "EG_NKdCBdmtVoee"}
0:00:00.214142308    23 0x7fbeac019e90 WARN                 default gstsfelement.c:97:gst_sf_create_audio_template_caps: format 0x120000: 'AVR (Audio Visual Research)' is not mapped
0:00:00.214166770    23 0x7fbeac019e90 WARN                 default gstsfelement.c:97:gst_sf_create_audio_template_caps: format 0x180000: 'CAF (Apple Core Audio File)' is not mapped
0:00:00.214174979    23 0x7fbeac019e90 WARN                 default gstsfelement.c:97:gst_sf_create_audio_template_caps: format 0x100000: 'HTK (HMM Tool Kit)' is not mapped
0:00:00.214179771    23 0x7fbeac019e90 WARN                 default gstsfelement.c:97:gst_sf_create_audio_template_caps: format 0xc0000: 'MAT4 (GNU Octave 2.0 / Matlab 4.2)' is not mapped
0:00:00.214183684    23 0x7fbeac019e90 WARN                 default gstsfelement.c:97:gst_sf_create_audio_template_caps: format 0xd0000: 'MAT5 (GNU Octave 2.1 / Matlab 5.0)' is not mapped
0:00:00.214188111    23 0x7fbeac019e90 WARN                 default gstsfelement.c:97:gst_sf_create_audio_template_caps: format 0x210000: 'MPC (Akai MPC 2k)' is not mapped
0:00:00.214193862    23 0x7fbeac019e90 WARN                 default gstsfelement.c:97:gst_sf_create_audio_template_caps: format 0xe0000: 'PVF (Portable Voice Format)' is not mapped
0:00:00.214199189    23 0x7fbeac019e90 WARN                 default gstsfelement.c:97:gst_sf_create_audio_template_caps: format 0x160000: 'SD2 (Sound Designer II)' is not mapped
0:00:00.214206562    23 0x7fbeac019e90 WARN                 default gstsfelement.c:97:gst_sf_create_audio_template_caps: format 0x190000: 'WVE (Psion Series 3)' is not mapped
2023-09-06T07:30:37.921Z        INFO    logger/logger.go:152    received offer for subscriber   {"nodeID": "NE_BmsUMFynL58w", "handlerID": "EGH_VgT3BqjbScde", "clusterID": "", "egressID": "EG_NKdCBdmtVoee"}
2023-09-06T07:30:37.938Z        INFO    logger/logger.go:152    track subscribed        {"nodeID": "NE_BmsUMFynL58w", "handlerID": "EGH_VgT3BqjbScde", "clusterID": "", "egressID": "EG_NKdCBdmtVoee", "participant": "U267_E267_host_rtmp", "track": "TR_AMBkKsDK8H22mg", "kind": "audio"}
2023-09-06T07:30:38.036Z        INFO    logger/logger.go:152    track subscribed        {"nodeID": "NE_BmsUMFynL58w", "handlerID": "EGH_VgT3BqjbScde", "clusterID": "", "egressID": "EG_NKdCBdmtVoee", "participant": "U267_E267_host_rtmp", "track": "TR_VCLeLKBs2dDtfo", "kind": "video"}
0:00:00.379520510    23 0x7fbeac019e90 WARN              cudaloader gstcudaloader.c:169:gst_cuda_load_library: Could not open library libcuda.so.1, libcuda.so.1: cannot open shared object file: No such file or directory
0:00:00.379546727    23 0x7fbeac019e90 WARN                 nvcodec plugin.c:94:plugin_init: Failed to load cuda library
2023-09-06T07:30:38.122Z        INFO    egress  pipeline/watch.go:223   videoAppSrc playing     {"nodeID": "NE_BmsUMFynL58w", "handlerID": "EGH_VgT3BqjbScde", "clusterID": "", "egressID": "EG_NKdCBdmtVoee"}
2023-09-06T07:30:38.122Z        INFO    egress  pipeline/watch.go:223   audioAppSrc playing     {"nodeID": "NE_BmsUMFynL58w", "handlerID": "EGH_VgT3BqjbScde", "clusterID": "", "egressID": "EG_NKdCBdmtVoee"}
2023-09-06T07:32:55.721Z        INFO    logger/logger.go:152    received offer for subscriber   {"nodeID": "NE_BmsUMFynL58w", "handlerID": "EGH_VgT3BqjbScde", "clusterID": "", "egressID": "EG_NKdCBdmtVoee"}
2023-09-06T07:32:55.721Z        INFO    egress  sdk/appwriter.go:180    writer finished {"nodeID": "NE_BmsUMFynL58w", "handlerID": "EGH_VgT3BqjbScde", "clusterID": "", "egressID": "EG_NKdCBdmtVoee", "trackID": "TR_VCLeLKBs2dDtfo", "kind": "video", "sampleDuration": "33.333333ms", "avgDrift": "0s", "maxDrift": "0s", "packetLoss": "0.00%"}
2023-09-06T07:32:55.721Z        INFO    egress  sdk/appwriter.go:180    writer finished {"nodeID": "NE_BmsUMFynL58w", "handlerID": "EGH_VgT3BqjbScde", "clusterID": "", "egressID": "EG_NKdCBdmtVoee", "trackID": "TR_AMBkKsDK8H22mg", "kind": "audio", "sampleDuration": "22.5ms", "avgDrift": "0s", "maxDrift": "0s", "packetLoss": "0.00%"}

(egress:23): GStreamer-CRITICAL **: 07:32:55.722: gst_mini_object_ref: assertion 'mini_object != NULL' failed

(egress:23): GStreamer-CRITICAL **: 07:32:55.722: gst_mini_object_ref: assertion 'mini_object != NULL' failed
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x48 pc=0x1305c3a]

goroutine 17 [running, locked to thread]:
<http://github.com/tinyzimmer/go-gst/gst.(*Buffer).PresentationTimestamp(...)|github.com/tinyzimmer/go-gst/gst.(*Buffer).PresentationTimestamp(...)>
        /root/go/pkg/mod/github.com/livekit/go-gst@v0.2.34-0.20230623013326-3ca8cb8f62e5/gst/gst_buffer.go:180
<http://github.com/livekit/egress/pkg/pipeline/output.(*Bin).buildSegmentOutput.func1(0x2|github.com/livekit/egress/pkg/pipeline/output.(*Bin).buildSegmentOutput.func1(0x2>?, 0xc00013a480?, 0xc00013a460)
        /workspace/pkg/pipeline/output/segments.go:63 +0x9a
reflect.Value.call({0x1728200?, 0xc00036d600?, 0x1f416a0?}, {0x19f2309, 0x4}, {0xc000481e00, 0x3, 0x7fbead165e30?})
        /usr/lib/go-1.18/src/reflect/value.go:556 +0x845
reflect.Value.Call({0x1728200?, 0xc00036d600?, 0x7fbead0f8ec0?}, {0xc000481e00, 0x3, 0x3})
        /usr/lib/go-1.18/src/reflect/value.go:339 +0xbf
<http://github.com/tinyzimmer/go-glib/glib.goMarshal(0x0|github.com/tinyzimmer/go-glib/glib.goMarshal(0x0>?, 0x7fbe67ffdea0, 0x3, 0x7fbe67ffdd70, 0x0?, 0xc000002601?)
        /root/go/pkg/mod/github.com/livekit/go-glib@v0.0.0-20230223001336-834490045522/glib/glib.go:271 +0x6f3
2023-09-06T07:32:55.733Z        WARN    egress  service/handler.go:198  egress failed   {"nodeID": "NE_BmsUMFynL58w", "clusterID": "", "egressID": "EG_NKdCBdmtVoee", "requestType": "track_composite", "outputType": "file, stream, segments", "error": "internal error"}
2023-09-06T07:32:55.735Z        INFO    egress  service/service_deprecated.go:33        shutting down   {"nodeID": "NE_BmsUMFynL58w", "clusterID": ""}
2023-09-06T07:32:55.735Z        INFO    egress  service/service.go:201  closing server  {"nodeID": "NE_BmsUMFynL58w", "clusterID": ""}
hey @dry-elephant-14928 it would be great if you can have a look at these logs i am using livekit 1.4.5
d
@nice-gpu-3229 which version of Egress are you running? @able-gigabyte-21598 looks like there's a panic here:
Copy code
goroutine 17 [running, locked to thread]:
<http://github.com/tinyzimmer/go-gst/gst.(*Buffer).PresentationTimestamp(...)|github.com/tinyzimmer/go-gst/gst.(*Buffer).PresentationTimestamp(...)>
        /root/go/pkg/mod/github.com/livekit/go-gst@v0.2.34-0.20230623013326-3ca8cb8f62e5/gst/gst_buffer.go:180
<http://github.com/livekit/egress/pkg/pipeline/output.(*Bin).buildSegmentOutput.func1(0x2|github.com/livekit/egress/pkg/pipeline/output.(*Bin).buildSegmentOutput.func1(0x2>?, 0xc000042e68?, 0xc000042e48)
a
that was fixed recently. Will be releasing a new version soon
🙌 3
n
@dry-elephant-14928 we are using latest one 1.7.5
hello @dry-elephant-14928, hello @able-gigabyte-21598, I tried the latest release, and still having issues with trackCompositeEgress while roomCompositeEgress works perfectly fine. I tried all three egress option with trackCompositeEgress : for HLS: only two files in the specified folder: 1. playlist.m3u8 - 137B 2. playlist.m3u8.json - 233B 3. no segement files in the folder 4. and the files are created after ending the stream from OBS for MP4: only two files in the specified folder: 1. name.mp4 - 64kb 2. name.mp4.json - 233B 3. file is not playable for RTMP: tried to stream via rtmp on youtube but dont know due to what reasons youtube didnt recieve any stream. I got all the webhooks, showing that everything is fine. and in the logs I find this: errors are at the bottom of the log file given below
r
Hi @dry-elephant-14928, Hi @able-gigabyte-21598, I am also facing similar issue, the MP4 file recorded was with 0B on S3 with the latest version of egress. Please help me on this for resolving the issue.
d
We just released a new version with a few bugfixes. Could you give it a try?
🙌 1
n
thanks a lot. I will try and let you know.
hey @dry-elephant-14928, thanks for the new version and sorry i didnt get the chance to reply back, since i got busy when track egress was not working we started finding way around RoomComposite and it worked for our use case, But now i came to know that roomcomposite takes 2-6 cores per Egress since it does a lot of transcoding, hence we are thinking switching back to track egress again. Will update you about the new version . AND THANKS FOR THIS AWESOME LIVEKIT TOOL
🙌 1
d
Thanks for the update @nice-gpu-3229! glad to hear you've found a workaround! We've made some improvements to TrackComposite to handle these weird cases, as well as introducing a simpler 
ParticipantComposite
. We'll be posting docs about it soon. But in the mean time, RoomComposite is a reliable way to do what you want
n
sorry @dry-elephant-14928 didnt get the chance to reply, we have continued with roomcomposite and it is working well as expected, and thanks for the 
ParticipantComposite
. and one more thing does 
ParticipantComposite
egress will need less resources(cpu) in comparison to 
RoomComposite
or same as 
RoomComposite
a
it will need much less resources, since it doesn’t use chrome like 
RoomComposite
😃 1
n
i hope this will cut down a lot on our server costing.
hey @able-gigabyte-21598 @dry-elephant-14928 in url ingress sometimes i am getting this error
Copy code
livekit.webhook webhook/url_notifier.go:88      sent webhook    {"url": "<https://livekit-api-client-v1.flutin.com/api/v1/stream/webhook-endpoint>", "event": "ingress_started"}
2023-10-19T12:40:55.675Z        INFO    livekit service/rtcservice.go:299       new client WS connected {"room": "6531178658fbc150ad301f6e_room", "roomID": "RM_fpvJ7yq7JmiT", "participant": "EG_AMoyTKPK4nU9", "pID": "PA_YBrrkzyXW7Te", "remote": false, "connID": "CO_SDoXWuotdhsr", "reconnect": false, "reconnectReason": "RR_UNKNOWN", "adaptiveStream": false}
2023-10-19T12:40:59.975Z        INFO    livekit.webhook webhook/url_notifier.go:88      sent webhook    {"url": "<https://livekit-api-client-v1.flutin.com/api/v1/stream/webhook-endpoint>", "event": "egress_updated"}
2023-10-19T12:41:01.742Z        INFO    livekit.webhook webhook/url_notifier.go:88      sent webhook    {"url": "<https://livekit-api-client-v1.flutin.com/api/v1/stream/webhook-endpoint>", "event": "egress_updated"}
2023-10-19T12:41:05.894Z        INFO    livekit.webhook webhook/url_notifier.go:88      sent webhook    {"url": "<https://livekit-api-client-v1.flutin.com/api/v1/stream/webhook-endpoint>", "event": "egress_updated"}
2023-10-19T12:41:07.596Z        INFO    livekit service/rtcservice.go:375       exit ws read loop for closed connection        {"room": "6531178658fbc150ad301f6e_room", "roomID": "RM_fpvJ7yq7JmiT", "participant": "618a980d7f6980199cd9e6ad_6531178658fbc150ad301f6e_host_preRecorded", "pID": "PA_Q7UyssZGEiXS", "remote": false, "connID": "CO_cog6knzv9iSK", "wsError": "websocket: close 1006 (abnormal closure): unexpected EOF"}
2023-10-19T12:41:07.596Z        INFO    livekit service/rtcservice.go:262       finishing WS connection {"room": "6531178658fbc150ad301f6e_room", "roomID": "RM_fpvJ7yq7JmiT", "participant": "618a980d7f6980199cd9e6ad_6531178658fbc150ad301f6e_host_preRecorded", "pID": "PA_Q7UyssZGEiXS", "remote": false, "connID": "CO_cog6knzv9iSK"}
2023-10-19T12:41:07.598Z        INFO    livekit service/ioinfo.go:145   ingress failed  {"error": "Couldn't download fragments", "ingressID": "IN_i9fyhnnxo63S"}
i know this sounds like the hls video source is corrupted or something wrong with the video source but when i put the same hls url in akamai hls player or any other hls player then videos plays perfectly. the one thing that i have figured out till now but not sure that, this error occurs when video is large, somewhere around 4 hours, but again not sure.
this is how i convert my mp4 videos to hls: ``ffmpeg -f concat -safe 0 -i "${fileinput}" -vf "scale=19201080force_original_aspect_ratio=decrease,pad=19201080(ow-iw)/2:(oh-ih)/2" -c:v h264_nvenc -preset fast -crf 23 -c:a aac -b:a 128k -ar 44100 -ac 2 -hls_time 10 -hls_list_size 0 -f hls "${hlsOutputDirectory}/output.m3u8"``
a
cc @chilly-animal-65189
c
This error indicates that ingress got an error trying to download the segments over http. Do you have the ingress logs? What is the playlist url?
n
@thanks @able-gigabyte-21598 and yes you are right @chilly-animal-65189 the problem is seems to be originating from storage service because i am getting these errors "Couldn't download fragments" only in case of long videos of length around 4 hours long but thanks a lot for your time and support and confirming that this something not related to livekit.
5 Views
Open in Slack
PreviousNext
Powered by