https://livekit.io logo
Join Slack
Powered by
This message was deleted.
# helpdesk
s
This message was deleted.
a
1. For long sessions (especially 10+ hours), I’d recommend using HLS/m3u8 with a longer segment duration as you said. If you really need to stick to mp4, you could try using VideoCodec 
H264_HIGH
for better compression and lowering the resolution and/or bitrate, but I think the files would still be pretty massive. 2. Do you have any errors from the egress logs? I haven’t seen upload issues on our end, it might be object storage or rate limit related
f
Quick update: According to Lionde support, there direct max file size is 5GB (in Object storage for a single file), if we need to upload more than 5GB+ files we need to upload using multipart uploads. Does you guys support multi upload for S3 API compatible object storage in egress when uploading files to object storage ? Let me try get logs as in tests we use k8s extend pods (scaled up worker nodes) which destroyed once the tests are complete. Let me do few checks on long duration and share egress service logs. Is above 10 hrs+ (~12-15 GB) and 24hrs+ (37-40 GB) file size is norm for mp4 recording generation with egress service in your experience @able-gigabyte-21598?
a
We don’t currently support multipart uploads, but it’s something we should be able to add. A 10 hour, 1080p, 30fps mp4 at 3 Mbps would be about 13.5GB, so those sizes seem normal
f
Noted with Thanks. It will greatly appreciate if you guys can enable multipart upload support in egress service. Although, we use HLS/m3u8, still those files will generate same size (as of mp4) with for long duration sessions. Customer always prefer to download files in mp4 format, so later we need to convert HLS/m3u8 files into mp4 again, which will lead double effort.
d
running long recordings with mp4 is a not a good idea from a reliability perspective. if your servers fail for any reason, you'd lose the entire recording.
f
Oh yes, got it @dry-elephant-14928, Thanks. So do you guys have any recommendations any oss or SaaS on converting HLS to mp4 later on reliably for longer duration sessions? Our idea is once the egress request complete ( upload complete to object storage) triggered ( via webhook - if it supported), we trigger a pipeline to convert hls to mp4 ( via set job in k8s) as well. Our initial idea is to use docker of ffmpeg or gstreamer ( as you guys utilized in egress) and run parallel jobs per instance. Or use media conversion via AWS or GCP SaaS and serve the content via GCP/AWS CDN service (more expensive).
@able-gigabyte-21598 egress logs:
Copy code
"error": "S3 upload failed: EntityTooLarge: \n\tstatus code: 400, request id: tx00000000000000daf7f61-0063c0c74b-5521732-default

	"EG_VT5uS6AWDynn", "request_type": "room_composite", "output_type": "file", "status": "EGRESS_ENDING"}
	2023-01-13T08:19:43+05:30 2023-01-13T02:49:43.901Z	DEBUG	egress	pipeline/pipeline.go:469	sending EOS to pipeline	{"nodeID": "NE_XoKS6EGEr4HR", "handlerID": "EGH_R7pj2YudsxyX", "egressID": "EG_VT5uS6AWDynn"}
	2023-01-13T08:19:44+05:30 2023-01-13T02:49:44.079Z	DEBUG	egress	pipeline/pipeline.go:290	EOS received, stopping pipeline	{"nodeID": "NE_XoKS6EGEr4HR", "handlerID": "EGH_R7pj2YudsxyX", "egressID": "EG_VT5uS6AWDynn"}
	2023-01-13T08:19:44+05:30 2023-01-13T02:49:44.463Z	DEBUG	egress	pipeline/pipeline.go:641	pipeline stopped	{"nodeID": "NE_XoKS6EGEr4HR", "handlerID": "EGH_R7pj2YudsxyX", "egressID": "EG_VT5uS6AWDynn"}
	2023-01-13T08:19:44+05:30 2023-01-13T02:49:44.604Z	DEBUG	egress	pipeline/pipeline.go:668	uploading to s3	{"nodeID": "NE_XoKS6EGEr4HR", "handlerID": "EGH_R7pj2YudsxyX", "egressID": "EG_VT5uS6AWDynn"}
	2023-01-13T08:19:48+05:30 2023-01-13T02:49:48.084Z	INFO	egress	utils/cpu.go:77	high cpu load	{"nodeID": "NE_XoKS6EGEr4HR", "load": 0.9185185185185185}
	2023-01-13T08:21:55+05:30 2023-01-13T02:51:55.411Z	INFO	egress	pipeline/pipeline.go:691	could not upload file	{"nodeID": "NE_XoKS6EGEr4HR", "handlerID": "EGH_R7pj2YudsxyX", "egressID": "EG_VT5uS6AWDynn", "error": "EntityTooLarge: \n\tstatus code: 400, request id: tx00000000000000daf7f61-0063c0c74b-5521732-default, host id: ", "location": "S3", "filepath": "egress-loadtest-k8/room-composite/2023-01-12T23_19_38Z-mp4/room-composite-egress-test_mp4_2023-01-12T23_19_38Z-mp4_1.mp4"}
	2023-01-13T08:21:55+05:30 2023-01-13T02:51:55.413Z	DEBUG	egress	pipeline/pipeline.go:668	uploading to s3	{"nodeID": "NE_XoKS6EGEr4HR", "handlerID": "EGH_R7pj2YudsxyX", "egressID": "EG_VT5uS6AWDynn"}
	2023-01-13T08:21:56+05:30 2023-01-13T02:51:56.084Z	DEBUG	egress	pipeline/pipeline.go:731	removing temporary directory	{"nodeID": "NE_XoKS6EGEr4HR", "handlerID": "EGH_R7pj2YudsxyX", "egressID": "EG_VT5uS6AWDynn", "path": "/tmp/lkegress/EG_VT5uS6AWDynn/"}
	2023-01-13T08:21:56+05:30 2023-01-13T02:51:56.714Z	WARN	egress	service/handler.go:121	egress failed	{"nodeID": "NE_XoKS6EGEr4HR", "handlerID": "EGH_R7pj2YudsxyX", "egressID": "EG_VT5uS6AWDynn", "egressID": "EG_VT5uS6AWDynn", "request_type": "room_composite", "output_type": "file", "error": "S3 upload failed: EntityTooLarge: \n\tstatus code: 400, request id: tx00000000000000daf7f61-0063c0c74b-5521732-default, host id: "}
it seems the culprit is indeed the object storage limitation with single upload API cmd (without support for multi-part upload)
@able-gigabyte-21598 @dry-elephant-14928 At the moment, in zoom for free tier, the recordings are save in .zoom format(Assume this is m3u3 - TS files) and once user click on on one of the *.zoom file it automatically create the mp4 file from existing files. Can we embed such script(s) (to support in win and *nix kernal OSs) with egress upload which allow to convert m3u8 to mp4 by clicking the script after user download all HLS / m3u8 files of his/her recording to local device.
a
f
Thanks team 🙂
@able-gigabyte-21598 @dry-elephant-14928 Can we get EGRESS_FAILED event into livekit-server webhooks (https://docs.livekit.io/server/webhooks/#egress-started), which will help developers to handle ERGRESS failed scenarios such as ; • trigger another request based on the failed reason / scenario • inform customer about the recording failed.
a
it will send an egress ended webhook, and you can check 
egressInfo.Status
for 
EGRESS_FAILED
along with an error message in 
egressInfo.Error
f
Aha ok, got it, Thanks
c
an ffmpeg based service/command can convert hls to mp4 fairly easily. You'll still have to deal with massive files adterwards.
f
Yes @chilly-animal-65189, what we decide is to give 'convert to mp4' option, where users who need to have the mp4 can request to convert the file and will use compression as @able-gigabyte-21598 suggested and to convert keeping the quality and size in balance , compress the file with tar.gz as well for now.
c
You may have operational reasons to put the resulting media in gziped tarball, but gzip will not give you any further compression
f
Thank you for your update. Do you any have any insights/ suggestions to futher optimize the process or better solution that we can follow ?
d
Hey Ashan, these are on the roadmap but the team does not have any time to look into them in the next month or so. If #215 is a blocker for you, I would encourage you to submit a PR! It should be well contained in the upload path
f
Sure @dry-elephant-14928, I'm no GO expert at the moment, but will try to go through issue and share feedback.
4 Views
Open in Slack
PreviousNext
Powered by