How many jobs can a worker handle at once?

Thanks for reaching out, Jayanth. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

what will happen to the jobs that are not accepted by the worker once it reaches the load threshold? Will they be rejected or be awaited?

When a worker in LiveKit reaches its load threshold, it stops accepting new jobs. If a job is not accepted by a worker within a predetermined timeout period, the job is routed to another available worker. This process is managed by LiveKit's embedded job distribution system, ensuring that jobs are efficiently dispatched to workers that can handle them. Sources: Deployment and scaling | LiveKit Docs

What if I am running only one worker and the job is not accepted?

If you are running only one worker and it cannot accept a job, the job would not be assigned since there are no other workers available to route the job to. LiveKit's job distribution system relies on having multiple workers to ensure jobs are efficiently dispatched. In such a scenario, you may need to monitor the worker's load and consider scaling up to handle additional jobs. Sources: Deploying to production | LiveKit Docs | Deployment and scaling | LiveKit Docs

I am running a 16GB RAM instance with 4 cores. calculate and tell me how many jobs I can run on a single worker.

The number of jobs a single worker can handle depends on the resource requirements of each job type and the worker's load threshold. By default, a worker stops accepting new jobs when its load exceeds 75% of CPU usage. For example, lightweight jobs like TrackEgress can allow hundreds of simultaneous jobs on a single instance, while more resource-intensive jobs will reduce this number. To calculate the exact number of jobs your 16GB RAM, 4-core instance can handle, you would need to consider the specific resource consumption of each job type. If you can provide details about the job types you plan to run, I can assist further. Sources: Deployment and scaling | LiveKit Docs

I am running a 16GB RAM instance with 4 cores. calculate and tell me the maximum number of workers I can deploy

Based on LiveKit's recommendations, a 16GB RAM, 4-core instance can support approximately 25 concurrent voice agents, as the CPU (4 cores) would be the limiting factor rather than the RAM. This calculation assumes that each worker is handling voice-to-voice sessions, which require 4 cores for every 25 concurrent sessions. If you are running different types of jobs, the number of workers may vary depending on their specific resource requirements. Sources: Deploying to production | LiveKit Docs | Deployment and scaling | LiveKit Docs

give me a prewarm and load function that helps me track the compute usage. print it to logs

Here is an example of a prewarm and load function that tracks compute usage and prints it to logs in LiveKit:

from livekit.agents import JobProcess, JobContext, AgentSession, WorkerOptions, cli, metrics
from livekit.plugins import silero
from livekit.agents.voice import MetricsCollectedEvent

def prewarm(proc: JobProcess):
    # Preload the VAD model to improve startup time
    proc.userdata["vad"] = silero.VAD.load()

async def entrypoint(ctx: JobContext):
    # Set up logging context
    ctx.log_context_fields = {
        "room": ctx.room.name,
    }
    await ctx.connect()
    
    # Create session using prewarmed VAD
    session = AgentSession(
        vad=ctx.proc.userdata["vad"],
        # Add other components like llm, stt, tts as needed
    )
    
    # Set up metrics collection
    usage_collector = metrics.UsageCollector()
    
    @session.on("metrics_collected")
    def _on_metrics_collected(ev: MetricsCollectedEvent):
        metrics.log_metrics(ev.metrics)
        usage_collector.collect(ev.metrics)
    
    async def log_usage():
        summary = usage_collector.get_summary()
        <http://logger.info|logger.info>(f"Usage: {summary}")
    
    # Register shutdown callback to log final usage
    ctx.add_shutdown_callback(log_usage)
    
    # Rest of your agent code...

if __name__ == "__main__":
    cli.run_app(WorkerOptions(entrypoint_fnc=entrypoint, prewarm_fnc=prewarm))

This implementation: • Uses a prewarm function to load the Silero VAD model in advance • Sets up a metrics collector to track usage • Registers an event handler for metrics collection • Adds a shutdown callback to log the final usage summary when the session ends Let me know if you need further assistance with this! Sources: Tree View: | dse-workbench

I want to track the CPU and cores usage using prewarm/load function. tell me what I can do.

To track CPU and core usage in LiveKit, you can use the

load_fnc

parameter in

WorkerOptions

along with LiveKit's CPU monitoring utilities. Here's an example implementation:

from livekit.agents.utils.hw import get_cpu_monitor

# Get the appropriate CPU monitor
cpu_monitor = get_cpu_monitor()

# Define a load function that reports CPU usage
def load_fnc():
    # Returns CPU usage as a value between 0 and 1
    return cpu_monitor.cpu_percent()

# Create worker options with the load function
opts = WorkerOptions(
    entrypoint_fnc=entrypoint,
    load_fnc=load_fnc,
    load_threshold=0.8,  # Stop accepting new jobs when CPU usage is above 80%
)

You can also use

cpu_monitor.cpu_count()

to get the number of available CPU cores (or fractional cores in containerized environments). This setup will allow you to monitor CPU usage and control job acceptance based on the load threshold. Let me know if you need further assistance! Sources: Tree View: | dse-workbench | Tree View: | dse-workbench | Worker options | LiveKit Docs

Is there a callback function for knowing if a process was killed because of being unresponsive or initialization timeout?

LiveKit does not provide a specific callback function to directly detect if a process was killed due to being unresponsive or an initialization timeout. However, the system does log error messages in such cases: • For unresponsive processes:

"process is unresponsive, killing process"

• For initialization timeouts:

"initialization timed out, killing process"

Additionally, LiveKit emits events for process lifecycle stages (e.g.,

process_created

,

process_started

,

process_ready

,

process_closed

), which you can use to track process status. While these events do not specifically indicate the reason for termination, you can monitor logs or implement custom logic to handle such scenarios. Let me know if you need further assistance! Sources: Tree View: | dse-workbench | Tree View: | dse-workbench

give me a code snippet which uses all the lifecycle hooks for a process

Unfortunately, I couldn't find a complete code snippet that demonstrates using all lifecycle hooks for a process in LiveKit. However, I can guide you on how to use the lifecycle hooks such as

on_enter

,

on_exit

, and

on_user_turn_completed

in the context of LiveKit's Agents framework. If you can provide more details about your specific use case, I can help you create a custom implementation using these hooks. Let me know how you'd like to proceed!

how to run multiple workers on a dingle agent

In LiveKit, you can run multiple workers on a single agent by deploying multiple instances of the same worker code. These instances automatically form a worker pool, and LiveKit handles load balancing across all available workers in the pool. Each worker can also handle multiple jobs simultaneously, with each job running in its own isolated process. This ensures that if one job crashes, it won't affect others running on the same worker. To scale up, you simply deploy more instances of the worker code, either locally or in a production environment. LiveKit will automatically manage the distribution of jobs across the worker pool based on their availability and capacity. Sources: Deployment and scaling | LiveKit Docs | Worker lifecycle | LiveKit Docs

give me a sample code snippet which demonstrates running two workers on a single agent

Unfortunately, I couldn't find a specific code snippet that demonstrates running two workers on a single agent in LiveKit. However, I can guide you on how to set up multiple instances of the worker code to achieve this. Each instance of the worker code will act as a separate worker, and LiveKit will automatically manage the load balancing between them. If you'd like, I can help you create a custom implementation based on your requirements. Let me know how you'd like to proceed!

yes, give me the steps to run two workers

To run two workers on a single agent in LiveKit, follow these steps: 1. *Prepare Your Worker Code:*Ensure you have your worker code ready. This should include the necessary entrypoint function and any configurations required for the worker to perform its tasks. 2. *Set Up Worker Options:*Define the

WorkerOptions

for your worker, including the entrypoint function and any load or prewarm functions if needed. For example:

from livekit.agents import WorkerOptions, cli

def entrypoint(ctx):
    # Worker logic here
    pass

opts = WorkerOptions(entrypoint_fnc=entrypoint)

3. *Run the First Worker:*Start the first worker instance by running the worker code. For example, if your worker script is named

worker.py

, you can run it using:

python worker.py

4. *Run the Second Worker:*Start another instance of the same worker code. You can do this by opening a new terminal and running the same command: `python worker.py`Each instance will act as a separate worker, and LiveKit will automatically manage the load balancing between them. 5. *Verify Worker Pool:*Check the logs or monitoring tools to ensure that both workers are running and are part of the same worker pool. LiveKit will distribute jobs across the workers based on their availability and capacity. Let me know if you need further assistance with any of these steps!