For the agent setup, if I use aws, should I use fargate? or continously running ec2 to enable concurrent call and service continuity, I don't want calls to be abruptly ending, how should the architecture be?

Thanks for reaching out, taeun. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

what about EKS and KEDA design? should I still work on EC2 with ASG?

Thanks — I’m responding to the three sub‑tasks you asked for: (1) a sample EC2 Auto Scaling setup with lifecycle hook for graceful draining, (2) suggested autoscaling metrics/thresholds for concurrency, and (3) a concise SIGTERM handling + testing checklist. I’ll keep examples practical so you can adapt them to your environment. 1) Sample EC2 Auto Scaling setup + lifecycle hook for graceful draining • ASG basics ◦ Launch template or launch configuration that boots an EC2 instance with your LiveKit agent installed and started via a systemd unit or startup script. ◦ Place instances in private subnets across multiple Availability Zones for HA. ◦ Health checks: use EC2 + ELB/target group health checks if you front agents with a load balancer or use a custom agent health endpoint. • Lifecycle hook configuration (recommended values) ◦ Lifecycle Hook Name: livekit-agent-drain-hook ◦ Lifecycle Transition: autoscaling:EC2_INSTANCE_TERMINATING ◦ Heartbeat Timeout: 3600 seconds (1 hour) — matches lengthy drain windows if needed ◦ Default Result: CONTINUE • Flow on scale‑in / termination a. ASG marks instance as terminating and enters lifecycle hook state. b. Your instance receives SIGTERM (ensure the agent process receives it — see SIGTERM checklist). c. Agent enters draining mode: stop accepting new sessions, allow active sessions to complete. d. When agent finishes draining (or reaches your drain_timeout), run a completion call to ASG to finish lifecycle: • Example completion command (instance must have IAM permissions for autoscaling):

aws autoscaling complete-lifecycle-action \
         --lifecycle-hook-name livekit-agent-drain-hook \
         --auto-scaling-group-name livekit-agent-asg \
         --lifecycle-action-result CONTINUE \
         --instance-id <INSTANCE_ID>

1. ASG will then terminate the instance. • Example termination script outline (run after drain completes) ◦ Obtain instance ID via metadata:

curl -s <http://169.254.169.254/latest/meta-data/instance-id>

◦ Call

aws autoscaling complete-lifecycle-action ...

as above ◦ Exit • Notes ◦ Grant instance profile IAM permission for

autoscaling:CompleteLifecycleAction

. ◦ Keep lifecycle hook heartbeat updated if draining takes longer than heartbeat timeout (you can send heartbeats via

record-lifecycle-action-heartbeat

). ◦ Set agent drain_timeout to match expected session lengths (default 30 minutes; can increase to 60+ minutes if needed). 2) Suggested autoscaling metrics and thresholds (based on concurrency) • Metric to use (preferred) ◦ Custom CloudWatch metric per instance: ActiveSessionsPerInstance (reported by the agent process) ◦ Alternatively, aggregate ActiveSessions across fleet and compute average sessions per instance. • Rules and thresholds (example starting point — tune with load testing) ◦ Capacity planning assumption: provision to ~10 concurrent sessions per instance as a starting example (adjust by codecs, CPU, features like STT/LLM usage). ◦ Scale‑out (add instances) ▪︎ Trigger: Average ActiveSessionsPerInstance > 9 sustained for 2 minutes ▪︎ Action: +1 instance (or +10% of fleet if large) ▪︎ Cooldown: 2 minutes (shorter for scale-out to react quickly) ◦ Scale‑in (remove instances) ▪︎ Trigger: Average ActiveSessionsPerInstance < 5 sustained for 10 minutes ▪︎ Action: -1 instance (or -10% of fleet) ▪︎ Cooldown: 10 minutes (longer to avoid flapping and to allow drains to complete) • Alternative auto‑scaling signals ◦ Queue length (if you dispatch jobs via a queue) with KEDA in K8s — scale on queue depth. ◦ Custom metric: EstimatedCpuLoad = reported load from agent’s load function (scale when EstimatedCpuLoad ~ 0.5 if load_threshold is 0.7). • Guidelines for tuning ◦ If

load_threshold

is 0.7 in your agent, use a lower threshold for autoscaling (e.g., trigger at 0.5) to keep headroom. ◦ For voice/agent workloads (long running), prefer conservative scale‑in thresholds and longer cooldowns to avoid killing active sessions. ◦ Monitor and adjust: measure actual sessions per instance, CPU, memory, and latency during peak tests. 3) SIGTERM handling and graceful shutdown checklist (implementation + testing) Implementation checklist • Ensure agent process receives SIGTERM directly ◦ Use

exec

in shell startup scripts (e.g.,

exec /usr/bin/my-agent ...

) so the agent is PID 1 or receives signals correctly. ◦ Avoid wrapper processes that swallow signals or don’t forward them. • Agent drain behavior ◦ Implement a draining state on SIGTERM: ▪︎ stop accepting new work/dispatches, ▪︎ allow existing sessions to finish (up to

drain_timeout

), ▪︎ periodically log drain progress (e.g., active session count). ◦ Configure

drain_timeout

appropriate to your expected longest call (default 30 min; increase if needed). • Lifecycle hook and completion ◦ On successful drain (or timeout), call

complete-lifecycle-action

to allow ASG termination. ◦ If using lifecycle heartbeats, extend heartbeat while draining. • Logging & observability ◦ Log messages for: "received SIGTERM", "entering draining mode", "active sessions: N", and "drain complete — completing lifecycle". ◦ Expose a health check endpoint that returns unhealthy once in draining mode so external load balancers stop routing new traffic. Testing checklist • Unit/Integration tests ◦ Unit test the SIGTERM handler logic in staging. • Manual SIGTERM test ◦ Start the agent in staging, create one or more active sessions, then send

kill -TERM <agent_pid>

. ◦ Verify: ▪︎ Agent logs show "draining worker" and stops accepting new sessions. ▪︎ Existing sessions remain active until completion or drain_timeout. ▪︎ After drain, instance runs the lifecycle completion call. • ASG termination test ◦ Trigger scale-in (or manually terminate an instance from the ASG console) and confirm lifecycle hook pauses termination while draining. ◦ Confirm the instance only terminates after lifecycle completion and that active sessions were not abruptly cut off. • Long session test ◦ Run long-running sessions that exceed typical duration and verify drain_timeout behavior and that heartbeats are extended if needed. • Rolling deploy test ◦ Do a controlled rolling deploy: ensure that multiple instances do not get drained simultaneously (use maximum unavailability or Pod Disruption Budget equivalents) and that overall concurrency capacity remains above desired minimum. • Observability checks ◦ Verify metrics (ActiveSessionsPerInstance, CPU, memory) and logs surface drain events and durations. Optional EKS/KEDA notes (if you later move to Kubernetes) • KEDA can scale on queue depth, but ensure pods have: ◦

preStop

hook or proper SIGTERM propagation, ◦

terminationGracePeriodSeconds

set to allow draining, ◦ PodDisruptionBudget to avoid draining too many pods at once. • Kubernetes adds more control but also more complexity in ensuring signals reach the agent and that drains are coordinated. If you want, I can: • Provide a concrete sample launch template user data script that starts the agent with

exec

and hooks into lifecycle completion. • Draft a CloudWatch metric publishing snippet (Python or shell) for reporting ActiveSessionsPerInstance from the agent. • Suggest initial instance counts and autoscaling tests given your expected concurrency and average call duration — share expected concurrent sessions and average call length and I’ll estimate instance counts and autoscaling parameters. Which of those follow-ups would be most helpful next? Sources: Pricing | LiveKit | Deploy and scale agents on LiveKit Cloud | How much can LiveKit scale? | LiveKit | Server options | LiveKit Documentation | Deployment reliability on LiveKit Cloud | LiveKit

show me ascii diagram of how you would design the calling agent with aws services.

Thanks — below is an ASCII diagram showing a recommended AWS architecture for a LiveKit calling agent deployment that supports concurrent calls and graceful draining (EC2 + ASG pattern). The diagram highlights multi‑AZ EC2 Auto Scaling Groups, lifecycle hooks for draining, networking, monitoring, and external integrations.

┌──────────────────────────────────────────┐
                   │                AWS REGION               │
                   └──────────────────────────────────────────┘
                                 │         ▲
               ┌─────────────────┴─────────┴─────────────────┐
               │                                           │
               │           Auto Scaling Group (ASG)         │
               │        livekit-agent-asg (multi-AZ)        │
               │                                           │
┌──────────────┴──────────────┐             │             ┌───────────────┴──────────────┐
│ Availability Zone A         │             │             │ Availability Zone B           │
│ ┌────────────────────────┐  │             │             │ ┌────────────────────────┐   │
│ │ EC2 Instance (agent)   │  │             │             │ │ EC2 Instance (agent)   │   │
│ │ - systemd / docker     │  │             │             │ │ - systemd / docker     │   │
│ │ - LiveKit Agent server │  │             │             │ │ - LiveKit Agent server │   │
│ │ - publishes metrics    │  │             │             │ │ - publishes metrics    │   │
│ │ - handles SIGTERM ->   │  │             │             │ │ - handles SIGTERM ->   │   │
│ │   drain (stop new jobs,│  │             │             │ │   drain (stop new jobs,│   │
│ │   allow sessions to end)│  │             │             │ │   allow sessions to end)│   │
│ └────────────────────────┘  │             │             │ └────────────────────────┘   │
│                            ┌┴┐           ┌┴┐           ┌┴┐                          │
│                            │LB│ <─► Optional ALB/ NLB for│LB│                         │
│                            └┬┘     health checks / SNI  └┬┘                          │
└─────────────────────────────┼────────────────────────────┼──────────────────────────────┘
                              │                            │
                       ┌──────┴────────────────────────────┴───────┐
                       │             Networking / NAT / SGs         │
                       │ - Security Groups allow TCP 443, 7880-1   │
                       │   and UDP 7882 (WebRTC) from Internet     │
                       │ - Private subnets for agent instances     │
                       └──────┬────────────────────────────────────┬┘
                              │                                   
      ┌───────────────────────┴──────────────────────────┐
      │        Lifecycle Hook: autoscaling:EC2_INSTANCE_TERMINATING
      │ - Hook name: livekit-agent-drain-hook
      │ - Heartbeat timeout: e.g., 3600s (adjust as needed)
      │ - On termination: instance receives SIGTERM -> agent drains -> 
      │   instance calls aws autoscaling complete-lifecycle-action
      └────────────────────────────────────────────────────┘

                              │
       ┌──────────────────────┴──────────────────────────┐
       │                 Observability / Control          │
       │ ┌────────────┐  ┌────────────┐  ┌───────────────┐ │
       │ │CloudWatch  │  │SSM / S3 /  │  │ Custom Metrics │ │
       │ │(metrics/logs) │ │ Parameter  │  │ ActiveSessions │ │
       │ └────────────┘  │ Store/ S3  │  │ PerInstance   │ │
       │                 └────────────┘  └───────────────┘ │
       └───────────────────────────────────────────────────┘

                              │
       ┌──────────────────────┴──────────────────────────┐
       │                Supporting Services              │
       │ - Amazon S3   : recordings, logs, artifacts     │
       │ - Amazon RDS  : call metadata, agent state      │
       │ - Secrets Manager / SSM Parameter Store: keys   │
       └──────────────────────────────────────────────────┘

                              │
       ┌──────────────────────┴──────────────────────────┐
       │                 External Integrations            │
       │ ┌─────────────┐  ┌──────────────┐  ┌─────────────┐│
       │ │ LiveKit     │  │ AI Services  │  │ SIP / PSTN  ││
       │ │ Server/Cloud│  │ (LLM, STT,   │  │ Providers   ││
       │ │ (job dispatch,│ │  TTS)        │  │ (Twilio)    ││
       │ │  media router)│ │ (low-latency │  │             ││
       │ └─────────────┘  │  endpoints)   │  └─────────────┘│
       └──────────────────────────────────────────────────┘

Notes and mapping to diagram elements • EC2 instances run long‑lived LiveKit agent processes (not short‑lived Fargate tasks). Use non‑burstable instance types (c5/c6i/m5 families). • ASG provides scaling; configure lifecycle hook (livekit-agent-drain-hook) so instances enter draining state on termination. The agent should stop accepting new sessions and call back to ASG when draining completes. • Publish per‑instance metric ActiveSessionsPerInstance to CloudWatch for autoscaling decisions. • Use CloudWatch logs/metrics and a health endpoint so load balancers and orchestration know when an instance is draining/unhealthy. • Store recordings and artifacts in S3; use RDS for metadata if needed. • External AI services (LLM/STT/TTS) should be reachable with low latency—co‑locate in same region if possible. If you want, I can: • Produce a simplified ASCII sequence diagram showing the termination flow (SIGTERM → drain → complete-lifecycle-action). • Generate a sample user-data snippet that uses exec to start the agent and includes the lifecycle-complete script. Which of these would you like next? Sources: Checklist for Regional Deployments | LiveKit | Self-hosted deployments | LiveKit Documentation | Running LiveKit on AWS | Building Performant Voice Agents in India | LiveKit

show me the same output with EKS KEDA approach

Thanks — below is an ASCII diagram and brief explanation showing a recommended EKS + KEDA architecture for running LiveKit calling agents. It highlights cluster components, KEDA scaling (queue/metric-driven), pod lifecycle and graceful draining, observability, and external integrations. The layout mirrors the EC2+ASG diagram so you can compare tradeoffs easily. ASCII diagram (EKS + KEDA)

┌──────────────────────────────────────────┐
                      │                AWS REGION               │
                      └──────────────────────────────────────────┘
                                     │
                 ┌───────────────────┴───────────────────┐
                 │           Amazon EKS Cluster           │
                 │     (managed control plane, nodegroups)│
                 └───────────────────┬───────────────────┘
                                     │
             ┌───────────────────────┴────────────────────────┐
             │            KEDA ScaledDeployment / HPA         │
             │  ┌───────────────────────────────────────────┐ │
             │  │ KEDA Scaler (triggers on queue depth,     │ │
             │  │  CloudWatch metric, or custom metric)     │ │
             │  └───────────────────────────────────────────┘ │
             │                      │                          │
             │    ┌─────────────────┴─────────────────┐        │
             │    │       livekit-agent Deployment     │        │
             │    │  (multiple pods across nodegroups) │        │
             │    └─────────────────┬─────────────────┘        │
             │                      │                          │
┌────────────┴────────────┐  ┌──────┴──────┐  ┌────────────┴────────────┐
│ Node Group A (EC2 nodes)│  │  Node Group B │  │ Node Group C (spot?)  │
│ ┌────────────────────┐  │  │──────────────│  │ ┌──────────────────┐  │
│ │ livekit-agent pod   │  │  │ livekit-agent│  │ │ livekit-agent pod │  │
│ │ - SIGTERM → drain   │  │  │ pod         │  │ │ - preStop handler  │  │
│ │ - preStop hooks     │  │  │ - preStop   │  │ │ - readiness=false  │  │
│ │ - readiness probe   │  │  │ - readiness │  │ │ - drains sessions  │  │
│ └────────────────────┘  │  └──────────────┘  │ └──────────────────┘  │
└─────────────────────────┘                    └─────────────────────────┘

             ┌──────────────────────────────────────────────────┐
             │         KEDA / Kubernetes control plane          │
             │ - Scales Deployment replicas based on triggers   │
             │ - Uses ScaledObject referencing metric source    │
             │   (SQS/RabbitMQ/CloudWatch/custom-metric)       │
             └──────────────────────────────────────────────────┘

                        ┌────────────────────────────┐
                        │   Ingress / Service (LB)   │
                        │ - ALB / NLB for signaling  │
                        │ - NodePort/LoadBalancer svc│
                        └────────────────────────────┘
                                     │
             ┌───────────────────────┴────────────────────────┐
             │   Observability & Control Plane                │
             │  - CloudWatch / Prometheus / Grafana           │
             │  - EKS + CloudWatch Container Insights         │
             │  - SSM / Secrets Manager for secrets           │
             │  - Metrics: ActiveSessionsPerPod, cpu, mem     │
             └────────────────────────────────────────────────┘

             ┌────────────────────────────────────────────────┐
             │         Storage / Backend / External          │
             │  - Amazon S3 (recordings)                     │
             │  - Amazon RDS (metadata)                      │
             │  - LiveKit Server / Job Dispatcher            │
             │  - External AI services (LLM/STT/TTS)         │
             │  - SIP / PSTN Providers for dialing           │
             └────────────────────────────────────────────────┘

Key notes and behaviors • KEDA scaler ◦ KEDA watches an external metric (e.g., job queue depth, CloudWatch custom metric, or LiveKit dispatch metric). ◦ When metric exceeds threshold, KEDA increases replicas of the livekit-agent Deployment (or ScaledObject). ◦ When metric drops, KEDA scales down replicas. You must protect active sessions during scale-in (see draining). • Graceful shutdown / draining in Kubernetes ◦ Use

preStop

hook and set

terminationGracePeriodSeconds

large enough to allow active sessions to finish (e.g., 30–60 minutes depending on expected longest call). ◦ On pod termination Kubernetes sends SIGTERM to the container; agent must: ▪︎ Mark readiness=false (so Service/Ingress stops routing new traffic). ▪︎ Stop accepting new jobs. ▪︎ Allow existing sessions to complete or reach configured

drain_timeout

. ▪︎ Log drain progress and call any lifecycle completion if using external lifecycle coordination. ◦ Ensure startup scripts use

exec

so SIGTERM reaches the agent process. ◦ Use PodDisruptionBudget to limit how many pods can be drained at once (avoid reducing capacity below safe minimum). • Metrics to scale on ◦ Preferred: queue depth or ActiveJobsPending metric (KEDA ScaledObject source). ◦ Alternative: custom metric ActiveSessionsPerPod published to CloudWatch/Prometheus and used by KEDA or HPA. ◦ If the agent exposes a

load

metric (0..1), scale when fleet‑wide average exceeds a conservative threshold (e.g., scale at 0.5 when load_threshold is 0.7). • Node groups and instance types ◦ Use EKS managed node groups with non‑burstable instance types (c5/m5/c6i) for agent pods. ◦ Consider separate node group types (on‑demand for critical, spot for lower priority) and taints/tolerations to control pod placement. • Safety and stability ◦ Use PodDisruptionBudgets and limit concurrent evictions. ◦ Use readiness probes: on drain, set readiness to fail so no new sessions are routed to draining pod. ◦ Extend

terminationGracePeriodSeconds

in Deployment spec to match drain timeout. ◦ Monitor and tune scale‑in cooldowns to avoid killing pods with active sessions. When to choose EKS + KEDA vs EC2 + ASG • Choose EKS + KEDA if: ◦ You want event‑driven scaling (queue depth) and fine‑grained autoscaling control. ◦ You already operate Kubernetes clusters and can manage the extra complexity. • Choose EC2 + ASG if: ◦ You prefer simpler lifecycle hooks and predictable instance-level draining with lower operational overhead. ◦ You want fewer layers between the termination signal and the agent process. If you want, I can: • Produce an ASCII sequence diagram showing pod termination flow (SIGTERM → readiness=false → drain → termination). • Generate a sample Kubernetes Deployment + ScaledObject/ScaledJob snippet showing preStop, readinessProbe, and terminationGracePeriodSeconds configured. Which of those would you like next?