LiveKit: Self-Hosted WebRTC SFU for Real-Time Video, Audio, and AI Agents

What it does

• Scalable multi-user video/audio conferencing
• Real-time data channels (text, files, bytes, remote procedure calls)
• Speaker detection, simulcast, SVC codecs (VP9, AV1)
• End-to-end encryption
• Selective subscription (clients choose which tracks to receive)
• Moderation APIs (mute, remove, change permissions)
• Embedded TURN server (no separate coturn needed)
• Distributed and multi-region deployment via Redis

The ecosystem

LiveKit isn’t just the SFU server. The org maintains a full stack:

Client SDKs cover: JavaScript/TypeScript, Swift (iOS/macOS), Kotlin (Android), Flutter, React Native, Unity, Rust, C++, and even ESP32.

Networking

WebRTC is notoriously tricky to deploy because it uses UDP. LiveKit needs:

That’s a 10,000-port UDP range open to the internet. This is the single biggest operational headache — firewalls, security groups, and NAT traversal all need to cooperate.

You need a domain with a trusted SSL certificate (self-signed won’t work — WebRTC browsers enforce this). If you enable TURN/TLS, that needs its own domain and cert too.

Production config

1
port: 7880
2
log_level: info
3
rtc:
4
  tcp_port: 7881
5
  port_range_start: 50000
6
  port_range_end: 60000
7
  use_external_ip: true
8
redis:
9
  address: redis-server:6379
10
keys:
11
  my-api-key: my-api-secret
12
turn:
13
  enabled: true
14
  tls_port: 5349
15
  domain: turn.myhost.com
16
  cert_file: /path/to/turn.crt
17
  key_file: /path/to/turn.key

Redis is required for production — it coordinates state across multiple LiveKit nodes. Without it, you’re limited to a single instance.

Hardware

LiveKit is CPU and bandwidth bound, not memory bound. The official recommendation:

• 10 Gbps ethernet or faster
• Compute-optimized VM instances on cloud providers
• For Docker, use --network host for best performance

Performance benchmarks (16-core, c2-standard-16)

All benchmarks measure max participants in a single room. Rooms can scale horizontally — each room fits on one node, but you can have unlimited rooms across nodes.

Audio-only (10 publishers, 3,000 subscribers):

Video meeting (150 publishers, 150 subscribers, 720p simulcast):

Livestreaming (1 publisher, 3,000 subscribers):

The livestreaming scenario is the most bandwidth-heavy — 531 MBps outbound. If you’re on a cloud provider, network egress costs will dominate.

Deployment options

1
# macOS
2
brew install livekit
3

4
# Linux
5
curl -sSL https://get.livekit.io | bash
6

7
# Docker
8
docker run --network host livekit/livekit-server --config config.yaml
9

10
# Dev mode (no config needed)
11
livekit-server --dev

For Kubernetes, use the official livekit/livekit-helm chart. For distributed multi-region, Redis handles cross-node coordination.

Local development

LiveKit’s dev mode strips away all production complexity — no Redis, no SSL, no domain, no TURN. One command and you have a working SFU:

1
livekit-server --dev

This starts a server on 127.0.0.1:7880 with hardcoded credentials (devkey / secret). To accept connections from other devices on your LAN:

1
livekit-server --dev --bind 0.0.0.0

The official client SDKs and example apps (like LiveKit Meet) can point to http://localhost:7880 out of the box. For AI agent development, the LiveKit Agents Python/Node.js SDK connects to the same local endpoint.

What it actually is

The framework provides a Python and Node.js SDK that handles the entire real-time AI pipeline — audio ingestion, voice activity detection, speech-to-text, LLM reasoning, text-to-speech, and audio output — all managed through a unified AgentSession object. You define your agent’s behavior (instructions, tools, model choices) and the framework handles the streaming orchestration.

Core abstractions:

Two pipeline architectures

You can choose between two approaches, or mix them:

STT-LLM-TTS pipeline — strings together three specialized providers. More control over each component, broader provider choice, and you can swap any part independently:

1
User speech → VAD (Silero) → STT (Deepgram Nova-3) → LLM (GPT-4.1 mini) → TTS (Cartesia Sonic-3) → User

Realtime model — a single model handles the full speech-to-speech loop. Lower latency, more natural conversation flow, but limited to providers that offer realtime endpoints (OpenAI, Google Gemini):

1
User speech → OpenAI Realtime API → User

Plugin ecosystem — 65 providers

The framework ships with 65+ plugins under livekit-plugins/, each a separate package following the pattern livekit-plugins-<provider>:

The plugin system uses provider-agnostic interfaces with fallback adapters and stream adapters — swap Deepgram for Google STT by changing one line, everything else stays the same.

A voice agent with tools

This is a working agent that can look up weather — the @function_tool decorator makes any async function available to the LLM:

1
from livekit.agents import (
2
    Agent, AgentServer, AgentSession,
3
    JobContext, RunContext, cli, function_tool, inference,
4
)
5
from livekit.plugins import silero
6

7
@function_tool
8
async def lookup_weather(context: RunContext, location: str):
9
    """Used to look up weather information."""
10
    return {"weather": "sunny", "temperature": 70}
11

12
server = AgentServer()
13

14
@server.rtc_session()
15
async def entrypoint(ctx: JobContext):
16
    session = AgentSession(
17
        vad=silero.VAD.load(),
18
        stt=inference.STT("deepgram/nova-3", language="multi"),
19
        llm=inference.LLM("openai/gpt-4.1-mini"),
20
        tts=inference.TTS("cartesia/sonic-3", voice="9626c31c-bec5-4cca-baa8-f8ba9e84c8bc"),
21
    )
22
    agent = Agent(
23
        instructions="You are a friendly voice assistant.",
24
        tools=[lookup_weather],
25
    )
26
    await session.start(agent=agent, room=ctx.room)
27
    await session.generate_reply(instructions="greet the user and ask about their day")
28

29
if __name__ == "__main__":
30
    cli.run_app(server)

Multi-agent handoffs

Agents can hand off to other agents mid-conversation — an intake agent gathers information, then transfers to a specialist agent. Each agent can use different models, instructions, and tools:

1
class IntroAgent(Agent):
2
    def __init__(self):
3
        super().__init__(
4
            instructions="Gather the user's name and location for a personalized story."
5
        )
6

7
    @function_tool
8
    async def information_gathered(self, context: RunContext, name: str, location: str):
9
        """Called when the user has provided their information."""
10
        context.userdata.name = name
11
        context.userdata.location = location
12
        story_agent = StoryAgent(name, location)
13
        return story_agent, "Let's start the story!"  # handoff
14

15
class StoryAgent(Agent):
16
    def __init__(self, name, location):
17
        super().__init__(
18
            instructions=f"Tell a personalized story for {name} from {location}.",
19
            llm=openai.realtime.RealtimeModel(voice="echo"),  # switch to realtime
20
        )

Built-in testing with LLM judges

Testing LLM agents is hard because responses are non-deterministic. LiveKit Agents includes a test framework with LLM judges — you assert intent rather than exact output:

1
@pytest.mark.asyncio
2
async def test_order_flow():
3
    llm = google.LLM()
4
    async with AgentSession(llm=llm) as sess:
5
        await sess.start(MyAgent())
6
        result = await sess.run(user_input="Hello, I need to place an order.")
7

8
        result.expect.next_event().is_function_call(name="start_order")
9
        await (
10
            result.expect.next_event()
11
            .is_message(role="assistant")
12
            .judge(llm, intent="should be asking the user what they would like")
13
        )

MCP support

Native MCP (Model Context Protocol) integration — connect any MCP server and its tools become available to your agent with one line of configuration. This means your agent can use tools from any MCP-compatible source without writing adapter code.

Running modes

1
# Terminal mode — mic/speaker directly, no server needed
2
python myagent.py console
3

4
# Dev mode — hot reload, connects to LiveKit (cloud or self-hosted)
5
LIVEKIT_URL=http://localhost:7880 LIVEKIT_API_KEY=devkey LIVEKIT_API_SECRET=secret \
6
  python myagent.py dev
7

8
# Production mode
9
python myagent.py start

console mode is the fastest iteration loop — no browser, no server, just your voice and the agent.

Repository structure

1
livekit/agents/
2
├── livekit-agents/          # Core framework
3
│   └── livekit/agents/
4
│       ├── voice/           # AgentSession, Agent, room I/O, transcription
5
│       ├── llm/             # LLM integration, tool definitions, MCP support
6
│       ├── stt/             # Speech-to-text with fallback adapters
7
│       ├── tts/             # Text-to-speech with stream pacing
8
│       ├── ipc/             # Inter-process communication for job distribution
9
│       ├── cli/             # CLI commands (console, dev, start, connect)
10
│       ├── inference/       # Remote model inference via LiveKit Cloud
11
│       └── telemetry/       # OpenTelemetry traces + Prometheus metrics
12
├── livekit-plugins/         # 65+ provider plugins
13
├── examples/                # Domain examples (healthcare, drive-thru, telephony, survey, frontdesk)
14
├── tests/                   # Test suite with mock implementations
15
├── AGENTS.md                # Coding agent instructions
16
└── CLAUDE.md                # Claude Code instructions

Install

1
# Core framework with popular plugins
2
pip install "livekit-agents[openai,silero,deepgram,cartesia,turn-detector]~=1.5"
3

4
# Full development setup from source
5
git clone https://github.com/livekit/agents.git
6
cd agents
7
uv sync --all-extras --dev
8
make check

The developer stack

Starter projects and CLI

LiveKit provides a CLI (lk) that handles project scaffolding, token generation, deployment, and agent management:

1
# Install CLI
2
brew install livekit-cli     # macOS
3
curl -sSL https://get.livekit.io/cli | bash  # Linux
4

5
# Authenticate with LiveKit Cloud (optional — works with self-hosted too)
6
lk cloud auth
7

8
# Scaffold an AI voice agent from a template
9
lk agent init my-agent --template agent-starter-python
10
lk agent init my-agent --template agent-starter-node

Starter projects come with everything wired up: VAD, STT, LLM, TTS, noise cancellation, tests, and an AGENTS.md file optimized for AI coding agents.

Development with coding agents

LiveKit has first-class support for AI coding agents (Claude Code, Cursor, Codex). Two components:

1. LiveKit Docs MCP server — gives your coding agent access to up-to-date documentation, code search across LiveKit repositories, and working examples.
2. LiveKit Agent Skill — provides architectural guidance and best practices for building voice AI applications:

   1
   npx skills add livekit/agent-skills --skill livekit-agents

Each repo also ships an AGENTS.md file that turns any coding agent into a LiveKit expert — covering build commands, architecture, plugin system, job execution flow, and code style.

For self-hosted development, point your coding agent at the livekit/agents repo with the SFU running in --dev mode. It can scaffold agents, add tools, swap model providers, and write tests from a natural language prompt.

Building toward production

The progression from local dev to a deployed solution:

1. Prototype: livekit-server --dev + agent in console mode
2. Integrate frontend: Add a web/mobile client using LiveKit’s client SDKs, connect to the dev server
3. Add business logic: Server-side room management, token generation, user auth
4. Self-host the SFU: Deploy livekit-server with a proper config (Redis, SSL, TURN)
5. Deploy agents: Run agent workers as containerized services, connected to your self-hosted SFU
6. Scale: Horizontal scaling via Redis coordination, load test with lk load-test

The lk load-test command simulates real-world load (publishers, subscribers, bitrate) and is useful for capacity planning before going live.

Open-source SFUs

Mediasoup (~5.4k stars, C++/Node.js) — LiveKit’s closest rival. More low-level, gives you finer control over the media layer. Uses a C++ worker with a Node.js API layer. No built-in TURN, no built-in Redis coordination — you assemble those yourself. Better if you want maximum control and don’t mind building more infrastructure.

Jitsi Meet (~20k stars, Java/JS) — A complete video conferencing solution, not just an SFU. More opinionated, heavier stack (Java backend). Good if you want a drop-in Zoom replacement, less ideal if you’re building a custom application.

Pion WebRTC (~3.5k stars, Go) — A WebRTC toolkit, not an SFU. LiveKit itself is built on Pion. Use this directly if you want to build your own SFU from scratch.

Commercial / managed

Daily, Dyte, 100ms, Vonage, Agora — These are SaaS APIs. You pay per minute, no infrastructure to manage. The tradeoff: vendor lock-in, no control over the media plane, costs scale with usage.

Vapi — Specifically for AI voice agents. Audio-only, closed source. Can provision phone numbers directly. Simpler to start but less flexible than LiveKit Agents.

Comparison