TL;DR: Google’s LiteRT-LM enables tiny LLMs (under 1B parameters) and agent skills to run directly on edge devices like phones and Raspberry Pis. Cormac Brick walks through Gemma 4 performance benchmarks, on-device function calling, fine-tuning workflows, and the memory/safety/UX constraints that shape edge-native AI systems.
Why Edge AI Matters Now
Running AI on the edge isn’t just about privacy or offline capability—it’s about latency and user experience. For live voice translation, real-time image processing, or interactive agents, the round-trip to a cloud server introduces unacceptable delays.
But here’s the shift: tiny language models (TLMs) under 1 billion parameters are now good enough to run locally, and they enable agent skills that were previously impossible without cloud support.
LiteRT-LM: Google’s Edge Runtime
LiteRT-LM (formerly LiteRT) is Google’s runtime for running LLMs on edge devices. It’s the same infrastructure Google uses internally for products like Google Lens, Gboard, and the Pixel’s AI features.
Key capabilities:
- Cross-platform: Android, iOS, Linux, embedded systems
- Hardware acceleration: GPU, NPU, CPU backends
- Model optimization: Quantization, pruning, compilation
- Open source: Available for developers to use
Gemma on Edge: Performance Numbers
Cormac presented benchmarks for Gemma models running on various edge devices:
| Device | Model | Tokens/sec |
|---|---|---|
| Pixel 8 Pro | Gemma 3 4B | ~15-20 |
| iPhone 15 Pro | Gemma 3 4B | ~12-18 |
| Raspberry Pi 5 | Gemma 3 1B | ~5-8 |
The key insight: Gemma 4’s 2B and 4B variants are specifically designed for edge deployment, with a roadmap to even smaller models.
Agent Skills: On-Device Function Calling
Agent skills are the big revelation. Instead of sending every query to the cloud, you can:
- Mood Tracker: Analyze sentiment locally, trigger actions based on emotional state
- Wikipedia Lookup: Use retrieval-augmented generation without network calls
- Custom Skills: Build your own tools that the model can invoke
Constrained Decoding for Reliability
The challenge: how do you ensure the model uses tools correctly?
Solution: Constrained decoding. Instead of letting the model generate anything, you constrain the output to valid tool calls. This prevents the model from hallucinating function names or malformed arguments.
Fine-Tuning Tiny Models
The second half of the talk covers deployment workflows:
- Base model: Start with Gemma 3 1B or 2B
- LoRA fine-tuning: Add task-specific weights without retraining the full model
- Quantization: Convert to INT4 or INT8 for size/speed
- Edge compilation: Convert to LiteRT format for deployment
Google provides the AI Edge Gallery (GitHub) with pre-tuned models and examples.
Synthetic Data for Training
When real training data is scarce, generate synthetic data:
- Use larger models to create training pairs
- Filter for quality with smaller models
- Iterate: generate → filter → fine-tune → evaluate
Third-Party Models and Hardware Acceleration
Beyond Gemma, LiteRT supports:
- Fast VLM: Vision-language models for on-device understanding
- Mobile Actions: Pre-trained models for gesture and activity recognition
- Embedding Gemma: For semantic search without cloud
Hardware acceleration is crucial—NPUs on modern phones can achieve 10-50x speedup over CPU-only inference.
Key Takeaways
- Tiny LLMs are viable: Sub-1B models now match GPT-3.5 quality for many tasks
- Agent skills enable autonomy: On-device function calling removes cloud dependency
- Memory constraints shape design: 2-4GB RAM limits require careful model selection
- Safety remains a challenge: Running unfiltered models on user devices needs guardrails
- The edge-first mindset: Design for constraints from day one, not as an afterthought
The future is edge-native AI—models that live on your phone, in your car, on your IoT devices—not just in data centers.
Speaker: Cormac Brick, Tech Lead on Edge AI at Google
Timestamps:
- 0:00 - Intro: AI on the Edge, Small Language Models, and Gemma
- 0:04 - Enabling App Development: MediaPipe, LiteRT, and System Services
- 0:09 - Small Language Models: Performance, Reach, and Fine-tuning
- 0:11 - Gemma 4: Sizes (E2B and E4B) and AI Core Roadmap
- 0:16 - Gemma on Edge Runtime: Performance Benchmarks
- 0:18 - Agent Skills: Google AI Gallery, Mood Tracker, and Wikipedia Lookup
- 0:31 - Skill Development Deep Dive: Orchestrator and Registry
- 0:41 - Deploying Tiny Models (sub-1B parameters) In-App
- 1:08 - Summary and Key Takeaways
- 1:12 - Q&A
