How to Build an AI Voice Agent: A Founder's Stack Guide
To build an AI voice agent, stream four stages in parallel: speech to text, an LLM, text to speech and a transport that plays the reply while the model is still generating it. The hard parts are turn detection, interruption handling and latency budget, not picking a model.
Key takeaways
- A voice agent is a streaming pipeline, not a single model: transport, speech to text, an LLM and text to speech.
- Pipecat's docs say typical voice interactions complete in 500 to 800ms, so treat that as your latency budget.
- Turn detection decides when the user has finished, and a fixed silence timer is a blunt instrument.
- Interruptions should cancel the LLM, clear the speech buffer and discard audio that has not played.
- Start with an open framework and a hosted model, then self-host the parts whose cost or latency you can measure.
On this page 8
A voice agent feels like one product and is actually four services racing each other. Audio arrives, a speech model transcribes it, a language model decides what to say, and a speech model reads the reply back, all while the next chunk is still being generated. Get the pipeline shape right and the rest is ordinary product work. Get it wrong and no model upgrade will save the demo.
TL;DR
- Wire the pipeline as a streaming loop, not a sequence of blocking calls.
- Budget 500 to 800ms for the full round trip, the range Pipecat's docs give for typical interactions (Pipecat docs).
- Treat turn detection as a product decision, not a config value.
- Handle barge-in from the first day, because real users interrupt.
- Start hosted, measure cost per call, then self-host what pays off.
Disclosure: ZeroRaised publishes this guide and is a launch site for bootstrapped products. Facts come from the linked sources as checked on 30 Oct 2026; ZeroRaised has no commercial relationship with the companies named.
A voice agent pipeline is the chain of services that receives audio, transcribes it, generates a reply with a language model, converts that reply back to speech and streams the audio to the listener, with every stage running concurrently.
What does an AI voice agent pipeline actually look like?#
The flow is documented plainly. The transport receives audio, speech to text transcribes it, a context aggregator adds conversation history, the LLM streams a reply, text to speech converts it to audio, and the transport streams it back (Pipecat docs). Earlier parts of a reply are spoken while the LLM is still generating later parts, so the caller hears the first sentence almost immediately (Pipecat docs).
Pipecat is an open source Python framework, free under BSD-2, that works with any AI provider and any hosting setup, ships client SDKs for JavaScript, React, React Native, iOS, Android and C++, and orchestrates 150+ AI services (Pipecat docs).
The mental model is small: frames are the data packages moving through the system, frame processors do work on them, pipelines connect the processors, and workers run the pipelines (Pipecat docs).
As Deepgram's chief executive puts it, voice AI is "a distinct infrastructure discipline, with its own demands around latency, accuracy, and scale" (Deepgram).
Which speech to text model should you start with?#
Split the decision by workload. Recorded audio suits a batch model tuned for accuracy per dollar. A phone call needs a streaming model tuned for time to first token.
On the batch side, Nemotron 3.5 Transcribe is a free open-weight model that transcribes 40 languages, translates 31 to or from English and attributes speakers in 13 languages, handling recordings up to an hour and up to eight speakers (NVIDIA). It has 30B parameters with about 4B active per token and runs on a single chip (NVIDIA). It ranks first among open-weight models on the Hugging Face Open ASR Leaderboard for English at 5.33% average word error rate (NVIDIA).
It is built for recorded audio, and TechCrunch calls the lack of streaming the drawback (TechCrunch). So use it for call review, notetaking and analysis, and keep a streaming provider for the conversation itself. If you are weighing hosted options, the LLM API selection guide covers the same trade-offs for the text stage.
Do you need turn detection or just a silence timer?#
A silence timer is the obvious first build, and often the first thing to break. A brief pause does not always mean the user is done (Pipecat docs). People pause to think, and a fixed timer can cut them off mid-sentence.
Voice activity detection is the cheaper layer. Pipecat uses Silero VAD, which runs locally on CPU, processes 30+ms chunks in under 1ms, and defaults to start_secs 0.2 and stop_secs 0.2 (Pipecat docs). VAD tells you when speech starts and stops, but it does not tell you whether the thought is finished.
For that, the default is a turn detection model called Smart Turn, with a speech timeout or a turn-completeness option as alternatives (Pipecat docs). The right choice depends on your callers. A restaurant booking tolerates a short timeout. A support call with a frustrated customer needs more patience and a clear way to interrupt.
How do you handle interruptions?#
Barge-in should be on from day one. In Pipecat it is on by default: when the user speaks, the LLM completion is cancelled, the speech synthesiser clears its buffers, and audio that has not played is flushed (Pipecat docs). Without that, the agent keeps talking over the person, which is the fastest way to make a voice product feel broken.
Three details matter in practice. First, cancel early, because a cancelled completion still costs tokens but a late cancellation costs trust. Second, make the resume graceful: after an interruption the agent should acknowledge and continue, not restart. Third, log interruptions: a high rate means turn detection is too eager or replies are too long.
Should you self-host or use a hosted platform?#
Start hosted, because measurement comes first. You cannot choose between a managed platform and your own GPUs until you know cost per call, median latency and tail latency. Pipecat Cloud exists as the hosted option for the same framework you can self-host (Pipecat docs), which makes the migration a deployment change rather than a rewrite.
Deepgram is acquiring the company behind Pipecat, and says the framework stays fully open source and free under BSD 2-clause, with an independent foundation being set up and a vendor-neutral stance (Deepgram). Co-founder Kwindla Hultman Kramer said: "Keeping Pipecat open was fundamental to our decision to join Deepgram" (Deepgram).
One more open piece: the Real-Time Context Standard, co-developed with the Pipecat team, carries audio context such as a whisper or an unclear word, with instructions for the LLM and for how the reply should sound (Deepgram).
What is a sensible build order?#
- Ship a text only prototype. Prove the conversation logic and the tool calls before you add audio. The LLM cost guide applies directly here.
- Add streaming transcription and speech. Keep the whole loop in one process so you can measure latency end to end.
- Tune turn detection on real calls. Record ten awkward conversations and count false turn ends before adding features.
- Add barge-in and test it deliberately. Interrupt the agent mid-sentence in every scenario.
- Instrument cost per call. Split it by transcription, model tokens and speech synthesis, then decide what to self-host using the open versus closed model comparison.
Where ZeroRaised fits#
ZeroRaised is a launch and discovery site for products built without institutional venture capital or accelerator equity. If you have shipped a voice agent, or anything else people can try, you can submit your product. The pricing page explains the current options, and there are no promises of traffic or results.
For the market side of the same story, read where indie voice builders still win. To follow new posts, join the ZeroRaised newsletter.
Sources#
Frequently asked questions
What is the minimum stack for an AI voice agent?
A transport that carries audio, a speech to text model, a language model and a text to speech model, plus logic that decides when the user has finished speaking. A framework such as Pipecat wires those parts together and handles streaming.
How fast should a voice agent respond?
Pipecat's documentation says typical voice interactions complete in 500 to 800ms. Treat that as the budget for the whole loop, so spend it carefully across transcription, model time, speech synthesis and network hops.
Do I need a turn detection model?
Not always, but a plain silence timer performs badly when people pause to think. Pipecat defaults to a turn detection model called Smart Turn and also offers a speech timeout or a turn-completeness check.
Should I self-host the models or use an API?
Start hosted so you can measure real latency and cost per call, then move a stage to self-hosting when the numbers justify it. An open framework keeps that swap possible without a rewrite.
Can I use a free open-weight transcription model?
Sometimes. Models such as Nemotron 3.5 Transcribe are free and accurate, but they are built for recorded audio rather than streaming, so they suit notetaking and analysis more than a live call.