How to Build a Voice AI Agent on a VPS: A Step-by-Step Guide
Voice agents are already handling practical business tasks: answering calls, covering routine requests, and helping call centers absorb peak-hour traffic. A ready-made SaaS platform is not the only way to build one. You can run a voice AI agent on a VPS, keep control of the stack, and decide where customer voice data is processed and stored.
This guide covers the full setup: choosing a server, installing the required software, and connecting speech recognition, an LLM, and text-to-speech into one pipeline. We will also look at deployment mistakes, latency, infrastructure limits, and the use cases where a self-hosted agent already makes sense.
Our view is simple: the first version does not need to be fully local or overly complex. It needs to respond fast enough, handle real calls reliably, and scale without forcing you to rebuild the system. A Serverspace VPS fits this approach because CPU and RAM can be increased as traffic grows.
What a Voice AI Agent Actually Is
A voice AI agent listens to a person, understands the request, and replies out loud — ideally before the caller assumes the line has dropped.
The system has three core parts: speech recognition (STT) converts audio into text, a language model generates the answer, and speech synthesis (TTS) turns it back into voice. An orchestration layer connects them, tracks pauses, and decides whether the person has finished speaking or is simply thinking.
Unlike a traditional IVR, a voice agent does not force callers through a rigid “press 1, press 2” menu. It works with natural speech and responds to the meaning of the request. Audio is usually processed as a stream, so the answer starts forming before the speaker finishes. Otherwise, every reply comes with an awkward pause that makes the conversation feel anything but natural.
You also do not need several GPUs for the first version. Lightweight STT and TTS models can run on CPU, while the language model can be accessed through an external API. The VPS handles calls and orchestration, and the heavier computation runs elsewhere. For testing the idea, this is usually more practical than investing in powerful hardware and fully local inference from day one.
What the Pipeline Is Built From
Before moving on to setup, it helps to know which tools are typically used at each stage. In practice, the toolkit looks roughly like this.
| Task | Tool | Notes |
|---|---|---|
| Speech recognition | Whisper | Strong accuracy, but resource-hungry |
| Speech recognition (streaming) | faster-whisper | Same model family, but faster and lighter on CPU servers |
| Local language model | Ollama | Simple way to run open models without manual inference setup |
| Speech synthesis (lightweight) | Piper | Runs on CPU, low latency, voice sounds a bit mechanical |
| Speech synthesis (natural) | Coqui TTS | More natural-sounding voice, but needs a beefier setup |
| Dialogue orchestration | Pipecat | Ready-made building blocks for wiring STT, LLM, and TTS into one flow |
Building a Voice Agent on a VPS: Step by Step
Step 1. Choosing and Preparing the Server
For a lightweight setup without a local language model, 2–4 vCPUs and 4–8 GB of RAM are enough: lightweight STT and TTS versions barely tax the server, while an external API does the heavy lifting. It's also worth budgeting for an SSD or NVMe drive — recognition and synthesis models load into memory on service startup, and on a slow drive that noticeably drags out the first launch after a reboot. If you plan to run the language model locally — say, via ollama run — plan for at least 8 vCPUs and 16 GB of RAM, and for models above 7B parameters a GPU configuration comes in handy. For the OS, Ubuntu 22.04 or 24.04 LTS is a sensible choice: it's easier to find ready-made guides and Docker images for it. It's also worth thinking about server region up front: the closer it is to your users, the smaller the network delay on top of the pipeline's own latency.
Step 2. Setting Up the Environment
Once you've connected to the server over SSH, update the system and install the base set of packages: Docker to isolate services, Python for orchestration scripts, and ffmpeg for handling audio streams.
sudo apt update && sudo apt upgrade -y
sudo apt install -y docker.io docker-compose python3-pip ffmpeg
sudo systemctl enable docker --now
This is also a good moment to configure the firewall and close every port except the ones you actually need — more on that in the risks section.
Step 3. Setting Up Speech Recognition
faster-whisper is a practical choice: it's compatible with Whisper models but noticeably faster in streaming mode. A mid-sized model (small or medium) gives a reasonable balance between accuracy and speed on a CPU server.
pip install faster-whisper
The initialization code specifies model size, language, and device: model_size="small", language="en", device="cpu". On a GPU server, the device value switches to cuda, and recognition speeds up several times over.
Step 4. Connecting the Language Model
There are two routes here. The first is local inference through Ollama: data never leaves the server, but reply quality is limited by the capabilities and size of the open model. The second is calling an external API, which gives access to more capable models but adds network latency and a per-token cost.
curl -fsSL https://ollama.com/install.sh | sh
ollama run llama3
What matters a lot when working with voice is reply length. It's worth explicitly constraining the system prompt: ask the model to answer in short sentences without lists or tables, because speech synthesis will read absolutely everything aloud, bullet markers included.
Step 5. Speech Synthesis
Piper fits scenarios where low latency and CPU-only operation matter: the voice sounds clear, if a little mechanical. Coqui TTS sounds more natural, but needs more server resources and takes a bit longer to generate audio.
pip install piper-tts
You'll need to download the matching voice model for your target language — usually a separate file that gets referenced through a configuration parameter when the synthesis service starts.
Step 6. Assembling the Pipeline and Orchestration
At this stage the three components get wired into a single flow through a framework like Pipecat or LiveKit Agents. The orchestrator handles voice activity detection (VAD), manages interruptions — when a person starts talking before the agent has finished its reply — and controls overall dialogue timing. In practice, latency breaks down roughly like this: 200–400 ms for recognition, 500 ms to 1 second for the language model to generate a reply, and 200–300 ms for synthesis. In total, staying under a second and a half keeps the dialogue feeling natural.
Step 7. Connecting a Communication Channel
A finished pipeline still needs something to plug into. For telephony, a SIP trunk or a service like Twilio works; for a website, a WebRTC widget; for messengers, handling voice messages in Telegram. If you're using WebRTC through NAT, you'll need a STUN/TURN server, or a chunk of calls simply won't connect.
Step 8. Testing, Monitoring, and Production
Before launch, test the agent on real voices — with accents, background noise, and varying speech speed. After that you need monitoring for CPU and memory load, transcript logging for quality control, and regular system updates. VPS snapshots help you recover quickly after a configuration failure, and as load grows you can move some components onto separate servers — for example, the language model onto a GPU box while orchestration and telephony stay on the original VPS. It also helps to keep a separate log of recognition errors: if the agent consistently misunderstands the same phrases, that's a signal to tune the STT vocabulary or add the missing terms to the language model's prompt.
More Control, More Responsibility
A self-hosted voice agent gives you far more freedom than a ready-made platform, but that freedom comes with extra work.
You control where voice data is stored and processed, can replace any part of the pipeline, and choose between open-source models and paid APIs as the project grows. Integration with internal systems such as a CRM, knowledge base, or ticketing platform is also much easier when you own the stack.
The trade-off is that there is no real turnkey setup. Your team is responsible for deployment, security, updates, monitoring, and reliability. Open-source STT and TTS models may also sound less natural than leading commercial services, while the first setup will take longer than connecting a SaaS widget.
In our view, self-hosting makes sense when data control, customization, or integration matters more than launching in a single afternoon.
Where Voice Agents Usually Break
A self-hosted voice agent can work well, but several issues need attention before it becomes part of a critical business process.
- Latency. Every stage adds a few hundred milliseconds. Combined, they can turn a normal conversation into an awkward exchange of long pauses. If the agent answers too slowly, the quality of the model no longer matters.
- Recognition errors. Accents, background noise, poor phone lines, and unusual phrasing can confuse even strong STT models. One wrong transcription may send the rest of the dialogue in the wrong direction.
- Confident but incorrect answers. An LLM may invent details that were never in the company’s knowledge base. This is especially risky in voice calls, where a calm, confident tone can make a false answer sound convincing. Ground the model in verified data and instruct it to admit when it does not know.
- Privacy and security. Voice recordings may contain personal data, so define storage periods, access rights, and deletion rules before launch. SIP and WebRTC endpoints also need authentication and call limits. Otherwise, your agent may suddenly develop a costly interest in international calls.
- Costs at scale. Self-hosting is not automatically cheaper than SaaS. API expenses grow with call volume, and at some point the price gap may become surprisingly small.
The advice is simple: test the agent on real calls and calculate at least a month of expected usage before making it part of a core workflow.
Practical Use Cases
Customer Support and Intake
The agent answers routine questions — order status, business hours, delivery terms — and only routes complex cases to a live operator. This takes the load off the line during peak hours and cuts wait times for everyone else calling in.
An Internal Voice Assistant for Staff
For example, a voice interface for IT support: an employee describes the problem out loud, the agent searches the company knowledge base, and either reads out an instruction or opens a ticket for an engineer.
Bookings and Reservations
Restaurants, clinics, and salons use a voice agent to take booking requests outside the receptionist's working hours — without losing customers who call in the evening or on weekends.
A Voice Interface for People with Visual Impairments
Fully voice-driven control removes the need to interact with a visual interface at all, which widens the product's reach to a broader audience.
Outbound Reminders and Callbacks
The agent calls customers itself to remind them of an appointment, confirm a delivery, or clarify order details. Unlike a mass SMS blast, a live voice conversation lets the customer ask a question or reschedule on the spot — and the agent handles that without an operator involved.
Common Deployment Mistakes
- Choosing overly heavy models without accounting for latency — the dialogue starts to "freeze" for several seconds.
- Letting the language model give long, list-heavy answers that speech synthesis awkwardly reads out in full.
- Pushing the agent to production without testing on real voices, accents, and background noise.
- Leaving SIP and WebRTC ports open without authentication or limits on call volume.
- Skipping monitoring and logging, so problems only surface once users start complaining.
Start Small, Then Listen
Building a voice AI agent on a VPS is mostly a matter of choosing the right components, connecting them into a stable pipeline, and keeping latency under control. For a first version, lightweight STT and TTS models with an external LLM API are usually enough. Local inference and more powerful hardware can come later, once traffic, costs, or privacy requirements justify them.
The real test starts after launch. People leave gaps of only about 200 milliseconds between turns in a normal conversation, so even a technically impressive agent will feel slow if every reply takes several seconds.
Review logs, listen to real calls, and adjust prompts based on the phrases customers actually use. In our view, the hardest part is not teaching the agent to speak. It is making the conversation feel natural enough that callers stop thinking about the software behind it.
For more practical guides on cloud services and server infrastructure, visit the Serverspace blog.
Frequently Asked Questions (FAQ)
Do you need a GPU to run a voice AI agent?
Not necessarily. A lightweight setup can run speech recognition and speech synthesis on CPU, while the language model is accessed through an external API. A GPU becomes useful when you want to run larger local models, support more simultaneous calls, or reduce inference latency.
What VPS configuration is recommended for a voice AI agent?
For an initial deployment using an external LLM API, a VPS with 2–4 vCPUs, 4–8 GB of RAM, and SSD or NVMe storage is usually sufficient. Fully local inference or higher call volumes may require at least 8 vCPUs, 16 GB of RAM, or a GPU-enabled server.
What is the recommended latency for a natural voice conversation?
A good target is approximately 1–1.5 seconds between the end of the user's sentence and the beginning of the agent's reply. Longer delays make the interaction feel unnatural, especially in phone calls or live website conversations.
Can a voice AI agent work without telephony?
Yes. A website-based voice assistant can use WebRTC without a SIP trunk or phone number. Telephony is only required when the agent must receive or make regular phone calls through services such as SIP providers or Twilio.
How can a self-hosted voice agent be protected from abuse?
Use firewall rules, authentication for SIP and WebRTC endpoints, TLS encryption, call-duration limits, rate limiting, restricted API credentials, and continuous log monitoring. Only required ports should be exposed to the public internet.
When is self-hosting better than using a voice AI SaaS platform?
Self-hosting is a better fit when you need greater control over customer data, custom integrations, flexible model selection, or infrastructure that can be adapted to specific business requirements. A managed SaaS platform is usually easier when rapid deployment matters more than customization and operational control.
