The IT Nerd

Deepgram today announced the general availability (GA) of its Voice Agent API, a single, unified voice-to-voice interface that gives developers full control to build context-aware voice agents that power natural, responsive conversations. Combining speech-to-text, text-to-speech, and large language model (LLM) orchestration with contextualized conversational logic into a unified architecture, the Voice Agent API gives developers the choice of using Deepgram’s fully integrated stack (leveraging industry-leading Nova-3 STT and Aura-2 TTS models) or bringing their own LLM and TTS models. It delivers the simplicity developers love and the controllability enterprises need to deploy real-time, intelligent voice agents at scale. Today, companies like Aircall, Jack in the Box, StreamIt, and OpenPhone are building voice agents with Deepgram to save costs, reduce wait times, and increase customer loyalty.

In today’s market, teams building voice agents are often forced to choose between two extremes: rigid, low-code platforms that lack customization, or DIY toolchains that require stitching together STT, TTS, and LLMs with significant engineering effort. Deepgram’s Voice Agent API eliminates this tradeoff by providing a unified API that simplifies development without sacrificing control. Developers can build faster with less complexity, while enterprises retain full control over orchestration, deployment, and model behavior, without compromising on performance or reliability.

Developer Simplicity and Faster Time to Market

For teams taking the DIY route, the challenge isn’t just connecting models but also building and operating the entire runtime layer that makes real-time conversations work. Teams must manage live audio streaming, accurately detect when a user has finished speaking, coordinate model responses, handle mid-sentence interruptions, and maintain a natural conversational cadence. While some platforms offer partial orchestration features, most APIs do not provide a fully integrated runtime. As a result, developers are often left to manage streaming, session state, and coordination logic across fragmented services, which adds complexity and delays time to production.

Deepgram’s Voice Agent API removes this burden by providing a single, unified API that integrates speech-to-text, LLM reasoning, and text-to-speech with built-in support for real-time conversational dynamics. Capabilities such as barge-in handling and turn-taking prediction are model-driven and managed natively within the platform. This eliminates the need to stitch together multiple vendors or maintain custom orchestration, enabling faster prototyping, reduced complexity, and more time focused on building high-quality experiences.

In addition to the Voice Agent API, organizations seeking broader integrations can leverage Deepgram’s extensive partner ecosystem, including Kore.ai, OneReach.ai, Twilio and others, to access comprehensive conversational AI solutions and services powered by Deepgram APIs.  

Maximum Control and Flexibility

While the Voice Agent API streamlines development, it also gives teams deep control over performance, behavior, and scalability in production. Built on Deepgram’s Enterprise Runtime and full model ownership across the entire voice AI stack, the platform enables model-level optimization at every layer of the interaction loop. This allows for precise tuning of latency, barge-in handling, turn-taking, and domain-specific behavior in ways not possible with disconnected components.

Key capabilities include:

• Flexible Deployment: Run the complete voice stack in cloud, VPC, or on-prem environments to meet enterprise requirements for security, compliance, and performance.
• Runtime-Level Orchestration: Deepgram’s runtime supports mid-session control, real-time prompt updates, model switching, and event-driven signaling to adapt agent behavior dynamically.
• Bring-Your-Own Models: Teams can integrate their own LLMs or TTS systems while retaining Deepgram’s orchestration, streaming pipeline, and real-time responsiveness.

This tightly coordinated design translates directly into measurable performance gains. In recent benchmark testing using the Voice Agent Quality Index (VAQI), Deepgram achieved the highest overall score among all evaluated providers (see Figure 1). VAQI is a composite benchmark that measures the core elements of voice agent quality: latency (how quickly the agent responds), interruption rate (how often it cuts users off), and response coverage (how often it misses valid input).

Deepgram outperformed OpenAI by 6.4% and ElevenLabs by 29.3%, reflecting the advantage of its integrated architecture and model-driven turn-taking. The result is smooth, responsive conversations without missed inputs, premature responses, or unnatural delays.

Cost-Effectiveness at Scale

In addition to control and performance, the Voice Agent API is built for cost efficiency across large-scale deployments. When teams run entirely on Deepgram’s vertically integrated stack, pricing is fully consolidated at a flat rate of $4.50 per hour (see Figure 2). This provides predictable, all-in-one billing that simplifies planning and scales with usage. Deepgram’s vertically integrated runtime also delivers unmatched compute efficiency, optimizing every stage of the speech pipeline to minimize infrastructure costs while maintaining real-time responsiveness.

For teams that bring their own LLM or TTS models, Deepgram offers built-in rate reductions, enabling even lower total cost of ownership for production-scale deployments.

Start Building with the Voice Agent API

Experience how fast and flexible voice agents can be with Deepgram’s unified voice-to-voice API. Explore the API in our interactive playground, review documentation, or integrate in minutes using our SDK. New users receive $200 in free credits, enough to process over 40 hours of real-time voice agent usage. Start building natural, responsive conversations with infrastructure built for real-time performance and enterprise-scale.

Additional Resources:

• Explore the blog for an in-depth breakdown of Voice Agent API’s capabilities
• Watch a fun demo of Deepgram’s voice agent API
• Try Deepgram’s interactive demo
• Get $200 in free credits and try Deepgram for yourself

Deepgram today announced the launch of Nova‑3 Medical, its next‑generation AI-powered speech‑to‑text (STT) model specifically engineered for the healthcare industry. Designed to meet the rigorous demands of clinical environments, Nova‑3 Medical enables developers to build highly accurate, customizable, and secure voice AI products and solutions tailored for healthcare settings. It seamlessly integrates with Deepgram’s enterprise runtime platform—including advanced text-to-speech (TTS) and speech-to-speech (STS) capabilities—providing a comprehensive suite of AI-driven tools that deliver enterprise-grade performance, adaptability, and cost efficiency. From streamlining clinical documentation to revolutionizing therapeutic scribing, Deepgram powers transformative medical transcription applications for industry leaders, driving exceptional outcomes across the healthcare spectrum.

Meeting the Growing Demand for AI-Powered Healthcare Transcription

As healthcare rapidly digitizes—with the widespread adoption of electronic health records, telemedicine, and digital health platforms—the demand for AI-powered transcription has never been greater. Traditional off-the-shelf speech-to-text models often struggle with the complexities of clinical terminology, leading to transcription errors and “hallucinations” that can compromise patient care. With the medical transcription market projected to grow from USD 85.3 billion in 2023 to USD 190.2 billion by 2032, developers building voice-AI applications for healthcare need infrastructure that not only delivers exceptional accuracy and speed but also provides the flexibility to meet diverse regulatory and operational requirements.

Built to meet these demands, Nova-3 Medical leverages advanced machine learning and specialized medical vocabulary training to set a new standard in healthcare transcription. Engineered for real-world clinical environments, the model accurately captures specialized medical terms, acronyms, and clinical jargon—even in challenging far-field audio conditions where providers step away from recording devices such as desktops and tablets. Moreover, it delivers structured transcriptions that seamlessly integrate with clinical workflows and EHR systems, ensuring vital patient data is accurately organized and readily accessible. Its flexible, self‑service customization—featuring Keyterm Prompting for up to 100 key terms—allows developers to tailor the solution to the unique needs of various medical specialties while versatile deployment options, including on‑premises and VPC configurations, ensure enterprise‑grade security and HIPAA compliance.

Benchmarking Nova-3 Medical: Accuracy, Speed, and Efficiency

Nova-3 Medical delivers industry-leading transcription accuracy, optimizing both overall word recognition and critical medical term accuracy for voice-driven healthcare applications.

WER Comparison (see figure 1)

With a median Word Error Rate (WER) of 3.45%, Nova-3 Medical outperforms competing models, achieving a 63.6% reduction in errors compared to the next best competitor. This improvement enhances documentation precision, minimizes manual corrections, and streamlines workflows for healthcare providers.

KER Comparison (see figure 2)

However, medical transcription accuracy isn’t limited to WER—correctly capturing critical medical terms is essential for minimizing patient care risks. Nova-3 Medical achieves a Keyword Error Rate (KER) of 6.79%, marking a 40.35% reduction in errors compared to the next best competitor. This ensures that fewer critical drug names, conditions, and procedures are misrecognized, reducing the chances of transcription errors that could lead to miscommunication, improper documentation, or even patient safety risks.

In addition to transcription accuracy, Nova-3 Medical’s performance excels in real-time applications, where speed and scalability are crucial. Optimized for real-time use, Nova‑3 Medical transcribes speech 5 to 40 times faster than most alternative speech recognition vendors, making it ideal for telemedicine and digital health platforms. Its scalable architecture ensures that as transcription volumes grow, healthcare tech companies can maintain high performance without incurring excessive costs. Starting at $0.0077 per minute of streaming audio, Nova‑3 Medical is more than 2x more affordable than leading cloud providers, reducing operational expenses and enabling companies to reinvest in innovation, accelerate product development, and offer competitive pricing to drive market adoption.

Visit Deepgram at Booth #136 in the AI Pavilion at HIMSS25, March 3-6, 2025, to see Nova-3 Medical in action; and don’t miss these sessions:

Session: From AI Scribes to EHR Automation: How Deepgram Enables Healthtech with Voice AI and Amazon Bedrock

When: Tuesday, March 4, 3:40 PM to 4:00 PM

Where: AI Pavilion, Venetian, Level 2, Hall A

Session: Voice AI Mixer with Deepgram & OneReach.ai

When: Wednesday, March 5, 6:00 PM to 7:30 PM

Where: Venetian, Palazzo Ballroom, Palazzo A

For more information about Nova‑3 Medical and how it is revolutionizing healthcare transcription, please visit www.deepgram.com.

Deepgram has announced a significant technical achievement in speech-to-speech (STS) technology for enterprise use cases. The company has successfully developed a speech-to-speech model that operates without relying on text conversion at any stage, marking a pivotal step toward the development of contextualized end-to-end speech AI systems. This milestone will enable fully natural and responsive voice interactions that preserve nuances, intonation, and emotional tone throughout real-time communication. When fully operationalized, this architecture will be delivered to customers via a simple upgrade from our existing industry-leading architecture. By adopting this technology alongside Deepgram’s full-featured voice AI platform, companies will gain a strategic advantage, positioning themselves to deliver cutting-edge, scalable voice AI solutions that evolve with the market and outpace competitors.

Advancements Over Existing Architectures

Existing speech-to-speech (STS) systems are based on architectures that process speech through sequential stages, such as speech-to-text, text-to-text, and text-to-speech. These architectures have become the standard for production deployments for their modularity and maturity, but eliminating text as an intermediary offers opportunities to improve latency and better preserve emotional and contextual nuances.

Meanwhile, multimodal LLMs like Gemini, GPT-4o, and Llama have evolved beyond text-only capabilities to accept additional inputs such as images, videos, and audio. However, despite these advancements, they struggle to capture the fluidity and nuance of human-like conversation. These models still rely on a turn-based framework, where audio input is tokenized and processed within a textual domain, restricting real-time interactivity and expressiveness.

To advance the frontier of speech AI, Deepgram is setting the stage for end-to-end STS models, which offer a more direct approach by converting speech to speech without relying on text. Recent research on speech-to-speech models, such as Hertz and Moshi, has highlighted the significant challenges in developing models that are robust and reliable enough for enterprise use cases. These difficulties stem from the inherent complexities of modeling conversational speech and the substantial computational resources required. Overcoming these hurdles demands innovations in data collection, model architecture, and training methodologies.

Delivering Speech-to-Speech with Latent Space Embeddings

Deepgram is transforming speech-to-speech modeling with a new architecture that fuses the latent spaces of specialized components, eliminating the need for text conversion between them. By embedding speech directly into a latent space, Deepgram ensures that important characteristics such as intonation, pacing, and situational and emotional context are preserved throughout the entire processing pipeline. What sets Deepgram apart is its approach to fusing the hidden states—the internal representations that capture meaning, context, and structure—of each individual function: Speech-to-Text (STT), Large Language Model (LLM), and Text-to-Speech (TTS). This fusion is the first step toward training a controllable single, true end-to-end speech model, enabling seamless processing while retaining the strengths of each best-in-class component. This breakthrough has significant implications for enterprise applications, facilitating more natural conversations while maintaining the control and reliability businesses require.

This technical advancement builds on Deepgram’s expertise in enterprise speech AI, with over 200,000 developers using its platform, more than 50,000 years of audio processed, and over 1 trillion words transcribed. Key benefits of the new architecture include:

• Optimized latency design for faster, more responsive interactions
• Enhanced naturalness, preserving emotional context and conversational nuances
• Native ability to handle complex, multi-turn conversations
• Unified, end-to-end training across the entire model, creating a more cohesive and inherently adaptive system that fine-tunes its understanding and response generation directly in the audio space

Utilizing Transfer Learning for Cost-Efficient, High-Accuracy Speech-to-Speech

Deepgram’s research in the space is accelerated by its use of transfer learning and best-in-class pre-trained models, allowing it to achieve high accuracy with significantly less training data than traditional methods. Without latent techniques, training a model at the scale needed for speech-to-speech would require over 80 billion hours of audio—more than humanity has ever recorded. However, Deepgram’s latent space embeddings and transfer learning approach achieve superior comprehension while significantly reducing costs, maintaining interpretability, and accelerating enterprise deployment. This efficiency enables Deepgram to deliver scalable, end-to-end speech AI that meets the demands of real-world voice applications.

Empowering Developers with Full Debuggability

One of the requirements in enterprise speech-to-speech modeling is the ability to understand and troubleshoot each step of the process. This is particularly challenging when text conversion between steps isn’t involved, as verifying both the accuracy of the initial perception and the alignment of the spoken output with the intended response is not straightforward. Deepgram recognized this need and addressed it by designing a new architecture that enables debuggability throughout the entire process.

This architecture allows developers to inspect and understand how the system processes spoken dialogue. The design incorporates speech modeling of perception, natural language understanding/generation, and speech production, preserving distinct capabilities during training. Through the ability to decode intermediate representations back to text at specific points, developers can gain insight into what the model perceives, thinks, and generates, ensuring its internal representation aligns with the model output and stays true to the intent of the business user, addressing hallucination concern in scaled business use cases. This capability allows the user to peer into each step throughout generation, helping refine models, improve performance, and deliver more accurate, lifelike, and reliable speech-to-speech solutions.

Beyond Speech-to-Speech – A Complete, Enterprise-Ready Voice AI Stack

While building an advanced speech-to-speech (STS) model is a major technical achievement, enterprises need more than just a model—they need a complete, scalable platform that ensures seamless deployment, adaptability, and cost efficiency. Deepgram delivers not just cutting-edge STS technology, but an enterprise-ready infrastructure designed for real-world applications.

Seamless Integration & Continuous Improvement – Once Deepgram’s end-to-end STS model moves to production, businesses will be able to adopt this breakthrough directly through our developer-friendly voice agent API from within the current Deepgram platform. Through continued innovation, enterprises will benefit from the latest advancements, ensuring seamless integration and a future-proof platform for their voice AI applications.

Enterprise-Grade Performance & Cost Efficiency – Built for low customer COGS, our platform enables enterprises to deploy high-performance voice AI without excessive costs. This ensures scalability, whether for customer service automation, real-time voice agents, or multilingual applications.

Full-Featured Platform and High-Performance Runtime – Deepgram’s platform includes powerful capabilities such as:

• Adaptability – Dynamically fine-tune models for specific industry language, ensuring high accuracy across diverse applications without needing constant retraining.
• Automation – Streamline transcription, model updates, and data processing, reducing overhead and accelerating deployment.
• Synthetic data generation – Generate synthetic voice data to improve model training, even with limited real-world data, enhancing accuracy for niche use cases.
• Data curation – Clean, manage, and organize training data to ensure high-quality, relevant input, improving model performance.
• Model hot-swapping – Seamlessly switch between different models to optimize performance for specific tasks.
• Integrations – Effortlessly integrate Deepgram’s voice AI with cloud platforms, enterprise systems, and third-party applications, embedding it within existing workflows.

With Deepgram, enterprises don’t just get speech-to-speech—they get the most advanced, enterprise-ready voice AI platform, designed for real-world deployment and long-term innovation.

For more information about Deepgram’s novel approach for speech-to-speech, read the technical brief. To learn more about Deepgram’s suite of voice AI infrastructure, visit www.deepgram.com.

 Deepgram, the leading voice AI platform for enterprise use cases, today announced the launch of Nova-3, its most advanced speech-to-text (STT) model to date. Nova-3 pushes the boundaries of AI-driven transcription, offering unmatched accuracy in challenging audio environments while offering flexible, self-service customization to tailor results for industry-specific needs. Trusted by industry leaders like Twilioand Jack in the Box, Deepgram’s infrastructure also includes powerful text-to-speech (TTS) and full speech-to-speech (STS) capabilities, offering a comprehensive suite of cloud or self-hosted APIs for seamless voice AI integration. Its full-featured platform and high-performance runtime include powerful automation and data capabilities – such as synthetic data generation and model curation – along with model hot-swapping and robust integrations, empowering developers to efficiently build and scale voice-enabled applications. With over 450 enterprise customers, Deepgram is powering the fast-growing enterprise voice AI market.

Nova-3 Expands Voice AI for a Broader Range of Enterprise Use Cases

Leapfrogging the success of its predecessor, Nova-3 is engineered for real-time use cases, delivering unparalleled accuracy and performance in dynamic environments where traditional solutions often fall short. Unlike generalized models that lack domain-specific precision, Nova-3 leverages an advanced latent space architecture to encode complex speech patterns into a highly efficient representation. This enables superior transcription accuracy, even in noisy or specialized settings, driving improved productivity, customer satisfaction, and cost efficiency. With its expanded capabilities, Nova-3 now delivers enhanced accuracy for real-world enterprise challenges such as:

• Adverse acoustic conditions – Accurately transcribes speech in distant, noisy, and multi-speaker scenarios, making it ideal for air traffic control, drive-thrus, and call centers.
• Real-Time Multilingual Support – Enables real-time transcription across multiple languages – the first model of its kind to do so – making it ideal for emergency response, global customer service, and multilingual operations.
• Industry-Specific Accuracy – Recognizes domain-specific terminology for specialized fields like medical and legal transcription.
• Precision Data Handling – Ensures accurate numeric recognition for retail, banking, and finance while supporting real-time redaction of sensitive information for compliance and data privacy.

Personalize Voice AI with Self-Service Customization

Nova-3 is the industry’s first voice AI model to enable self-serve customization, allowing users to fine-tune the model for specialized domains without requiring deep expertise in machine learning. Many conventional models require expensive and time-consuming expert-led customization, delaying deployment and increasing costs. With the addition of Keyterm Prompting, developers can instantly improve transcription accuracy by optimizing up to 100 key phrases without waiting for extensive model retraining or customization cycles. This flexibility accelerates deployment, enhances accuracy, and reduces costs—allowing businesses to rapidly unlock value from their voice AI solutions.

Benchmarking Excellence – Deepgram Extends Its Lead

Nova-3 continues to set a new standard for transcription accuracy, significantly widening the gap between itself and competing voice AI providers. Nova-3 outperforms competitors in both batch and streaming use cases, with consistently lower Word Error Rates (WER) that drive superior performance in real-world audio environments, including multilingual scenarios.

Batch WER Comparison

Nova-3 achieves a WER of 5.26%, extending its lead over the next-best competitor by 47.4% (10% WER). This reduced error rate translates to more accurate transcriptions for industries that require high precision, such as healthcare, legal, and finance.

Streaming WER Comparison

In streaming WER, Nova-3 leads with a WER of 6.84%, extending its advantage over the next-best competitor by 54.2% (14.92% WER). This improved accuracy ensures real-time, reliable transcription for applications such as call centers and virtual assistants, enhancing overall customer experience.

Multilingual Performance

In multilingual testing, Nova-3 outperforms OpenAI’s Whisper across seven languages, delivering up to 8:1 preference ratios in some languages. Nova-3’s advanced real-time multilingual conversation transcription empowers enterprises to scale globally, delivering reliable, accurate results across multiple languages and enhancing international customer engagement.

These benchmark results underscore Deepgram’s continued lead in transcription accuracy, driving superior outcomes for businesses that rely on speech-to-text and voice AI technologies.

Nova-3 Marks a Major Advancement

Nova-3 represents a breakthrough in AI-driven speech-to-text technology, cementing Deepgram’s position at the forefront of voice AI innovation and empowering businesses and developers to build the next generation of enterprise voice AI applications. Deepgram’s focus on continuous model and platform improvements ensures users always have access to the latest advancements, maximizing long-term value. Built with low customer COGS, the platform offers cost-efficiency and seamless updates, helping businesses stay competitive and future-proofed as they scale.

For more information about Nova-3 and Deepgram’s suite of voice AI infrastructure, please visit www.deepgram.com.

Deepgram, the leading voice AI platform for developers building speech-to-text (STT), text-to-speech (TTS), and full speech-to-speech (STS) offerings, today announced record business growth and technical milestones achieved in the past year. Today, over 200,000 developers build with Deepgram’s voice-native foundational models, choosing Deepgram due to its unmatched accuracy, low latency, and pricing, as well as the flexibility for all voice-native AI models to be accessed through cloud APIs or self-hosted / on-premises APIs. Organizations that build on Deepgram’s infrastructure for STT, TTS, and AI Voice Agents include technology ISVs building voice products or platforms, co-sell partners working with large enterprises, and enterprises solving internal use cases. 

Looking forward to 2025, Deepgram will continue to innovate to extend its unique value proposition of offering the highest accuracy and lowest COGS at scale and highest model adaptability, and lowest latency. Through continued innovation, Deepgram expects to end 2025 as the industry’s only end-to-end speech-to-speech solution built to solve the four critical challenges of enterprise-ready voice AI:

1. Accuracy / audio perception: Enterprise use cases require high recognition, understanding, and generation of specialized vocabulary in often challenging audio conditions. Deepgram solves this through novel, non-lossy compressions of these spaces for rapid processing paired with generation, training, and evaluation on synthetic data that precisely matches Deepgram customers’ real-world conditions.
   
2. COGS at scale: Deepgram customers need to profitably build and scale voice AI solutions. Deepgram delivers this through its unique latent audio model with extreme compression combined with deep expertise in high-performance computing.
   
3. Latency: Real-time conversation requires near-instantaneous responses. Deepgram achieves this using streaming state space model architectures, optimized specifically for the underlying hardware to deliver minimal processing delays.
   
4. Context: Effective conversations are deeply contextualized. Deepgram will pass the speech Turing test thanks to its ability to train on vast bodies of data that thoroughly represent its customers’ use cases and pass that context through the entire system and interaction.

Additional Resources:

●      Read about Deepgram’s groundbreaking voice agent API

●      Watch a fun demo of Deepgram’s voice agent API

●      Try Deepgram’s interactive demo

●      Get $200 in free credits and try Deepgram for yourself