Voice Agents Research Brief — The Technical Stack Behind Realtime Conversation

Twelve months ago, full-duplex voice — a model that listens while it speaks — was a research architecture. Today it is the default in ChatGPT Voice. The intervening year moved realtime, agentic voice from prototype to production at a pace unusual even by current standards:

• OpenAI shipped GPT-Realtime-2 (May 2026: frontier-class reasoning and tool calling in a realtime voice model, controllable tone, streaming translation and transcription variants), then GPT-Live (July 2026): full-duplex models that process input and generate output continuously, handle interruptions and backchannels, and delegate hard reasoning to larger models mid-conversation. In late July, OpenAI Presence launched as a managed platform for governed agent deployment: policies, permissions, simulation-based evals, monitoring, human escalation.
• xAI released Grok Voice Think Fast 2.0 (July 2026): a speech-to-speech model emphasizing transcription robustness, native tool use, sub-second first audio, and a no-code voice agent builder with telephony integration; xAI reports leading results on agentic voice benchmarks such as τ-Voice.
• Google (Gemini Live) extended realtime voice with camera and screen-sharing context; Amazon continued the generative rebuild of Alexa (Alexa+) around task completion.
• Inworld moved Realtime TTS-2 to general availability: 100+ languages, natural-language voice steering with inline non-verbal events, voice design from a text description, cloning from seconds of audio, and roughly 200ms responses — served through a Realtime API that runs the full speech-to-speech loop over a single connection. The TTS-2 research preview holds the top position on the Artificial Analysis Speech Arena's realtime subset, the board restricted to models fast enough for live agents.
• Enterprise platforms proliferated: ElevenLabs' agent platform, PolyAI, Retell, and vertical entrants shipped monitoring, QA, failover, and compliance tooling — the markers of a category moving from demos to production.

Three claims summarize where this leaves the field, and they structure this primer:

1. Synthesis quality is commoditizing. Frontier text-to-speech systems are difficult to distinguish from human speech on short clips. Differentiation has moved to steerability, latency, language coverage, and behavior inside live conversation.
2. The open problems concentrate in conversational dynamics. Turn-taking, interruption handling, full-duplex behavior, and reasoning under realtime constraints are where research activity and product differentiation now sit.
3. Economics decide viability. Serving thousands of concurrent, latency-bound audio sessions is a distinct systems problem from batch LLM inference; cost per conversation-minute determines which applications can exist.

On August 1, AGI House is convening 100+ founders, engineers, and researchers in Hillsborough for Voice Agent Build Day — a full day of building on exactly this frontier, with speakers including Inworld CEO Kylan Gibbs and Artem Yerofieiev, Senior Principal Software Engineer at Tenstorrent, who leads work on ML frameworks and AI compilers. Builders get direct access to Inworld's Realtime API: text-to-speech (TTS — text plus style controls in, audio out), speech-to-text (STT — audio in, transcript and paralinguistic signals out), and a conversation-optimized LLM, callable individually or as a single end-to-end pipeline, with hands-on support from Inworld's engineering team. Space is limited; apply to attend.

This primer walks the stack layer by layer — core concepts, the research works defining each layer's frontier (linked inline), and open problems — and closes with project ideas scoped to a single build day.

Architectures: cascaded, speech-to-speech, full-duplex

Three architectural patterns organize the space. Terminology introduced here recurs throughout.

Cascaded pipelines chain three models with text in the middle: STT, then an LLM, then TTS. Text as the intermediate representation provides transcripts (compliance, logging, analytics), composability (swap any stage), and access to frontier LLM reasoning and tool use. Its costs: cumulative latency across stages, and information loss — prosody (rhythm, stress, intonation), hesitation, laughter, and affect are destroyed at the text boundary and must be re-synthesized on output. This remains the dominant production architecture.

Speech-to-speech (S2S) models map audio to audio in a single model, preserving paralinguistic information end to end. GPT-Realtime-2 and Grok Voice are the reference proprietary systems; both now support native tool calling, addressing an early S2S weakness. The unresolved cost is the loss of a native text control plane: no transcript for audit, harder observability, weaker debuggability.

Full-duplex systems listen while speaking — modeling both conversation streams simultaneously rather than enforcing strict turns. This enables overlap, backchannels ("mm-hm"), and graceful recovery from barge-in: the user talking over the agent, which must stop, flush buffered audio, and preserve context. Moshi (Kyutai, 2024) established the architecture; GPT-Live (2026) brought it to consumer default; ByteDance's Doubao deployment reported substantial reductions in false responses and talk-overs versus its half-duplex predecessor at production scale. For a builder-oriented taxonomy of what "full-duplex" actually means across systems — where duplex decisions are made, and which interaction types are genuinely supported — see the Survey of Full-Duplex Spoken Dialogue Systems (2026), the single best pre-read for this event.

Two latency concepts matter throughout: TTFT/TTFA (time to first token / first audio) and endpointing — deciding the user has finished speaking. Classical pipelines use a voice activity detector (VAD), effectively a silence timer, which adds several hundred milliseconds of dead air before the LLM starts; newer conversational STT (e.g., Deepgram Flux) integrates semantic end-of-turn detection into the recognition model itself. Endpointing is the least visible and most consequential latency term in the stack.

These architectures frame the two questions the event is organized around, and they are distinct. The architectural question: can full-duplex conversational dynamics be obtained without giving up text-level control? Production answers exist on both sides — GPT-Live delegates complex reasoning to frontier text models mid-conversation, while the cascaded camp's answer is to engineer the pipeline down to S2S-class latency and keep the transcript (Inworld's Realtime API is the representative implementation; research approaches are covered in the research-frontier section below). The systems question: whichever architecture wins, can it be served for thousands of simultaneous users without latency or cost collapsing? The compute and reasoning layers below argue this is now the binding constraint.

Layer 1 — Compute and inference infrastructure

Scope: The hardware and serving systems that run speech models; the layer where cost per conversation-minute is determined.

Why voice is a distinct systems problem:

• Streaming: audio must be produced at a steady realtime cadence; any stall is audible, unlike deferred text rendering.
• Latency-bound: TTFA is the product; throughput-optimized batch serving conflicts directly with it.
• Concurrency-bound: consumer voice means thousands of simultaneously open sessions, not queued jobs.
• Tail latency dominates perception: a low median with a high P95 fails one call in twenty. Public benchmarks rarely report latency under realistic concurrent load.

Recent developments:

• Streaming-native serving architectures and quantization-aware training (holding quality at reduced precision) from vertically integrated speech vendors; parts of these training stacks (codec through SpeechLM fine-tuning) have been open-sourced.
• Small open models — Kokoro (TTS), Parakeet (STT), Moshi derivatives (S2S) — make self-hosting and on-device deployment viable, shifting privacy and cost calculus.
• Alternative inference silicon reached general availability: Tenstorrent's Galaxy Blackhole systems (Tensix tensor cores, RISC-V, Ethernet scale-out, open software stack) target inference-heavy, latency-sensitive workloads at reported cost advantages over GPU deployments. Since LLM inference is now the largest per-minute cost line in a voice agent (see the reasoning layer), open-stack hardware alternatives bear directly on application viability.

Landscape:

• Hyperscalers: AWS, GCP, Azure — capacity plus first-party speech APIs.
• NVIDIA: the incumbent silicon plus an active open research program (Parakeet, Canary, PersonaPlex).
• Alternative silicon: Tenstorrent and other open-stack entrants.
• Managed inference platforms: Baseten, Together, Fireworks, Modal; several host open speech models as endpoints.
• Vertically integrated speech vendors: Inworld, ElevenLabs, Deepgram, Cartesia — operate their own fleets, which is the source of their pricing structures.

Open problems: Honest latency-under-concurrency benchmarks; the further cost reduction required for free-tier consumer voice; hybrid edge/cloud serving.

Layer 2a — Text-to-speech

Scope: Synthesis: text plus style controls in, audio out.

State of the art:

• Quality at the frontier is near-parity with human speech on short clips; competition has moved to steerability, latency, language coverage, cloning, and price.
• Natural-language steering is replacing SSML markup: tone directed in plain words ([calm, reassuring]) with inline non-verbal events ([laugh], [sigh]). Inworld's Realtime TTS-2 (the model available at this event: 100+ languages, roughly 200ms responses) and OpenAI's instructable TTS models exemplify the pattern.
• Zero-shot cloning from seconds of reference audio is now table stakes — with direct consequences for the safety layer.
• Voice design from text description (specify a voice in prose rather than selecting from a catalog) is emerging as a distinct capability.
• The market historically split into expressive offline models, realtime agent models, and high-volume budget models; the current frontier contest is holding top quality at realtime latency and consumer-scale cost simultaneously.

Landscape:

• Specialists and labs: Inworld, ElevenLabs, Cartesia, Hume (emotion-centric), Rime (telephony register), Speechify, PlayHT, MiniMax, Fish Audio, Sesame (open conversational speech model research), LMNT.
• Big tech: Google, OpenAI, Amazon, Microsoft, Alibaba, ByteDance.
• Open weights: Kokoro, XTTS-v2, Chatterbox, Orpheus, OpenAudio — the self-hosting tier.

Open problems: Prosody consistency over twenty-minute conversations rather than ten-second clips; mid-sentence language switching without voice identity drift; entity pronunciation (names, codes, brands); the reliability gap between vendor benchmarks and production behavior.

Layer 2b — Speech-to-text

Scope: Rcognition — increasingly, conversational recognition: transcription plus turn-taking inference plus paralinguistic signals.

State of the art:

• Integrated endpointing. Purpose-built conversational STT (Deepgram Flux) folds semantic end-of-turn detection into recognition, eliminating the separate VAD stage and its dead air. OpenAI's GPT-Live-Transcribe and streaming Whisper variants target the same regime.
• Speech-language-model architectures. AssemblyAI's Universal-3 line applies LLM-style architectures to recognition, with natural-language keyterm prompting for domain vocabulary.
• Predictive streaming. ElevenLabs' Scribe v2 Realtime generates transcript slightly ahead of the audio.
• Paralinguistics. Reading emotion, accent, and intent from audio rather than words (Inworld STT's focus) — the recognition-side prerequisite for applications that respond to how a user sounds.
• Entity accuracy over headline word error rate (WER). Aggregate WER conceals the errors that block deployment: phone numbers, confirmation codes, proper names, especially over compressed telephony audio.

Landscape:

• Voice-agent leaders: Deepgram, AssemblyAI, ElevenLabs.
• Specialists: Speechmatics (accent-heavy and code-switched audio), Gladia.
• Open: NVIDIA Parakeet and Canary; Whisper as the default baseline.
• Hosted big tech: Google Chirp, Azure Speech, Amazon Transcribe — enterprise-compliance defaults.
• Stack-completers: Inworld STT, Cartesia Ink.

Open problems: Real-world audio (telephony codecs, cross-talk, noise); live diarization; code-switching; fully semantic endpointing; vendor-versus-independent benchmark divergence.

Layer 2c — Speech-to-speech and full-duplex: the research frontier

Scope: The architectural question from the architectures section, as an active research program: can the text bottleneck be removed, and at what cost to control?

Production systems:GPT-Realtime-2 and GPT-Live (OpenAI), Grok Voice Think Fast 2.0 (xAI), Gemini Live (Google), Nova Sonic (Amazon), Doubao full-duplex (ByteDance — the largest published production evidence for full-duplex value).

Research lineage:

• Foundations: Moshi (Kyutai, 2024) — simultaneous modeling of both conversation streams; the architectural ancestor. PersonaPlex (NVIDIA, 2026) — a 7B full-duplex model on the Moshi architecture, MIT-licensed code, adding voice cloning and role control via hybrid prompts; strong conversational dynamics among open systems. Sesame's CSM (open release); SyncLLM (EMNLP 2024); OmniFlatten (ACL 2025); GLM-4-Voice; Voila.
• Retaining text intelligence: Freeze-Omni (S2S wrapped around a frozen LLM) — an explicit hybrid; GPT-Live's delegation to frontier reasoning models is the production analog.
• Reasoning under realtime constraints — the most active current thread: Chronological Thinking (strictly causal reasoning that unfolds in time with the conversation); Mind-Paced Speaking (dual-process: a "formulation brain" reasons while an "articulation brain" speaks); The Silent Thought (latent, unspoken reasoning); STITCH (chunked reasoning interleaved with speech output). Cascaded-side analogs use speculative generation against partial transcripts while the user is still speaking (LTS-VoiceAgent; RelayS2S).
• Tool use from speech: StreamRAG (retrieval queries predicted in parallel with incoming speech); SHANKS (unspoken chain-of-thought while listening, enabling mid-turn tool execution and informed interruption); Ultravox (open-weight, native tool use, no separate ASR stage).
• Duplex failure modes: activation steering against state inertia (models stuck speaking or listening); SoulX-Duplug (plug-and-play streaming state prediction); RL methods decoupling conversational dynamics from content.
• Multimodal: AV-Dialog (audio-visual dialogue — the agent sees the user), converging with the avatar layer.

Open problems: S2S debuggability (audio in, audio out, no span tree — tooling has not caught up); reasoning capacity at the roughly 7B scale current open full-duplex models occupy; scarce reproducible evaluation of proprietary tool-using voice agents; compliance without transcripts.

Layer 3 — Reasoning: LLMs, routing, memory, tools

Scope: The intelligence in the loop: model selection, per-turn routing, persistent memory, and tool/data access.

State of the art:

• The LLM is now the latency bottleneck. TTS synthesis latency has fallen far enough that LLM TTFT dominates the voice-to-voice budget. The practical fix for a slow agent is a faster model tier, a shorter prompt, or better endpointing — rarely an architecture change.
• Voice imposes a spoken register: short outputs, interruption-safe generation (the model may be cut off at any token), personality consistency, prose written for the ear. Most frontier models are not tuned for this by default; conversation-optimized hosted variants exist (Inworld's dialogue-tuned models; custom models inside companion companies).
• Per-turn routing. Different turns have different intelligence requirements; routers exposing many models behind one endpoint (Inworld's LLM Router, OpenRouter) allow swapping models per turn on cost and latency.
• Memory is the retention mechanism for companion-class products: mem0 (open-source memory layer), Letta (MemGPT lineage), Zep (temporal knowledge graphs).
• Live data access. Agents that answer questions about the current world need realtime retrieval mid-conversation: search APIs and web-data infrastructure (e.g., Bright Data-class platforms for structured web access) become pipeline dependencies, and speculative retrieval during user speech (StreamRAG) is the latency-hiding technique.

Open problems: TTFT under conversational constraints; moving thinking-while-listening from papers to products; personality drift over long horizons; the LLM as the dominant per-minute cost line.

Layer 4 — Orchestration, platforms, and transport

Scope: The runtime loop: capture microphone or telephony audio, manage WebRTC/SIP sessions, coordinate streaming across the STT, LLM, and TTS stages, detect barge-in, handle retries and failover.

State of the art:

• Two camps: open frameworks (you own the loop, the barge-in logic, and the on-call) versus managed platforms (rented loop; platform fees plus provider passthrough).
• Model vendors are moving up-stack: ElevenLabs, Deepgram, AssemblyAI, OpenAI (Realtime API and now Presence), and Inworld (Realtime API — the full speech-in to speech-out loop, including interruption handling and cross-turn context, behind a single connection) all bundle orchestration. Independent orchestrators remain model-neutral.
• No-code builders (Grok's Voice Agent Builder; comparable offerings across managed platforms) compress agent definition to natural-language flow descriptions plus tools and guardrails.

Landscape:

• Open frameworks: LiveKit Agents (open WebRTC infrastructure plus agent framework; adapters for every major STT/TTS; transport under several major consumer voice products), Pipecat (strong plugin ecosystem; publishes an open STT benchmark), Layercode, NLX.
• Managed platforms: Vapi, Retell (QA tooling, healthcare focus), Bland; OpenAI Presence at the governed-enterprise end.
• Transport/telephony: Twilio, Telnyx, Voximplant.
• Applied enterprise agents: Sierra, Decagon, Elise AI, PolyAI, Parloa, Lorikeet.

Open problems: Barge-in remains the hardest engineering problem in the loop (detect, stop, flush, recover — within roughly 200ms); mid-call vendor failover; telephony audio (8kHz codecs undo high-fidelity TTS); pricing opacity across stacked vendors; whether this layer stays independent or is absorbed into model vendors' realtime APIs.

Layer 5 — Evaluation tooling, simulation, and observability

Scope: Commercial tooling that answers whether an agent works: pre-launch simulation, CI regression, production tracing. (Academic benchmarks are covered in the benchmarks section; the two are converging into one discipline from opposite directions.)

State of the art:

• The category's founding distinction: observability is not evaluation. Observability reports that the agent responded in 450ms; evaluation reports whether the response was correct, on-policy, and goal-advancing.
• Voice adds failure modes text agents lack — accents, noise, interruptions, telephony compression, entity transcription — motivating a dedicated tooling category.
• The standard workflow: synthetic callers with configurable accents, noise, and interruption behavior (simulation); automatic re-testing on every prompt change (CI); live-call tracing and failure mining (production). Managed platforms increasingly bundle this (Retell's QA suite; Presence's simulation-based evals).

Landscape:

• Simulation/CI-first: Coval (methodology from autonomous-vehicle testing; also publishes independent provider benchmarks), Hamming (production replay into repeatable tests; load testing; broad accent coverage), Cekura (self-serve QA loop; failed calls feed future test runs).
• Production replay: Roark.
• General eval platforms extended to audio: Braintrust, Maxim.
• Enterprise incumbent: Cyara.
• Integrations are provider-aware across Vapi, Retell, LiveKit, and Pipecat — this layer composes directly with the orchestration layer.

Open problems: No standard metric suite; the realism gap between simulated callers and human conversational chaos; scoring S2S agents with no text to score; affordable load testing at production concurrency.

Layer 6 — Avatars and embodiment

Scope: Realtime rendered faces on voice agents. Architecturally downstream of TTS: consumes the audio stream, renders synchronized video.

State of the art:

• Avatar-as-plugin. LiveKit Agents ships an avatar-plugin interface with many providers behind it: build the voice agent once, hand audio to an avatar session, swap renderers without touching agent code.
• Latency convergence. Leading realtime avatar providers report response latencies approaching voice-only agents.
• Perception — the avatar seeing the user — is the newest sub-frontier, converging with AV-Dialog research.

Landscape:

• Realtime, developer-first: Tavus (rendering, perception, and turn-taking as a three-model API stack), Anam, HeyGen (consumer/marketing scale plus a realtime offering), Hedra, and a low-latency startup tier (Simli, Beyond Presence, bitHuman, LemonSlice) in LiveKit's catalog. D-ID for photo-to-video.
• Enterprise/async: UneeQ, Synthesia.

Open problems: Render cost (approximately one GPU per quality stream); lip-sync under interruption; uncanny-valley trust effects; identity consistency across sessions; young, vendor-contested benchmarks.

Layer 7 — Trust, safety, and compliance

Scope: Cross-cutting: the same cloning capabilities that enable products enable fraud, and legitimate agents are now human-sounding enough to require disclosure.

State of the art:

• Voice-phishing volumes have grown sharply since cloning commoditized; the canonical case is the Arup incident (deepfaked executives on a video call authorizing fraudulent transfers). Attacks are short and target payment approvals and credential resets.
• Regulation is now operative. EU AI Act Article 50 takes effect August 2, 2026 — the day after this event: mandatory disclosure when a human interacts with an AI system, and machine-readable marking of synthetic audio. In the US: the FCC ruled AI-generated voice calls illegal under the TCPA (2024); most states have deepfake statutes, with disclosure-model laws surviving First Amendment challenge while broad prohibitions are repeatedly enjoined; Tennessee's ELVIS Act covers voice likeness; the federal NO FAKES Act remains proposed.
• Compliance is becoming a purchasing criterion, favoring infrastructure with zero-data-retention and on-premises options.

Landscape: Resemble (cloning vendor pivoted to watermarking and detection), Pindrop (call-center voice fraud and authentication), Reality Defender (multimodal deepfake detection), Hiya and Truecaller (consumer screening); vendor-side controls include cloning consent verification and voice-similarity blocklists.

Open problems: Watermark robustness under re-encoding; the detection arms race; consent provenance at scale; disclosure UX — announcing "I am an AI" without degrading the experience is an unsolved design problem.

Evaluation and benchmarks

Voice evaluation decomposes into at least three distinct questions, frequently conflated: human preference for the voice (arena Elo), recognition and understanding on real conversations, and latency under production load. Systems that excel on one routinely fail on another; production failures concentrate in the gaps.

TTS:

• Artificial Analysis Speech Arena — the industry-standard blind Elo arena. Two internal distinctions matter: provider-voices versus controlled-voices boards, and the general board versus the Realtime subset (models fast enough for live agents). Rankings differ meaningfully across boards; any leaderboard claim should specify which.
• TTS Arena 2 (Hugging Face) — the open community arena.
• EmergentTTS-Eval (NeurIPS 2025) — the research critique of short-clip preference arenas: hard prosody, paralinguistics, foreign words, complex syntax, model-as-judge scoring.

STT:

• HF Open ASR Leaderboard (open models); AA-WER and its AgentTalk subset (hosted models).
• Pipecat benchmark — real agent conversations; produces materially different rankings than vendor self-tests.
• Coval independent latency runs — documenting wide TTFT spread across streaming providers.

Conversational dynamics — the benchmark stack that marks where the frontier moved:

• The Full-Duplex-Bench series: v1 (pauses, backchannels, turn-taking, interruptions), v1.5 (overlap handling), v2 (multi-turn evaluation with an automated real-time examiner), and v3 (multi-step tool use under real human disfluency — notable for evaluating GPT-Realtime, Gemini Live, Grok, Ultravox, and a cascaded baseline under one protocol).
• FD-Bench (Interspeech 2025); Talking Turns (ICLR 2025); τ-Voice (agentic voice tasks on realistic domains); EVA-Bench; MTR-DuplexBench; the ICASSP 2026 HumDial challenge, which contributes a dual-channel dataset of real human conversations with interruptions and overlap.

Avatars: Benchmarks exist but are young and vendor-contested; treat rankings accordingly.

Layer 8 — Applications

Scope: Where voice is monetized. Analyst forecasts project rapid multi-year growth across segments, with customer service driving near-term revenue and companions driving usage volume. The recurring design principle: winning products exploit realtime dynamics — responding to how the user sounds, not only what they say.

• Companions and social AI — the heaviest TTS consumers: long sessions, cross-session memory, emotional prosody, personality consistency. Character.AI, Replika, Talkie, Nomi, Kindroid, Tolan, Status; big-lab entries via Grok companions. Challenges: retention past novelty, dependency and minor-safety scrutiny, free-tier economics.
• Tutoring and education — cross-lingual voice consistency (the same voice switching languages mid-sentence), pronunciation assessment as an STT scoring problem, pacing, child-safety constraints. Duolingo, Speak, TalkPal, Praktika, ELSA, Khanmigo; early-literacy STT (Amira, Ello).
• Customer service and phone agents — telephony, tool calls into CRMs and booking systems, compliance transcripts (the strongest case for cascaded architectures). Sierra, Decagon, Elise AI, PolyAI, Parloa, Lorikeet, plus the SMB ecosystems on Vapi, Retell, and Bland. Live issues: hallucination adjacent to real transactions; escalation design; Article 50 disclosure.
• Wellness and coaching — the highest-stakes emotion detection; guardrails and crisis escalation; long-horizon memory. Slingshot AI, Wysa, Rosebud, Pi, Hume as the empathic-voice supplier. The category's cautionary tale is Woebot's consumer shutdown: regulatory positioning is decisive here.
• Healthcare operations — clinical intake, scheduling, follow-up calls; realtime conversational systems for clinical settings are an active research area. Entity accuracy and compliance transcripts are hard requirements.
• Creative tools and accessibility — long-form prosody, multi-speaker, dubbing timing; the strongest voice-rights pressure. ElevenLabs, Speechify, Descript, NotebookLM's audio formats, Wondercraft; professional dubbing (Papercup, Deepdub, Camb.ai); Apple Personal Voice for voice banking.
• Games and interactive characters — runtime NPC dialogue, character consistency, in-loop latency, cost per player-hour. Convai, Volley, NVIDIA ACE, studio experiments.
• Voice as a developer interface — an emerging category: conversational control of coding and research agents, where specification happens through dialogue rather than typed prompts. Little is established here; see project ideas below.

Synthesis: where the open problems concentrate

1. Dynamics. Semantic endpointing, barge-in recovery, full-duplex behavior, and reasoning-while-listening — the gap between sounding human and conversing like one.
2. Economics. LLM TTFT and cost per conversation-minute, now as much a serving and silicon problem as a modeling problem.
3. Trust. Evaluation that predicts production behavior; transcripts or their equivalent for compliance; disclosure and anti-fraud mechanisms that survive contact with real users.

Build Day project ideas

Scoped to one day; each lists the stack and experience assumed, plus an extension path. Every project is buildable on the API access provided at the event together with open-source frameworks and commercial tools that are cheap and quick to start with — nothing here depends on expensive or hard-to-access infrastructure, and nothing requires model training. The day's leverage is in the loop, the prompting, and the measurement.

Conversational dynamics and systems

1. Semantic endpointing head-to-head. Build the same agent twice — VAD-based endpointing versus integrated end-of-turn detection — and measure perceived latency and false-cut rate with live users in the room. Stack: Pipecat or LiveKit (both open source), one conversational STT provider, Python; intermediate. Extension: train a lightweight end-of-turn classifier on partial transcripts and beat both.
2. Thinking-while-listening. Implement the speculative pattern in a cascade: a background process generates against partial transcripts; commit or discard at endpoint. Measure TTFT savings. Stack: any orchestration framework plus a fast LLM endpoint; comfort with async Python; intermediate-advanced. Extension: speculative retrieval prefetch (StreamRAG pattern) against live web data.
3. Barge-in gauntlet. A synthetic adversarial caller — interruptions, backchannels, overlapping speech, mid-word topic changes — scoring any agent on time-to-stop, buffer-flush correctness, and context retention. Stack: two agent sessions wired together (TTS output into the target's input), audio buffer handling; intermediate. Extension: align metrics with Full-Duplex-Bench and publish results across the platforms present.
4. The latency waterfall. Per-turn instrumentation of a cascaded agent: capture, endpoint decision, STT final, LLM first token, first audio. Render it live. Stack: any framework plus timestamp logging and a small dashboard; beginner-friendly, high demo value. Extension: the same waterfall under simulated concurrency, reporting median versus tail.

Voice as a developer interface

5. Spec-by-conversation. A voice front-end for a coding agent: the user describes what they want; the agent interrogates ambiguities aloud, restates the spec, and only then dispatches an autonomous coding run — replacing prompt-typing with negotiated dialogue. Stack: realtime voice API plus any coding-agent CLI with a free tier; mostly integration work; intermediate. Extension: the agent interrupts with clarifying questions mid-build when it hits a genuine fork.
6. Ambient pair programmer. A full-duplex agent that watches a repository or agent-coding session and murmurs status, surfaces failures aloud, and accepts spoken redirection ("skip that test, try the other approach") without the developer leaving their editor. Stack: voice loop plus file-watcher or CI webhooks (all open source); the hard part is deciding when speaking helps; advanced. Extension: spoken arbitration between two coding agents proposing different implementations.
7. Research interlocutor. A voice agent over a paper corpus or live web search that conducts a literature discussion rather than a Q&A: it pushes back, asks what you already believe, and adjusts retrieval to the conversation's trajectory. Stack: voice loop plus RAG over arXiv (open access) or a web-search API; standard RAG experience suffices; intermediate. Extension: it drafts the related-work section from the transcript.

Coaching and realtime feedback

8. Live conversation coach. Dual-channel design: the agent listens to a real conversation (mock interview, sales call, negotiation) and coaches privately in one ear — pacing, filler words, tone — without joining. The design problem is when speaking helps versus distracts. Stack: STT with diarization or two input channels, plus a low-latency TTS side-channel; intermediate-advanced. Extension: post-call prosody analytics from paralinguistic STT signals.
9. Oral examiner. A Socratic tutor that probes understanding by voice — follow-up questions calibrated to hesitation and confidence in the user's speech, not just answer content. Stack: single realtime API plus prompt engineering; beginner-friendly. Extension: spaced-repetition scheduling driven by detected uncertainty.
10. Sparring partner. A debate or negotiation opponent with a persistent, steerable personality that escalates pressure realistically — an application of natural-language voice steering where the dynamics (interrupting, talking over you) are the product. Stack: realtime API plus steering tags; beginner-friendly, strong stage demo. Extension: two personas, and the user must win over a panel.

Verticals

11. Clinical intake agent. Structured medical intake by phone: entity-critical (medications, dosages, dates), transcript-dependent, escalation-aware. A deliberately cascaded design; the demo is the accuracy report on hard entities over telephony-quality audio. Stack: a managed platform free tier (Vapi or Retell) or an open framework, plus a structured-output LLM prompt; beginner-to-intermediate. Extension: a red-team caller with accents, noise, and interruptions.
12. Hands-busy assistant. A voice agent for work where hands and eyes are occupied — cooking, repair, lab protocols, field service — with tool use (timers, lookups, step tracking) and interruption-tolerant task state. Stack: realtime API with tool calling plus a small state machine; intermediate. Extension: a camera stream so the agent sees what the user sees.
13. Elder check-in service. Scheduled outbound calls with cross-session memory, affect tracking over weeks ("you've sounded flat the last three calls"), and family escalation. Raises the memory, safety, and disclosure problems simultaneously. Stack: any voice loop plus an open-source memory layer (mem0 or Letta); telephony optional — a web demo suffices for the day; intermediate. Extension: longitudinal prosody trend detection as the core signal.
14. Dungeon master. A persistent-personality game master with per-character voices (voice design from description), barge-in tolerance from a table of players, and world-state memory. Stack: realtime API plus voice design plus a state store; beginner-friendly, and it directly stresses multi-speaker dynamics. Extension: diarization so it addresses players by name.
15. Live interpreter. Speech-to-speech interpretation preserving the speaker's prosody and emotional register across languages, testing mid-sentence code-switching without voice identity drift. Stack: streaming STT plus cross-lingual TTS with a consistent voice; intermediate. Extension: simultaneous rather than consecutive interpretation, which requires full-duplex output.
16. Empathic companion. The demo named in the event description itself: a companion that notices you sound tired. Close the loop from paralinguistic STT signals into TTS steering — detected flatness or stress shifts the agent's prosody and its conversational tack — and demo the contrast with the loop switched off. Stack: realtime API with paralinguistic signals plus steering tags; beginner-to-intermediate. Extension: cross-session memory (mem0 or Letta) so it recalls how you sounded yesterday, working through the user's actual day.
17. Empathetic support agent. A customer-service agent that adjusts register to circumstance: detected frustration slows the pacing, drops the script, and escalates earlier, while a calm caller gets efficiency. Score resolution rate and caller sentiment against a static-register baseline. Stack: a managed platform or open framework plus paralinguistic STT; intermediate. Extension: fold in Article 50-compliant disclosure as part of the persona rather than a preamble.

Evaluation and safety

18. Disclosure UX study. Article 50 takes effect the day after this event. Build the same agent with three disclosure designs — upfront announcement, persona-integrated, on-demand — and measure task completion and user sentiment across attendees. Stack: one agent, three prompts, a short survey; beginner-friendly, unusually publishable. Extension: machine-readable synthetic-audio marking, tested for survival through telephony codecs.
19. Entity-accuracy stress harness. Phone numbers, alphanumeric codes, and names read through 8kHz compression and noise, scored across STT providers — the benchmark deployments actually need. Stack: TTS-generated test audio, ffmpeg (free) for codec simulation, STT free tiers; intermediate. Extension: publish it.
20. S2S span-tree reconstructor. Observability for transcript-free agents: parallel STT for pseudo-transcripts, turn segmentation, overlap detection, latency annotation over an S2S session's audio streams. Stack: open-source audio processing (silence and overlap detection) plus an STT API plus a trace viewer; advanced, targeting an open problem named in the research-frontier section. Extension: grow into an eval harness for full-duplex agents.

Pick a problem from the list, or bring your own — the best projects tend to attack one of the three clusters above with a working demo and an honest measurement. Apply to attend, and see you in Hillsborough on August 1.