Building a Low-Latency Realtime Voice Agent in ASP.NET Core
OpenAI's Realtime API (wss://api.openai.com/v1/realtime) enables direct speech-to-speech interaction with sub-300ms latency. Connecting frontend clients directly to OpenAI exposes your API keys and prevents server-side business logic, so hosting an ASP.NET Core WebSocket proxy is the recommended production architecture.
This guide covers setting up a bi-directional audio relay using ASP.NET Core and native ClientWebSocket connections.
System Architecture
[ Client Mic ] ──(Audio Data)──> [ ASP.NET Core Backend ] ──(Relay)──> [ OpenAI Realtime API ]
[ Client Speaker ] <──(Audio Data)── [ ASP.NET Core Backend ] <──(Relay)── [ OpenAI Realtime API ]- Client (Browser/Mobile/SIP): Captures microphone input (PCM16) and streams frames to your backend.
- ASP.NET Core Server: Holds the WebSocket connection to OpenAI, injects authorization, configures VAD (Voice Activity Detection), and inspects events for tool calling.
- OpenAI Realtime API: Processes audio, streams back voice responses, and handles function call execution triggers.
Architecture Choice: Realtime API vs. Pipeline Approach
- Realtime API (Audio-in / Audio-out): Optimized for low latency (<300ms), continuous speech conversations, and dynamic interruption handling via Server VAD.
- Pipeline Approach (STT → Chat Completions → TTS): Uses Whisper + GPT-4o + Azure/OpenAI TTS. Simpler to debug and lower cost, but higher latency (>1.5s).
Step 1: Configure Program.cs
Set up WebSocket middleware in ASP.NET Core, register configuration settings, and expose a dedicated /ws/voice WebSocket endpoint.
using VoiceAgent;
var builder = WebApplication.CreateBuilder(args);
// Pull OpenAI configuration settings securely
builder.Services.AddSingleton<RealtimeRelayOptions>(sp =>
{
var config = sp.GetRequiredService<IConfiguration>();
return new RealtimeRelayOptions
{
ApiKey = config["OpenAI:ApiKey"] ?? throw new InvalidOperationException("OpenAI:ApiKey not configured"),
Model = config["OpenAI:RealtimeModel"] ?? "gpt-realtime",
Voice = config["OpenAI:Voice"] ?? "marin",
Instructions = config["OpenAI:Instructions"]
?? "You are a helpful, concise voice assistant. Keep responses conversational and brief."
};
});
var app = builder.Build();
app.UseWebSockets(new WebSocketOptions
{
KeepAliveInterval = TimeSpan.FromSeconds(30)
});
// Endpoint for client connections: wss://yourserver/ws/voice
app.Map("/ws/voice", async (HttpContext context, RealtimeRelayOptions options) =>
{
if (!context.WebSockets.IsWebSocketRequest)
{
context.Response.StatusCode = StatusCodes.Status400BadRequest;
return;
}
using var clientSocket = await context.WebSockets.AcceptWebSocketAsync();
var relay = new RealtimeVoiceRelay(options);
await relay.RunAsync(clientSocket, context.RequestAborted);
});
app.Run();Step 2: Implement RealtimeVoiceRelay.cs
This class handles proxying raw WebSocket frames bi-directionally between the client and OpenAI using System.Net.WebSockets.ClientWebSocket.
using System.Net.WebSockets;
using System.Text;
using System.Text.Json;
using System.Text.Json.Nodes;
namespace VoiceAgent;
public class RealtimeRelayOptions
{
public required string ApiKey { get; init; }
public string Model { get; init; } = "gpt-realtime";
public string Voice { get; init; } = "marin";
public string Instructions { get; init; } = "You are a helpful voice assistant.";
}
/// <summary>
/// Bridges a browser/mobile client WebSocket to OpenAI's Realtime API WebSocket.
/// Relays audio and JSON control frames bi-directionally.
/// </summary>
public class RealtimeVoiceRelay
{
private readonly RealtimeRelayOptions _options;
private const string RealtimeEndpoint = "wss://api.openai.com/v1/realtime";
public RealtimeVoiceRelay(RealtimeRelayOptions options)
{
_options = options;
}
public async Task RunAsync(WebSocket clientSocket, CancellationToken cancellationToken)
{
using var openAiSocket = new ClientWebSocket();
openAiSocket.Options.SetRequestHeader("Authorization", $"Bearer {_options.ApiKey}");
openAiSocket.Options.SetRequestHeader("OpenAI-Beta", "realtime=v1");
var uri = new Uri($"{RealtimeEndpoint}?model={_options.Model}");
await openAiSocket.ConnectAsync(uri, cancellationToken);
// Send session initialization options immediately upon connection
await SendSessionUpdateAsync(openAiSocket, cancellationToken);
// Pump frames concurrently between client and OpenAI
var clientToOpenAi = PumpAsync(clientSocket, openAiSocket, "client->openai", cancellationToken);
var openAiToClient = PumpAsync(openAiSocket, clientSocket, "openai->client", cancellationToken);
await Task.WhenAny(clientToOpenAi, openAiToClient);
await CloseSocketSafelyAsync(clientSocket);
await CloseSocketSafelyAsync(openAiSocket);
}
private async Task SendSessionUpdateAsync(ClientWebSocket openAiSocket, CancellationToken ct)
{
var sessionUpdate = new JsonObject
{
["type"] = "session.update",
["session"] = new JsonObject
{
["modalities"] = new JsonArray { "audio", "text" },
["voice"] = _options.Voice,
["instructions"] = _options.Instructions,
["input_audio_format"] = "pcm16",
["output_audio_format"] = "pcm16",
["turn_detection"] = new JsonObject
{
["type"] = "server_vad",
["threshold"] = 0.5,
["silence_duration_ms"] = 500
},
["tools"] = new JsonArray() // Define backend function definitions here
}
};
await SendJsonAsync(openAiSocket, sessionUpdate, ct);
}
private async Task PumpAsync(WebSocket source, WebSocket destination, string label, CancellationToken ct)
{
var buffer = new byte[16 * 1024];
try
{
while (source.State == WebSocketState.Open && !ct.IsCancellationRequested)
{
using var messageStream = new MemoryStream();
WebSocketReceiveResult result;
do
{
result = await source.ReceiveAsync(buffer, ct);
if (result.MessageType == WebSocketMessageType.Close)
{
return;
}
messageStream.Write(buffer, 0, result.Count);
} while (!result.EndOfMessage);
var messageBytes = messageStream.ToArray();
// Optional hook for logging or intercepting function calls
InspectEvent(label, messageBytes);
if (destination.State == WebSocketState.Open)
{
await destination.SendAsync(
messageBytes,
WebSocketMessageType.Text,
endOfMessage: true,
ct);
}
}
}
catch (OperationCanceledException)
{
// Expected on stream termination
}
catch (WebSocketException)
{
// Handle network teardown cleanly
}
}
private void InspectEvent(string label, byte[] messageBytes)
{
try
{
var json = JsonNode.Parse(Encoding.UTF8.GetString(messageBytes));
var type = json?["type"]?.GetValue<string>();
// Filter out high-frequency audio deltas from debug logs
if (type is not null && !type.Contains("audio.delta"))
{
Console.WriteLine($"[{label}] event: {type}");
}
}
catch (JsonException)
{
// Ignore non-JSON payload frames
}
}
private static async Task SendJsonAsync(ClientWebSocket socket, JsonNode payload, CancellationToken ct)
{
var bytes = Encoding.UTF8.GetBytes(payload.ToJsonString());
await socket.SendAsync(bytes, WebSocketMessageType.Text, endOfMessage: true, ct);
}
private static async Task CloseSocketSafelyAsync(WebSocket socket)
{
if (socket.State is WebSocketState.Open or WebSocketState.CloseReceived)
{
try
{
await socket.CloseAsync(WebSocketCloseStatus.NormalClosure, "done", CancellationToken.None);
}
catch (WebSocketException)
{
// Sockets closing concurrently
}
}
}
}Key Features & Production Enhancements
- Server VAD (Voice Activity Detection): Set to
server_vadwith a500mssilence threshold. This instructs OpenAI to manage user interruptions automatically without manual stream pause handling on the client. - Audio Format: Configured to
pcm16by default. Ensure your frontend captures raw audio formatted at 24kHz or 16kHz mono PCM before sendinginput_audio_buffer.appendevents. - Tool Calling Execution: Intercept
response.function_call_arguments.doneinsideInspectEvent. Parse the arguments, execute your C# server logic or DB call, and push aconversation.item.createevent back into the OpenAI socket to resume speaking. - Telephony Integration: This setup maps directly to Twilio Media Streams. Point Twilio's incoming call WebSockets to your
/ws/voiceroute to bridge phone calls to OpenAI Realtime.
Running this proxy keeps your OpenAI API keys secured server-side while maintaining full control over audio interception, metrics, authorization, and function calling.
