Next-Gen Voice AI: Proxying OpenAI Realtime WebSockets in .NET

Building a Low-Latency Realtime Voice Agent in ASP.NET Core

OpenAI's Realtime API (wss://api.openai.com/v1/realtime) enables direct speech-to-speech interaction with sub-300ms latency. Connecting frontend clients directly to OpenAI exposes your API keys and prevents server-side business logic, so hosting an ASP.NET Core WebSocket proxy is the recommended production architecture.

This guide covers setting up a bi-directional audio relay using ASP.NET Core and native ClientWebSocket connections.

---

System Architecture

[ Client Mic ] ──(Audio Data)──> [ ASP.NET Core Backend ] ──(Relay)──> [ OpenAI Realtime API ]
[ Client Speaker ] <──(Audio Data)── [ ASP.NET Core Backend ] <──(Relay)── [ OpenAI Realtime API ]

• Client (Browser/Mobile/SIP): Captures microphone input (PCM16) and streams frames to your backend.
• ASP.NET Core Server: Holds the WebSocket connection to OpenAI, injects authorization, configures VAD (Voice Activity Detection), and inspects events for tool calling.
• OpenAI Realtime API: Processes audio, streams back voice responses, and handles function call execution triggers.

Architecture Choice: Realtime API vs. Pipeline Approach

• Realtime API (Audio-in / Audio-out): Optimized for low latency (<300ms), continuous speech conversations, and dynamic interruption handling via Server VAD.
• Pipeline Approach (STT → Chat Completions → TTS): Uses Whisper + GPT-4o + Azure/OpenAI TTS. Simpler to debug and lower cost, but higher latency (>1.5s).

---

Step 1: Configure Program.cs

Set up WebSocket middleware in ASP.NET Core, register configuration settings, and expose a dedicated /ws/voice WebSocket endpoint.

using VoiceAgent;

var builder = WebApplication.CreateBuilder(args);

// Pull OpenAI configuration settings securely
builder.Services.AddSingleton<RealtimeRelayOptions>(sp =>
{
    var config = sp.GetRequiredService<IConfiguration>();
    return new RealtimeRelayOptions
    {
        ApiKey = config["OpenAI:ApiKey"] ?? throw new InvalidOperationException("OpenAI:ApiKey not configured"),
        Model = config["OpenAI:RealtimeModel"] ?? "gpt-realtime",
        Voice = config["OpenAI:Voice"] ?? "marin",
        Instructions = config["OpenAI:Instructions"] 
            ?? "You are a helpful, concise voice assistant. Keep responses conversational and brief."
    };
});

var app = builder.Build();

app.UseWebSockets(new WebSocketOptions
{
    KeepAliveInterval = TimeSpan.FromSeconds(30)
});

// Endpoint for client connections: wss://yourserver/ws/voice
app.Map("/ws/voice", async (HttpContext context, RealtimeRelayOptions options) =>
{
    if (!context.WebSockets.IsWebSocketRequest)
    {
        context.Response.StatusCode = StatusCodes.Status400BadRequest;
        return;
    }

    using var clientSocket = await context.WebSockets.AcceptWebSocketAsync();
    var relay = new RealtimeVoiceRelay(options);
    await relay.RunAsync(clientSocket, context.RequestAborted);
});

app.Run();

---

Step 2: Implement RealtimeVoiceRelay.cs

This class handles proxying raw WebSocket frames bi-directionally between the client and OpenAI using System.Net.WebSockets.ClientWebSocket.

using System.Net.WebSockets;
using System.Text;
using System.Text.Json;
using System.Text.Json.Nodes;

namespace VoiceAgent;

public class RealtimeRelayOptions
{
    public required string ApiKey { get; init; }
    public string Model { get; init; } = "gpt-realtime";
    public string Voice { get; init; } = "marin";
    public string Instructions { get; init; } = "You are a helpful voice assistant.";
}

/// <summary>
/// Bridges a browser/mobile client WebSocket to OpenAI's Realtime API WebSocket.
/// Relays audio and JSON control frames bi-directionally.
/// </summary>
public class RealtimeVoiceRelay
{
    private readonly RealtimeRelayOptions _options;
    private const string RealtimeEndpoint = "wss://api.openai.com/v1/realtime";

    public RealtimeVoiceRelay(RealtimeRelayOptions options)
    {
        _options = options;
    }

    public async Task RunAsync(WebSocket clientSocket, CancellationToken cancellationToken)
    {
        using var openAiSocket = new ClientWebSocket();
        openAiSocket.Options.SetRequestHeader("Authorization", $"Bearer {_options.ApiKey}");
        openAiSocket.Options.SetRequestHeader("OpenAI-Beta", "realtime=v1");

        var uri = new Uri($"{RealtimeEndpoint}?model={_options.Model}");
        await openAiSocket.ConnectAsync(uri, cancellationToken);

        // Send session initialization options immediately upon connection
        await SendSessionUpdateAsync(openAiSocket, cancellationToken);

        // Pump frames concurrently between client and OpenAI
        var clientToOpenAi = PumpAsync(clientSocket, openAiSocket, "client->openai", cancellationToken);
        var openAiToClient = PumpAsync(openAiSocket, clientSocket, "openai->client", cancellationToken);

        await Task.WhenAny(clientToOpenAi, openAiToClient);

        await CloseSocketSafelyAsync(clientSocket);
        await CloseSocketSafelyAsync(openAiSocket);
    }

    private async Task SendSessionUpdateAsync(ClientWebSocket openAiSocket, CancellationToken ct)
    {
        var sessionUpdate = new JsonObject
        {
            ["type"] = "session.update",
            ["session"] = new JsonObject
            {
                ["modalities"] = new JsonArray { "audio", "text" },
                ["voice"] = _options.Voice,
                ["instructions"] = _options.Instructions,
                ["input_audio_format"] = "pcm16",
                ["output_audio_format"] = "pcm16",
                ["turn_detection"] = new JsonObject
                {
                    ["type"] = "server_vad",
                    ["threshold"] = 0.5,
                    ["silence_duration_ms"] = 500
                },
                ["tools"] = new JsonArray() // Define backend function definitions here
            }
        };

        await SendJsonAsync(openAiSocket, sessionUpdate, ct);
    }

    private async Task PumpAsync(WebSocket source, WebSocket destination, string label, CancellationToken ct)
    {
        var buffer = new byte[16 * 1024];

        try
        {
            while (source.State == WebSocketState.Open && !ct.IsCancellationRequested)
            {
                using var messageStream = new MemoryStream();
                WebSocketReceiveResult result;

                do
                {
                    result = await source.ReceiveAsync(buffer, ct);

                    if (result.MessageType == WebSocketMessageType.Close)
                    {
                        return;
                    }

                    messageStream.Write(buffer, 0, result.Count);
                } while (!result.EndOfMessage);

                var messageBytes = messageStream.ToArray();

                // Optional hook for logging or intercepting function calls
                InspectEvent(label, messageBytes);

                if (destination.State == WebSocketState.Open)
                {
                    await destination.SendAsync(
                        messageBytes,
                        WebSocketMessageType.Text,
                        endOfMessage: true,
                        ct);
                }
            }
        }
        catch (OperationCanceledException)
        {
            // Expected on stream termination
        }
        catch (WebSocketException)
        {
            // Handle network teardown cleanly
        }
    }

    private void InspectEvent(string label, byte[] messageBytes)
    {
        try
        {
            var json = JsonNode.Parse(Encoding.UTF8.GetString(messageBytes));
            var type = json?["type"]?.GetValue<string>();

            // Filter out high-frequency audio deltas from debug logs
            if (type is not null && !type.Contains("audio.delta"))
            {
                Console.WriteLine($"[{label}] event: {type}");
            }
        }
        catch (JsonException)
        {
            // Ignore non-JSON payload frames
        }
    }

    private static async Task SendJsonAsync(ClientWebSocket socket, JsonNode payload, CancellationToken ct)
    {
        var bytes = Encoding.UTF8.GetBytes(payload.ToJsonString());
        await socket.SendAsync(bytes, WebSocketMessageType.Text, endOfMessage: true, ct);
    }

    private static async Task CloseSocketSafelyAsync(WebSocket socket)
    {
        if (socket.State is WebSocketState.Open or WebSocketState.CloseReceived)
        {
            try
            {
                await socket.CloseAsync(WebSocketCloseStatus.NormalClosure, "done", CancellationToken.None);
            }
            catch (WebSocketException)
            {
                // Sockets closing concurrently
            }
        }
    }
}

---

Key Features & Production Enhancements

• Server VAD (Voice Activity Detection): Set to server_vad with a 500ms silence threshold. This instructs OpenAI to manage user interruptions automatically without manual stream pause handling on the client.
• Audio Format: Configured to pcm16 by default. Ensure your frontend captures raw audio formatted at 24kHz or 16kHz mono PCM before sending input_audio_buffer.append events.
• Tool Calling Execution: Intercept response.function_call_arguments.done inside InspectEvent. Parse the arguments, execute your C# server logic or DB call, and push a conversation.item.create event back into the OpenAI socket to resume speaking.
• Telephony Integration: This setup maps directly to Twilio Media Streams. Point Twilio's incoming call WebSockets to your /ws/voice route to bridge phone calls to OpenAI Realtime.

Running this proxy keeps your OpenAI API keys secured server-side while maintaining full control over audio interception, metrics, authorization, and function calling.