Architecture Engineering Reliability Tech stack Work with us → ← Back to Lab

Production system · case study

Designed, built, deployed, and operated end to end

Relay is not a prompt wrapped in a phone number.

It is a production AI system spanning real-time audio, model orchestration, durable knowledge retrieval, multi-tenant SaaS, billing, identity, observability, and cloud infrastructure. Sandy Brook DevWorks engineered the whole path.

Trace the architecture View the live product
Live signal path Production
01

Caller → Twilio

PSTN · signed webhook · SHAKEN/STIR

02

Twilio.API voice bridge

G.711 μ-law 8 kHz ⇄ PCM16 16 kHz

03

Gemini Live session

Bidirectional audio · VAD · session recovery

04

Mode-gated tools

Calendar · knowledge · search · redirect

05

Durable completion

Cloud Tasks · Firestore · email · telemetry

Current repository snapshot

The evidence is in the system.

Counts verified against the production codebase.

4

deployed services

Voice, API, dashboard, public site

126

REST endpoints

Across 26 Relay.API controllers

326

automated test definitions

Voice, API, and dashboard suites

115

composite indexes

Including vector search infrastructure

System architecture

Three planes. One product.

The architecture separates latency-sensitive conversation from transactional product work and retryable background processing. Each plane gets the failure semantics and security boundary it needs.

01 · Real-time voice plane

Keep the conversation moving.

latency-sensitive
INPUT
Caller / PSTN

A real phone call

EDGE
Twilio Voice

Webhook + media stream

BRIDGE
Twilio.API

Transcode + session state

MODEL
Gemini Live

Native audio + tools

ACTION
SK plugins

Calendar, RAG, transfer

02 · Control plane

Operate a multi-tenant product.

A Blazor Server BFF keeps browser credentials and backend access separated. Relay.Web talks to Relay.API; Relay.API owns tenant-scoped data, billing, configuration, teams, phone lifecycle, integrations, and signed recording access.

Browser
Relay.Web
Relay.API

cookie auth → BFF → JWT REST → Firestore / Stripe

03 · Async and data plane

Move expensive work off the hot path.

Cloud Tasks runs post-call enrichment, webhook delivery, and knowledge indexing with OIDC-authenticated targets. Cloud Scheduler drives cleanup, reconciliation, retries, retention, and refresh. Firestore, Cloud Storage, and BigQuery hold operational and analytical state.

Cloud Tasks
Workers
Durable state

deterministic jobs → idempotent work → observable completion

Identity

Google-signed OIDC between services. No shared internal API key.

Persistence

Tenant-scoped Firestore, protected object storage, atomic markers.

Infrastructure

Cloud Run, IAM, queues, schedulers, indexes, KMS, and Stripe catalog in Pulumi C#.

Operations

Structured logs, distributed traces, custom metrics, cost analytics.

Engineering depth

The hard parts are the product.

A convincing demo can ignore failure, tenancy, security, and operations. Relay cannot. These are the systems that turn a model capability into dependable software.

REAL-TIME AUDIO

A purpose-built bidirectional media bridge.

Twilio streams G.711 μ-law at 8 kHz; Gemini expects PCM16 at 16 kHz. Relay converts both directions in-process, moves frames through bounded System.Threading.Channels, tunes server-side voice activity detection, and measures latency from caller turn-end to first model audio.

  • →Separate send and receive loops prevent one direction from starving the other.
  • →Abnormal provider closes use bounded reconnects and Gemini session resumption.
  • →Terminal failures become persisted state and a localized caller response—not dead air.
AI ORCHESTRATION

Tools are granted by context, not exposed wholesale.

Semantic Kernel plugins are registered at call start according to mode, entitlement, configuration, and available data. Receptionist calls can book or cancel appointments, search approved tenant knowledge, and redirect callers. Trusted-owner calls can query message history or use Google Search grounding.

  • →Tool boundaries prevent capabilities from leaking between receptionist and assistant modes.
  • →Calendar operations re-check availability server-side and interpret time in the tenant’s timezone.
  • →Per-line overrides resolve against tenant defaults before the call snapshot is committed.
MULTI-TENANT SAAS

Product boundaries stay explicit.

The browser-facing dashboard is a BFF, the REST API owns product state, and the voice service alone owns telephony and email provider SDKs. Tenant identity comes from signed claims and scopes every data access. Stripe plan entitlements are centralized and repaired by an idempotent daily reconciler.

  • →Google OAuth, Firebase email/password, and WebAuthn passkeys converge on one JWT model.
  • →Phone numbers can be provisioned, held, resumed, or released through a reversible lifecycle.
  • →Team roles, API keys, webhooks, Zapier, billing, and retention are first-class product surfaces.
PLATFORM ENGINEERING

The deployment is part of the codebase.

Pulumi C# provisions the Cloud Run services and sidecars, IAM relationships, queues, schedulers, storage, KMS keys, secrets, databases, indexes, alerts, domains, and Stripe product catalog. Environments are repeatable because infrastructure decisions are reviewed like application code.

  • →Workload-specific service accounts use narrow, resource-scoped permissions.
  • →OIDC tokens are minted at dispatch time for task and scheduler targets.
  • →Cloud KMS envelope-encrypts Google Calendar refresh tokens before persistence.

Production RAG

Knowledge that can change without breaking the call.

Relay’s Knowledge Assistant ingests approved PDFs, DOCX files, and public websites, then lets the receptionist retrieve grounded answers during a live call. The interesting work is not embedding text. It is preserving a trustworthy active corpus while ingestion, refresh, deletion, downgrade, and failure happen around it.

DESIGN INVARIANT

A failed refresh never replaces the last known-good generation.

  1. 01

    Acquire and constrain

    Native-text PDF and DOCX extraction, plus an SSRF-safe website crawler with DNS/IP validation and a 50-page ceiling. Originals and normalized artifacts are retained in protected Cloud Storage.

  2. 02

    Prepare before publishing

    Deterministic Cloud Tasks split normalization and embedding into durable phases. Pending artifacts, hashes, vectors, and sensitive-data acknowledgement remain separate from the active generation.

  3. 03

    Commit one authoritative generation

    Chunks use 768-dimension Vertex embeddings. Only after every dependent record is durable do activeGeneration, content hash, and chunk count move together.

  4. 04

    Retrieve within hard boundaries

    Vector search is isolated by tenant, selected knowledge set, entitlement, and active source generation. It returns at most five passages, enforces a similarity threshold, and fails open after two seconds so the conversation continues.

Reliability engineering

Failure is part of the architecture.

The system assumes webhooks repeat, providers fail, workers stop halfway through, configuration drifts, and external calls time out. Recovery behavior is designed before the happy path is considered complete.

Duplicate Twilio webhook
Overwrite-safe message write and deterministic Cloud Task name keyed by call SID.
ALREADY_EXISTS = success
Partial enrichment failure
Retries resume from persisted completion and email markers instead of repeating costly model work.
enrichmentStatus
emailSentAt
Retry budget exhausted
The final worker attempt records a DLQ document; a scheduled sweep retries outside the original task lifecycle.
dlqEnrichmentTasks
Gemini session interruption
Bounded reconnect with the latest resumption handle; terminal failure ends the call cleanly with localized audio.
reconnects
terminal_failures
Knowledge refresh fails
Pending state remains pending; retrieval continues serving the previous active generation.
active ≠ pending
Plan state drifts
A daily idempotent reconciliation job repairs entitlement mirrors and reversibly suspends out-of-plan integrations.
catalog → reconcile

Security architecture

Trust is narrow and explicit.

Browser boundary

Secure cookies terminate at the Blazor BFF. Firebase credentials never reach Relay servers; passkeys use short-lived WebAuthn challenges.

Service boundary

Google-signed OIDC tokens are audience-pinned and service accounts are allow-listed. Shared internal API keys were removed.

Telephony boundary

Twilio HTTP signatures are validated; WebSockets require a five-minute pending-call record; demo URLs carry signed expiring tokens.

Data boundary

Every API request resolves tenant identity from signed claims. OAuth refresh tokens are KMS-encrypted and retrieval never crosses tenant or generation boundaries.

Observability

The AI session is not a black box.

Each deployed .NET service emits structured JSON logs, distributed traces, and metrics through an OpenTelemetry Collector sidecar to Cloud Trace and Cloud Monitoring. The voice bridge adds domain telemetry at the exact points generic HTTP instrumentation cannot see.

twilio-api.gemini
gemini.turn.latency

VAD end → first model audio

gemini.session.duration

Full session wall time

gemini.interruptions

Caller interruption behavior

gemini.reconnects

Recovered provider sessions

Versioned per-call cost breakdowns are written atomically with enrichment completion, then streamed to BigQuery for COGS analysis. A local-only founder operations dashboard reads Logging, Monitoring, Firestore, BigQuery, and Stripe without introducing another public service.

What ships on top

Engineering becomes product capability.

The architecture is not ornamental. It supports the customer-facing behavior while keeping each capability isolated, configurable, billable, and observable.

29-language voice behavior

Silent in-call language mirroring

Live appointment operations

Availability, booking, lookup, cancellation

Knowledge-grounded answers

Approved documents and websites

Live call transfer

Private whisper and outcome tracking

Per-line configuration

Overrides with explicit inheritance

Integration surfaces

API keys, webhooks, and Zapier

System inventory

The stack, by responsibility.

Application

  • .NET 10ASP.NET Core
  • Blazor ServerDashboard BFF
  • Semantic KernelTool orchestration

AI + voice

  • Vertex AIGemini Live, text, embeddings
  • Twilio VoiceMedia Streams + TwiML
  • Firestore vectorsTenant-scoped retrieval

Cloud platform

  • GCP Cloud RunContainerized services
  • Tasks + SchedulerDurable work and maintenance
  • Storage + KMSProtected artifacts and tokens

Operations

  • Pulumi C#Infrastructure as code
  • OpenTelemetryLogs, traces, metrics
  • Stripe + BigQueryBilling and unit economics

The point of the project

The model is one component.
The system is the work.

Relay demonstrates how Sandy Brook DevWorks approaches ambitious software: understand the constraints, draw clear boundaries, design for failure, instrument the unknowns, and ship the whole thing.

Talk to Sandy Brook Visit Relay