Designing a Provider-Agnostic Voice AI Ordering Pipeline

The Problem

You're building an AI-powered phone ordering system for restaurant chains. Customers call, an AI takes their order, collects payment, and places it in the POS. Sounds simple — until you factor in the real-world requirements:

Not every location wants AI. Some stores prefer human agents. The system must decide per-call whether AI handles it or a human does.
Voice AI providers change. You might start with one provider, hit quality or latency issues, and need to swap to another. The call flow shouldn't break when you switch.
Payment over voice is sensitive. The AI needs to collect credit card numbers via DTMF or speech, and the flow must handle cash, card, and gift card paths differently.
AI fails. When the AI can't understand the customer or the order gets too complex, it needs to hand off to a human agent cleanly — mid-call, with context.

This post walks through how I'd design a voice AI ordering pipeline that handles all of this — per-tenant routing, provider abstraction, payment branching, and graceful fallback to humans.

Architecture: The Telephony Layer Owns the Call

The most important design decision: the telephony system owns the call lifecycle, not the AI.

When a customer calls, the telephony system (your IVR / call routing platform) receives the call and keeps it. The AI is brought in as a SIP leg — a separate connection that the telephony system bridges into the call. When the AI is done, the telephony system takes the call back for post-call actions (like a satisfaction survey).

This separation matters because:

Provider swaps are invisible to the caller. The telephony system decides which AI provider handles the call. Switching providers means changing a config value, not rewriting the call flow.
Fallback is clean. If the AI leg fails or the AI decides to transfer to a human, the telephony system still has the original call. It just routes it to an agent queue. No call drops.
Post-call actions are guaranteed. CSAT surveys, call tagging, CDR logging — these happen on the telephony side regardless of what the AI did.

Design principle

Never let the AI own the call. The AI is a participant, not the controller. The telephony layer initiates, bridges, and terminates. This makes every failure mode recoverable.

Step 1: Per-Tenant AI Routing

When a call arrives, the telephony system needs to decide: should this call go to AI or to a human? This decision happens per restaurant, not globally.

The telephony system makes an API call to your backend:

GET /api/store-config?did={dialed_number}

Response:

{
  "restaurant_number": "555-0142",
  "ai_enabled": true,
  "sip_uri": "sip:ai-agent@provider.example.com",
  "sip_provider": "provider_a",
  "store_name": "Downtown Location"
}

Fig 1 — The store config API returns per-tenant routing decisions.

The key fields:

ai_enabled — Feature flag per store. If false, the call skips AI entirely and goes to the human/store path.
sip_uri — Where to send the AI leg. This points to whichever voice AI provider currently handles this store's calls.
sip_provider — Identifies which provider is behind the SIP URI. Useful for provider-specific SIP header formatting.

This gives you granular control. You can enable AI for 50 stores, keep 200 on human agents, and A/B test a new AI provider on 10 stores — all by changing database records, no deploys needed.

Step 2: Provider-Agnostic SIP Handoff

When ai_enabled is true, the telephony system places a SIP call to the AI provider's URI. The call carries context via SIP headers:

X-Call-Key — unique identifier linking this AI leg back to the original call
X-Restaurant-Number — so the AI knows which menu, hours, and pricing to use
X-Customer-Number — for order history lookup and personalization

Why SIP headers, not a webhook?

SIP headers arrive with the call setup itself — zero additional latency. A webhook requires the AI to make a separate HTTP call back to your API to fetch context, adding 100-300ms before the AI can even greet the customer. At the telephony layer, every millisecond of silence is noticeable.

Because the SIP URI and provider are stored per-tenant, swapping providers is a database update. Last month's provider had latency issues? Point 10 stores at a new provider's SIP URI, monitor quality, then roll out to the rest. The telephony flow doesn't change at all.

Handling provider differences

In theory, SIP is a standard protocol. In practice, every AI voice provider has quirks — different header naming conventions, different codec preferences, different ways of signaling call end. The sip_provider field lets your telephony system apply provider-specific adapters:

Provider A expects custom headers prefixed with X-
Provider B expects context in the SIP User-Agent field
Provider C requires a specific codec order in the SDP

A simple adapter pattern in your telephony layer handles this without leaking provider details into the rest of the system.

Step 3: The AI Ordering Conversation

Once the SIP leg connects, the AI takes over the conversation. It greets the customer, takes the order, and handles modifications. This post doesn't cover the LLM/NLU side — that's its own deep topic. What matters architecturally is how the AI signals outcomes back to the telephony system.

The AI conversation ends in one of three outcomes, and each requires a different next step:

Outcome 1: Cash Order

Customer pays cash on pickup. AI places the order in the POS via API, sets a custom variable routing = 'csat', and ends the AI leg. The telephony system sees the routing variable and plays a CSAT (customer satisfaction) survey before ending the call.

Outcome 2: Card / Gift Card Payment

Customer wants to pay by card. The AI collects card details (via speech or DTMF), processes payment through the payment gateway, then places the order in the POS. Same routing = 'csat' flow after.

Outcome 3: Transfer to Human / AI Failed

The AI can't handle the request — complex modification, customer frustration, or a system error. AI sets routing = 'agent' and a popup_url (a link to the partial order context), then ends the AI leg. The telephony system routes the call to a human agent queue, with the context URL available for screen pop.

Fig 2 — Three conversation outcomes, each with a different telephony routing action.

Step 4: Signaling via Custom Variables

The key mechanism here is custom variables set by the AI that the telephony system reads after the AI leg ends. This is a stateless, loosely-coupled handoff:

Variable	Value	Telephony Action
`routing`	`'csat'`	Play CSAT IVR survey, then end call
`routing`	`'agent'`	Route to human agent queue
`popup_url`	URL string	Pass to agent UI as screen pop (partial order context)

The AI doesn't need to know how the telephony system routes to an agent or plays a CSAT survey. It just sets a variable and hangs up its leg. The telephony system doesn't need to know what the AI discussed. It just reads the routing variable and acts. Neither system depends on the other's internals.

Why this matters

This decoupling means you can swap AI providers, change CSAT survey logic, restructure agent queues, or add new routing outcomes — all independently. The contract between the two systems is just a few string variables.

Step 5: Post-Call — CSAT and CDR

After the AI ends its leg, the telephony system still has the original call. For successful orders (routing = 'csat'), it plays a short satisfaction survey — "Press 1 if you're satisfied, press 2 if not" — collects the DTMF response, and ends the call.

For agent transfers (routing = 'agent'), the call enters the agent queue. The popup_url gives the human agent context about what the AI already discussed, so the customer doesn't have to repeat themselves.

Regardless of outcome, the telephony system writes a CDR (call detail record) covering the full call lifecycle: time in IVR, time with AI, outcome, time with agent (if transferred), CSAT score. This data drives reporting — which stores perform well with AI, which have high transfer-to-human rates, which need menu adjustments.

The Full Flow

              1.
              Customer dials restaurant number

              2.
              Telephony receives call, extracts DID

              3.
              Telephony → GET /api/store-config?did=... → gets ai_enabled, sip_uri, sip_provider

              4a.
              [ai_enabled = false] → call goes to human/store path directly

              4b.
              [ai_enabled = true] → Telephony places SIP call to sip_uri with context headers

              5.
              AI answers, begins ordering conversation

              6.
              AI reaches outcome → sets routing variable → ends AI leg

              7a.
              [routing = 'csat'] → Telephony plays CSAT IVR → call ends

              7b.
              [routing = 'agent'] → Telephony routes to agent queue with popup_url → human takes over
            

Fig 3 — Complete voice AI ordering flow, 7 steps from dial to resolution.

Evolving the AI Provider Without Downtime

The provider-agnostic design pays off when you need to migrate. A typical evolution:

Phase 1 — You build the initial AI ordering with Provider A. Quality is acceptable, but latency is high on complex orders and call audio quality isn't great.
Phase 2 — Provider B offers better voice quality and lower latency. You update sip_uri and sip_provider for 10 test stores. Monitor order completion rate, transfer-to-human rate, and CSAT scores for a week.
Phase 3 — Provider B performs better. Roll out to all AI-enabled stores by updating their config records. Zero code changes. Zero deploys. Zero downtime.

If Provider B has an outage, you revert those config records to Provider A's SIP URI. Recovery time: seconds, not a deployment cycle.

Lessons

The telephony layer owns the call, the AI borrows it. This makes every failure mode recoverable and every provider swap invisible to the caller.
Per-tenant config, not per-tenant code. AI-enabled, SIP URI, provider — these are database records, not branches in your codebase. Feature flags at the telephony layer.
SIP headers beat webhooks for call context. Context travels with the call setup, not as a follow-up HTTP request. Zero added latency.
Custom variables are a clean contract. The AI sets routing and popup_url. The telephony system reads them. Neither knows the other's internals. This is the boundary that lets both systems evolve independently.
Measure per-store, not globally. AI works great for some locations and poorly for others (noisy environments, complex menus, regional accents). Per-tenant metrics let you enable AI where it works and keep humans where it doesn't.

The core insight: treat voice AI as a pluggable leg in your telephony system, not as the system itself. The telephony layer handles the call lifecycle, routing decisions, and post-call actions. The AI handles the conversation. The contract between them is a SIP URI, a few headers, and a couple of string variables. Everything else can change independently.

Want to discuss this architecture or have questions?
hello@aawezkhan.com · LinkedIn