The Microsoft Stack
Start Here
Overview and approach
Foundry
The production AI platform
Building Agents
Pro-code and low-code
Data and Retrieval
What grounds the agents
Operating in Production
Running it safely

GenAI API Gateway

A GenAI API Gateway sits between your applications and Foundry Models endpoints. It handles load balancing across deployments, PTU overflow to PAYG, per-consumer billing, and centralized governance. Azure API Management (APIM) is the standard implementation.

Pricing update (effective September 2026): Microsoft Foundry introduces updated pricing for model deployments in the EU Data Zone and Regional deployments outside the US, alongside a new APAC Data Zone. Factor this into multi-region routing policies.

Problem It Solves

Enterprise AI deployments hit these problems fast:

  • Multiple teams sharing limited TPM quota
  • PTU capacity wasted during low-traffic, PAYG costs spiking during peaks
  • No visibility into which team/app consumes how many tokens
  • No centralized rate limiting or access control

Architecture

Consumers
App A
App B
App C
Azure API Management
Auth
Authentication with Entra ID
Rate Limiting
Per-app TPM quota
Load Balancer
Weighted routing
Token Metrics
Every call logged to Event Hub
Model Deployments
East US
PTU, priority 1
West Europe
PTU, priority 1
East US 2
PAYG, only on a 429

Key Capabilities

Load Balancing

Routing Strategies

StrategyHow It WorksUse When
Round-robinDistribute evenlyAll backends have equal capacity
WeightedTraffic percentages per backendPTU backends get higher weight
Priority + fallbackPTU first, PAYG on 429Maximize PTU utilization
Latency-basedRoute to lowest-latencyMulti-region with latency needs

APIM Policy: PTU overflow to PAYG

<policies>
  <inbound>
    <set-backend-service backend-id="ptu-eastus" />
  </inbound>
  <on-error>
    <choose>
      <when condition="@(context.Response.StatusCode == 429)">
        <set-backend-service backend-id="payg-eastus2" />
        <forward-request />
      </when>
    </choose>
  </on-error>
</policies>

Rate Limiting

Token-based rate limiting

Traditional rate limiting counts requests. For LLMs, you need to limit by tokens consumed - a single request can use 100 or 10,000 tokens.

<policies>
  <inbound>
    <rate-limit-by-key
      calls="100"
      renewal-period="60"
      counter-key="@(context.Subscription.Id)" />
  </inbound>
  <outbound>
    <log-to-eventhub logger-id="token-logger">
      @{
        return new JObject(
          new JProperty("consumer", context.Subscription.Id),
          new JProperty("prompt_tokens",
            context.Response.Headers.GetValueOrDefault("x-ms-prompt-tokens","")),
          new JProperty("completion_tokens",
            context.Response.Headers.GetValueOrDefault("x-ms-completion-tokens",""))
        ).ToString();
      }
    </log-to-eventhub>
  </outbound>
</policies>

Token Tracking

Per-consumer billing dashboard

APIM
Event Hub
Token logs
Data Explorer
Billing Dashboard

Output:

ConsumerTokens (30d)Cost
App A1.2M$48
App B450K$18
App C3.8M$152

Prioritization

Request priority routing

Interactive chat needs low latency; batch summarization can wait.

<policies>
  <inbound>
    <choose>
      <when condition="@(context.Request.Headers
        .GetValueOrDefault('X-Priority','normal') == 'high')">
        <set-backend-service backend-id="ptu-eastus" />
      </when>
      <otherwise>
        <set-backend-service backend-id="payg-eastus2" />
      </otherwise>
    </choose>
  </inbound>
</policies>

Semantic Caching

Cache semantically similar queries

Embedded Query
A user query arrives and is embedded
Cache hit?
Cosine similarity above 0.95
Return Cached
Return the cached response
Call the Model
Forward to Azure OpenAI, cache the response, and return it

Implementation options:

  • Azure Redis Enterprise with vector search
  • Azure AI Search as semantic cache layer
  • APIM built-in response caching (exact match only)

Production Considerations

Maximizing PTU Utilization

PTU is a fixed cost - unused capacity is wasted money.

  1. Spillover routing: Fill PTU first, overflow to PAYG only on 429
  2. Batch backfill: Route low-priority batch jobs to PTU during off-peak
  3. Multi-model sharing: Deploy multiple models on same PTU allocation
  4. Monitoring: Alert when PTU utilization drops below 70%

Resilience

  • Deploy APIM in multiple regions with Traffic Manager
  • Retry policies with exponential backoff
  • Circuit breakers on backends returning repeated 5xx
  • Health probes to detect backend degradation

Security

  • Authenticate consumers via Entra ID (OAuth2 tokens)
  • Never expose Azure OpenAI keys to consumers
  • Mutual TLS between APIM and backends
  • Log all requests for audit compliance

When to Build a Gateway

Yes, Build It

  • Multiple apps sharing deployments
  • PTU + PAYG hybrid strategy
  • Per-team billing/chargeback needed
  • Multi-region failover required
  • Centralized governance and audit

No, Skip It

  • Single app, single deployment
  • Low traffic, no cost concerns
  • No multi-team sharing
  • Simple prototype/POC
Info

If you have more than one team consuming Azure OpenAI, you need a gateway. The cost visibility alone pays for the APIM investment within weeks.


Key Takeaways

  • If more than one team consumes Azure OpenAI, put a gateway in front of it. For a single app on a single deployment, skip it.
  • Rate-limit by tokens, not requests. One request can cost 100 tokens or 10,000.
  • Fill PTU first and spill over to PAYG only on a 429. Alert when PTU utilisation drops below 70%.
  • Per-consumer token tracking is what makes chargeback possible, and the cost visibility alone usually pays for APIM.
  • Authenticate consumers with Entra ID and never expose model keys to them.

Next: Security & Guardrails - the last layer before production: the threats specific to LLM applications and the Azure control for each.