A GenAI API Gateway sits between your applications and Foundry Models endpoints. It handles load balancing across deployments, PTU overflow to PAYG, per-consumer billing, and centralized governance. Azure API Management (APIM) is the standard implementation.

Problem It Solves

Enterprise AI deployments hit these problems fast:

• Multiple teams sharing limited TPM quota
• PTU capacity wasted during low-traffic, PAYG costs spiking during peaks
• No visibility into which team/app consumes how many tokens
• No centralized rate limiting or access control

Architecture

Key Capabilities

Load Balancing

Routing Strategies

APIM Policy: PTU overflow to PAYG

<policies>
  <inbound>
    <set-backend-service backend-id="ptu-eastus" />
  </inbound>
  <on-error>
    <choose>
      <when condition="@(context.Response.StatusCode == 429)">
        <set-backend-service backend-id="payg-eastus2" />
        <forward-request />
      </when>
    </choose>
  </on-error>
</policies>

Rate Limiting

Token-based rate limiting

Traditional rate limiting counts requests. For LLMs, you need to limit by tokens consumed - a single request can use 100 or 10,000 tokens.

<policies>
  <inbound>
    <rate-limit-by-key
      calls="100"
      renewal-period="60"
      counter-key="@(context.Subscription.Id)" />
  </inbound>
  <outbound>
    <log-to-eventhub logger-id="token-logger">
      @{
        return new JObject(
          new JProperty("consumer", context.Subscription.Id),
          new JProperty("prompt_tokens",
            context.Response.Headers.GetValueOrDefault("x-ms-prompt-tokens","")),
          new JProperty("completion_tokens",
            context.Response.Headers.GetValueOrDefault("x-ms-completion-tokens",""))
        ).ToString();
      }
    </log-to-eventhub>
  </outbound>
</policies>

Token Tracking

Per-consumer billing dashboard

Output:

Prioritization

Request priority routing

Interactive chat needs low latency; batch summarization can wait.

<policies>
  <inbound>
    <choose>
      <when condition="@(context.Request.Headers
        .GetValueOrDefault('X-Priority','normal') == 'high')">
        <set-backend-service backend-id="ptu-eastus" />
      </when>
      <otherwise>
        <set-backend-service backend-id="payg-eastus2" />
      </otherwise>
    </choose>
  </inbound>
</policies>

Semantic Caching

Cache semantically similar queries

Implementation options:

• Azure Redis Enterprise with vector search
• Azure AI Search as semantic cache layer
• APIM built-in response caching (exact match only)

Production Considerations

PTU is a fixed cost - unused capacity is wasted money.

1. Spillover routing: Fill PTU first, overflow to PAYG only on 429
2. Batch backfill: Route low-priority batch jobs to PTU during off-peak
3. Multi-model sharing: Deploy multiple models on same PTU allocation
4. Monitoring: Alert when PTU utilization drops below 70%

• Deploy APIM in multiple regions with Traffic Manager
• Retry policies with exponential backoff
• Circuit breakers on backends returning repeated 5xx
• Health probes to detect backend degradation

• Authenticate consumers via Entra ID (OAuth2 tokens)
• Never expose Azure OpenAI keys to consumers
• Mutual TLS between APIM and backends
• Log all requests for audit compliance

When to Build a Gateway

Yes, Build It

• Multiple apps sharing deployments
• PTU + PAYG hybrid strategy
• Per-team billing/chargeback needed
• Multi-region failover required
• Centralized governance and audit

No, Skip It

• Single app, single deployment
• Low traffic, no cost concerns
• No multi-team sharing
• Simple prototype/POC