Observability & Evaluation
Production AI needs more than "it works". You need to trace what happened, evaluate quality, identify failures, and continuously improve. Foundry provides this end-to-end.
Tracing
Foundry emits OpenTelemetry traces for every agent run. Automatic when Application Insights is connected - no code changes required.
What gets captured
| Span | Data |
|---|---|
| agent_run | Full request lifecycle, total latency, status |
| model_call | Model name, token count (input/output), latency, cost |
| tool_call | Tool name, input parameters, output, execution time |
| retrieval | Search query, results count, relevance scores |
| content_filter | Filter category, severity, blocked/allowed |
Tracing flow
Export options
- Application Insights - default, zero-config for Foundry agents
- Datadog - via OTLP exporter
- Any OTLP backend - standard OpenTelemetry
Framework support
Works with any framework: Microsoft Agent Framework, LangChain, Semantic Kernel, CrewAI, LangGraph, custom.
Evaluation
Built-in Evaluators
| Evaluator | Measures | Requires |
|---|---|---|
| Groundedness | Are claims supported by context? | Context + response |
| Relevance | Does the response address the query? | Query + response |
| Coherence | Is the response logically structured? | Response |
| Fluency | Is the language natural? | Response |
| Similarity | Token overlap with ground truth | Ground truth + response |
| Violence/Hate/Sexual/Self-harm | Safety classification | Response |
Running evaluations
from azure.ai.evaluation import evaluate, GroundednessEvaluator, RelevanceEvaluator
results = evaluate(
data="test_dataset.jsonl",
evaluators={
"groundedness": GroundednessEvaluator(model_config=model_config),
"relevance": RelevanceEvaluator(model_config=model_config),
}
)
print(f"Groundedness: {results['groundedness']:.2f}")
ASSERT
Open-source framework for policy-driven agent evaluation. Converts your written policies into executable evaluations and generates targeted test scenarios.
| Step | What it does |
|---|---|
| 1. Define policies | Write rules in natural language ("Agent must not disclose pricing to non-customers") |
| 2. Generate scenarios | ASSERT creates targeted test cases from policies |
| 3. Run evaluations | Execute against your agent |
| 4. Surface defects | Identify safety and quality failures before production |
Works across LangChain, CrewAI, OpenAI, LightLLM, and more.
Agent Control Specification (ACS)
Open industry specification for deterministic safety controls at five agent lifecycle checkpoints:
Expressed as portable YAML. Versionable, auditable, framework-agnostic.
Rubric Evaluator
Auto-generates evaluation criteria based on your agent's context (public preview):
- Creates custom quality criteria from agent definition and use case
- Supports weighted dimensions for aggregate scoring
- Runs alongside safety evaluators for unified scorecard
- Feeds into Agent Optimizer
Agent Optimizer
Closes the loop from production signal to improvement (coming soon in public preview):
- Collects production traces
- Runs evaluation suite
- Generates ranked, reviewable improvement suggestions
- Developer applies changes
Agent ROI
Measures business impact of agents (private preview):
- Task completion rates
- Time saved
- Cost efficiency
- Stakeholder justification data
Guided Guardrail Setup
Questionnaire-based guardrail configuration (public preview):
- Answer questions about agent audience, data access, use case
- System recommends relevant controls (PII filters, jailbreak protection, task adherence)
- Apply with one click
No security expertise required.
AI Red Teaming Agent
Automated adversarial testing (preview). Probes your agent for vulnerabilities before deployment.