The green check that hid a real problem
During a recent customer conversation, I was shown a customer-service refund agent that looked healthy by every traditional measure. Requests completed successfully. The endpoint returned HTTP 200. Latency was within the team's target, and there were no obvious exceptions in the application logs.
Then a customer asked a simple thing:
Please refund the duplicate $500 charge on invoice 83457.
The agent processed the refund and confirmed success. There was only one problem: in the failure case explored in this article, the agent calls the correct tool with the wrong parameter—refunding $5,000 instead of $500.
From the application's perspective, the request succeeded. From the business's perspective, the agent failed.
That moment changed the direction of our discussion. We were no longer asking, "Is the endpoint available?" We were asking:
- Which tool did the agent call, and with what arguments?
- Was the refund amount correct?
- Did the agent honor the approval threshold?
- How many other customers were affected?
- How would we prove that the correction actually worked?
This is the gap that AI observability must close.
Traditional monitoring tells us whether a system is operating. Agent observability helps us determine whether the system is behaving as intended—and whether that behavior produces a trustworthy business outcome.
Below screenshot captures successful but incorrect action by agent. Later in this article, we will build this agent step by step and explore how observability helps us identify the failure.
![]()
Observability must extend beyond logs, metrics, and traces
Logs, metrics, and traces remain essential, but an AI application introduces another dimension: the application can be technically healthy and semantically wrong.
I frame this as five layers of AI observability. A refund that returns HTTP 200 can still fail on the layers that matter:
| Layer | Question | Refund-scenario result |
|---|
| 1. Technical | Is it available, fast, healthy, and affordable? | Pass — HTTP 200, normal latency, zero exceptions |
| 2. Agent execution | What did the agent actually do? | Fail — wrong parameter passed to the tool |
| 3. Safety / policy | Was the behavior permitted? | Fail — approval threshold bypassed |
| 4. Quality | Was the action correct? | Fail — task executed incorrectly |
| 5. Business | Did it achieve the outcome? | Fail — financial loss |
These layers are related, but they are not interchangeable. A response can have low latency and high fluency while still passing the wrong amount to a payment tool. Traditional APM only covers Layer 1.
The Microsoft Foundry observability landscape
For this implementation, Microsoft Foundry was the development and evaluation experience, while Azure Monitor Application Insights became the telemetry store and investigation surface.
The responsibilities were intentionally separated:
- Microsoft Foundry Traces gave the development team an ordered view of the agent run.
- Foundry Evaluation measured quality, safety, and agent behavior.
- The Foundry Monitor experience showed operational and evaluation trends for the deployed agent.
- Application Insights Agent Observability connected runs, models, tools, token usage, latency, and failures.
- Log Analytics supported customer-specific questions with Kusto Query Language.
- Azure Monitor alerts turned important signals into an operational response.
OpenTelemetry provides the connective tissue. It gives us a standard trace model for agent, model, tool, and custom application spans instead of restricting observability to one application framework.
![]()
Step 1: Prepare the project
The sample uses Python 3.10 or later, a Microsoft Foundry project, a deployed model, and an Application Insights resource connected to the project. It does not require a preexisting repository or any additional source files. Create a new empty folder and run every snippet in this article in order.
Create and enter the working folder:
mkdir foundry-observability-demo
cd foundry-observability-demo
python -m venv .venv
Activate the virtual environment on Windows PowerShell:
.\.venv\Scripts\Activate.ps1
On macOS or Linux, use:
source .venv/bin/activate
Install the packages used by the article:
python -m pip install --upgrade pip
python -m pip install \
"azure-ai-projects>=2.0.0" \
azure-identity \
azure-monitor-opentelemetry \
azure-core-tracing-opentelemetry \
opentelemetry-sdk \
python-dotenv
For local development, authenticate with the Azure CLI:
az login
Create a local .env file. Copy the endpoint and deployment name from the Foundry project rather than hard-coding them in source control.
FOUNDRY_ENDPOINT=<copy-from-the-Foundry-project-overview>
FOUNDRY_MODEL=gpt-4o
AZURE_TENANT_ID=<your-tenant-id>
APPINSIGHTS_CONNECTION_STRING=<copy-from-Project-Tracing-Manage-data-source>
LOG_ANALYTICS_WORKSPACE_ID=<workspace-guid>
Run Python from foundry-observability-demo. The following code loads .env from that folder and validates every value required by the later snippets
Load and validate the configuration:
import os
from dotenv import load_dotenv
# Load .env from the directory where Python or Jupyter was started.
if not load_dotenv(dotenv_path=".env"):
raise FileNotFoundError(
"No .env file was found. Create it in the current working directory."
)
# Validate the variables required by the article.
required_variables = [
"FOUNDRY_ENDPOINT",
"AZURE_TENANT_ID",
"APPINSIGHTS_CONNECTION_STRING",
]
missing_variables = [
name for name in required_variables if not os.environ.get(name)
]
if missing_variables:
raise ValueError(
"Missing required .env values: " + ", ".join(missing_variables)
)
foundry_endpoint = os.environ.get("FOUNDRY_ENDPOINT")
tenant_id = os.environ.get("AZURE_TENANT_ID")
model_deployment = os.environ.get("FOUNDRY_MODEL", "gpt-4o")
app_insights_conn = os.environ.get("APPINSIGHTS_CONNECTION_STRING")
log_analytics_workspace_id = os.environ.get("LOG_ANALYTICS_WORKSPACE_ID")
print(f" Foundry endpoint : {foundry_endpoint[:50]}...")
print(f"Model deployment : {model_deployment}")
print(f"Tenant ID : {tenant_id[:8]}...")
print(" Application Insights connection string loaded")
print(
" Log Analytics workspace ID loaded"
if log_analytics_workspace_id
else " LOG_ANALYTICS_WORKSPACE_ID is optional until the KQL section"
)
Initialize the clients:
from azure.identity import AzureCliCredential
from azure.ai.projects import AIProjectClient
credential = AzureCliCredential(tenant_id=tenant_id)
project_client = AIProjectClient(endpoint=foundry_endpoint, credential=credential)
openai_client = project_client.get_openai_client()
print(" AIProjectClient initialized")
print("OpenAI client ready")
The identity running the sample needs permission to use the Foundry project. To query the resulting telemetry, it also needs appropriate access to Application Insights and its Log Analytics workspace. In production, assign access through Microsoft Entra groups and least-privilege roles.
Step 2: Create the refund agent and observe what it actually did
The agent is a function-tool agent named refund-agent-observability-demo. It exposes a single process_refund tool. The instructions are intentionally ambiguous about amount handling to demonstrate how an agent can misinterpret a parameter.
import json
from azure.ai.projects.models import PromptAgentDefinition, FunctionTool, Tool
# Define the refund tool
process_refund_tool = FunctionTool(
name="process_refund",
description="Process a refund for a customer invoice. Returns confirmation with refund details.",
parameters={
"type": "object",
"properties": {
"invoice_id": {
"type": "string",
"description": "The invoice ID to refund"
},
"amount": {
"type": "number",
"description": "The refund amount in dollars"
}
},
"required": ["invoice_id", "amount"],
"additionalProperties": False,
},
strict=True,
)
tools: list[Tool] = [process_refund_tool]
print(" Refund tool defined")
print(f" Tool: process_refund(invoice_id, amount)")
print(f" Tool: process_refund(invoice_id, amount)")
# Create the refund agent
# NOTE: The instructions are intentionally ambiguous about amount handling
# to demonstrate how agents can misinterpret parameters
agent = project_client.agents.create_version(
agent_name="refund-agent-observability-demo",
definition=PromptAgentDefinition(
model=model_deployment,
instructions="""You are a customer service agent that processes refund requests.
When a customer asks for a refund, use the process_refund tool.
Extract the invoice ID and amount from the customer's message.
Always confirm the refund was processed successfully.
""",
tools=tools,
),
)
print(f" Agent created: {agent.name} (version {agent.version})")
Now send a refund request and inspect exactly what the agent proposed. The "Traditional Monitoring View" stays green while we independently check the tool argument:
# Send a refund request
user_message = "Please refund the duplicate $500 charge on invoice 83457."
print(f" User: {user_message}")
print("─" * 60)
response = openai_client.responses.create(
model=model_deployment,
instructions=agent.definition.instructions,
tools=[{
"type": "function",
"name": "process_refund",
"description": "Process a refund for a customer invoice.",
"parameters": {
"type": "object",
"properties": {
"invoice_id": {"type": "string", "description": "The invoice ID"},
"amount": {"type": "number", "description": "The refund amount in dollars"}
},
"required": ["invoice_id", "amount"],
"additionalProperties": False,
},
"strict": True,
}],
input=user_message,
)
# Inspect what the agent did
print("\n Traditional Monitoring View:")
print(f" HTTP Status: 200")
print(f" Exception Count: 0")
print(f" Response Time: ~2-4s")
print(f" Status: SUCCESS")
print("\n Agent Output:")
for item in response.output:
if item.type == "function_call":
args = json.loads(item.arguments)
print(f"\n Tool Called: {item.name}")
print(f" Invoice ID: {args.get('invoice_id')}")
print(f" Amount: ${args.get('amount')}")
# Check if the amount is correct
expected_amount = 500.0
actual_amount = args.get('amount', 0)
if actual_amount != expected_amount:
print(f"\n PARAMETER ERROR DETECTED!")
print(f" Expected: ${expected_amount}")
print(f" Actual: ${actual_amount}")
print(f" Loss: ${abs(actual_amount - expected_amount)}")
else:
print(f"\n Parameters correct")
elif item.type == "message":
print(f" Response: {item.content[0].text if item.content else 'N/A'}")
This is the whole point of observability: the HTTP status, exception count, and latency all look healthy, while the argument check is the only thing that reveals whether the refund amount is right.
The all-green moment:
![]()
Step 3: Classify the failure across the five layers
When the amount is wrong, the workshop classifies the same interaction across all five layers—so the "green check" and the real failure sit side by side:
# Visualize the five-layer assessment print("═" * 65) print(" FIVE-LAYER AI OBSERVABILITY ASSESSME
# Visualize the five-layer assessment
print("═" * 65)
print(" FIVE-LAYER AI OBSERVABILITY ASSESSMENT")
print("═" * 65)
print()
layers = [
("1. Technical", "PASS", "HTTP 200, 0 errors, 2.8s latency"),
("2. Agent Execution", "FAIL", "Wrong parameter: $5000 instead of $500"),
("3. Safety & Policy", "FAIL", "Approval threshold bypassed"),
("4. Quality", "FAIL", "Task executed incorrectly"),
("5. Business", "FAIL", "$4,500 financial loss"),
]
for layer, status, detail in layers:
print(f" {layer:<22} {status:<10} {detail}")
print()
print("─" * 65)
print(" Traditional APM verdict: ALL GREEN")
print(" AI Observability verdict: CRITICAL FAILURE")
print("─" * 65)
Traditional monitoring only covers Layer 1. The remaining sections show how to detect, evaluate, monitor, and prevent the other four.
Step 4: Turn on tracing and read the run as a story
Server-side traces become available after Application Insights is connected to the project. Client-side tracing adds visibility into the application logic around the agent call. The workshop enables both with a few lines: it turns on content recording, configures Azure Monitor, and gets a tracer.
# Enable experimental GenAI tracing and content capture before instrumentation.
os.environ["AZURE_EXPERIMENTAL_ENABLE_GENAI_TRACING"] = "true"
os.environ["OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT"] = "true"
from azure.ai.projects.telemetry import AIProjectInstrumentor
from azure.monitor.opentelemetry import configure_azure_monitor
from opentelemetry import trace
configure_azure_monitor(connection_string=app_insights_conn)
AIProjectInstrumentor().instrument(
enable_content_recording=True,
enable_trace_context_propagation=True,
enable_baggage_propagation=False,
)
tracer = trace.get_tracer("foundry-observability-blog")
print("Azure Monitor configured")
print("Foundry GenAI instrumentation enabled")
print("Content recording enabled")
print("Tracer initialized")
Wrapping an agent interaction in a custom parent span ties the model call and each tool call into one ordered trace. Here the workshop wraps a customer-service session, dispatches the tool the model requested, and returns the result with previous_response_id so the follow-up model call joins the same conversation:
from openai.types.responses.response_input_param import FunctionCallOutput
# Run the agent with tracing
with tracer.start_as_current_span("customer_service_session") as session_span:
session_span.set_attribute("session.type", "customer_inquiry")
session_span.set_attribute("customer.segment", "premium")
# Initial model call
response = openai_client.responses.create(
model=model_deployment,
instructions=agent.definition.instructions,
tools=[{
"type": "function",
"name": t.name,
"description": t.description,
"parameters": t.parameters,
"strict": True,
} for t in tools],
input=user_message,
)
# Process function calls, return tool output, then continue the conversation
input_items = []
for item in response.output:
if item.type == "function_call":
result = {"status": "processed", "arguments": json.loads(item.arguments)}
input_items.append(
FunctionCallOutput(
type="function_call_output",
call_id=item.call_id,
output=json.dumps(result),
)
)
if input_items:
final_response = openai_client.responses.create(
model=model_deployment,
instructions=agent.definition.instructions,
input=input_items,
previous_response_id=response.id,
)
# Record the trace ID for investigation
span_context = session_span.get_span_context()
trace_id = format(span_context.trace_id, '032x')
print(f" Trace ID: {trace_id}")
print(f" View in Application Insights → Transaction Search → {trace_id}")
The resulting trace reads as a sequence, not a pile of logs:
Trace anatomy:
![]()
![]()
![]()
Graph view of Conversation id:
![]()
End-to-end investigation in Application Insight
![]()
![]()
![]()
Application Insight- Agents (Preview)
![]()
Agents Preview — Models & Tools
![]()
Application Insight captures the platform where the agent lives (foundry,copilot studio etc)
![]()
Application Insight - failures
![]()
Step 5: Ask custom questions with KQL
The portal is an excellent starting point, but customers eventually ask questions specific to their business. The workshop's cookbook targets a Log Analytics workspace and uses the workspace tables (AppDependencies, AppEvents). The legacy aliases (dependencies, customEvents) only resolve in an Application Insights resource query context—use the schema that matches your query scope.
Tool failure rate:
AppDependencies
| where DependencyType == "GenAI"
| where Name == "ExecuteTool"
| summarize
Total = count(),
Failed = countif(Success == false),
FailPct = round(100.0 * countif(Success == false) / count(), 2)
by bin(TimeGenerated, 1h)
| render timechart
Token consumption over time:
AzureMetrics
| where MetricName in ("InputTokens", "OutputTokens", "TotalTokens")
| summarize Tokens = sum(Total)
by MetricName, bin(TimeGenerated, 1h)
| render timechart
Follow one trace end to end:
// Replace <operation-id> with the actual OperationId let TraceId = "<operation-id>"; AppDependencies | where OperationId == TraceId | project TimeGenerated, Name, DurationMs, Success, ResultCode, OperationId, ParentId, Properties | order by TimeGenerated asc
// Replace <operation-id> with the actual OperationId
let TraceId = "<operation-id>";
AppDependencies
| where OperationId == TraceId
| project
TimeGenerated,
Name,
DurationMs,
Success,
ResultCode,
OperationId,
ParentId,
Properties
| order by TimeGenerated asc
Correlate human/evaluation feedback with responses:
AppEvents | where Name == "gen_ai.evaluation.result" | extend responseId = tostring(Properties["gen_ai.response.id"]), score = todouble(Properties["gen_ai.evaluation.score.value"]), label = tostring(Properties["gen_ai.evaluation.score.label"]), source = tostring(Properties["microsoft.gen_ai.human_evaluation.source"]) | project TimeGenerated, responseId, score, label, source | order by TimeGenerated desc
AppEvents
| where Name == "gen_ai.evaluation.result"
| extend
responseId = tostring(Properties["gen_ai.response.id"]),
score = todouble(Properties["gen_ai.evaluation.score.value"]),
label = tostring(Properties["gen_ai.evaluation.score.label"]),
source = tostring(Properties["microsoft.gen_ai.human_evaluation.source"])
| project TimeGenerated, responseId, score, label, source
| order by TimeGenerated desc
Telemetry schemas continue to evolve. Before using a query in production, inspect a representative span and confirm the table and attribute names emitted by the SDK version in use.
Custom KQL:
![]()
Step 6: Evaluate the process — did the agent pass the right parameters?
Tracing explains how an action happened. Evaluation measures whether it was correct, consistently. For a refund agent, the most direct check is process evaluation: did it select the right tool and pass the right arguments?
The following example runs builtin.tool_call_accuracy over a set of banking scenarios—including a process_refund case—using the Evals API:
import json
# Banking tool definitions
banking_tools = [
{
"type": "function",
"name": "get_account_balance",
"description": "Retrieve the current balance for a customer account.",
"parameters": {
"type": "object",
"properties": {
"account_number": {"type": "string", "description": "Account number (e.g., CHK-12345)"}
},
},
},
{
"type": "function",
"name": "process_refund",
"description": "Process a refund to a customer account.",
"parameters": {
"type": "object",
"properties": {
"invoice_id": {"type": "string"},
"amount": {"type": "number"}
},
},
},
]
# Test scenarios with expected tool calls
# Note: ToolCallAccuracyEvaluator requires a `tool_call_id` on every tool_call item.
scenarios = [
{
"query": "What's the balance in account CHK-12345?",
"tool_definitions": banking_tools,
"tool_calls": [{
"type": "tool_call",
"tool_call_id": "call_1",
"name": "get_account_balance",
"arguments": {"account_number": "CHK-12345"}
}],
},
{
"query": "Please refund $75 on invoice INV-9876.",
"tool_definitions": banking_tools,
"tool_calls": [{
"type": "tool_call",
"tool_call_id": "call_3",
"name": "process_refund",
"arguments": {"invoice_id": "INV-9876", "amount": 75}
}],
},
]
print(f"{len(scenarios)} test scenarios defined")
import time
# Prepare test data — file_content expects an array of {"item": {...}} objects
test_content = [{"item": s} for s in scenarios]
testing_criteria = [
{
"type": "azure_ai_evaluator",
"name": "tool_accuracy",
"evaluator_name": "builtin.tool_call_accuracy",
"initialization_parameters": {"deployment_name": model_deployment},
"data_mapping": {
"query": "{{item.query}}",
"tool_definitions": "{{item.tool_definitions}}",
"tool_calls": "{{item.tool_calls}}",
},
},
]
data_source_config = {
"type": "custom",
"item_schema": {
"type": "object",
"properties": {
"query": {"type": "string"},
"tool_definitions": {"type": "array"},
"tool_calls": {"type": "array"},
},
"required": ["query", "tool_definitions", "tool_calls"],
},
}
eval_object = openai_client.evals.create(
name="Banking Agent — Tool Call Accuracy",
data_source_config=data_source_config,
testing_criteria=testing_criteria,
)
eval_run = openai_client.evals.runs.create(
eval_id=eval_object.id,
name="Tool Accuracy Run",
data_source={
"type": "jsonl",
"source": {"type": "file_content", "content": test_content},
},
)
print(f"Evaluation run started: {eval_run.id}")
while eval_run.status not in ["completed", "failed"]:
time.sleep(5)
eval_run = openai_client.evals.runs.retrieve(
run_id=eval_run.id, eval_id=eval_object.id
)
print(f" Status: {eval_run.status}")
print(f"\n{'' if eval_run.status == 'completed' else '' } Final: {eval_run.status}")
The workshop also runs a quality-and-safety suite (builtin.violence, builtin.fluency, builtin.task_adherence) against a registered agent using evals.create and azure_ai_target_completions. Together, these give both a process signal (right tool, right arguments) and a quality/safety signal.
One honest caveat for publication: in the workshop's saved run, the tool-call records are hand-authored fixtures, not calls captured from a live agent. For a production regression test, capture the agent's actual tool call and its tool_call_id, then evaluate that. For a refund, the exact-amount and approval checks belong in the transaction path as deterministic controls—an LLM judge should not be the only guard.
Evaluation
![]()
At first glance, this evaluation run appears healthy: the run completed successfully and achieved 80% tool accuracy. However, the final test case tells a different story. The agent failed the duplicate-refund scenario, receiving a tool-accuracy score of 2 and a result of 0/1. This illustrates why aggregate scores and completion status alone are insufficient—production observability must make individual failures easy to identify and investigate.
A completed evaluation run can still contain a critical failure. The final refund scenario failed tool-call accuracy despite an overall score of 80%.
![]()
Step 7: Red-team the agent before it ships
Quality evaluation asks whether expected tasks succeed. Red teaming asks whether deliberate adversarial pressure pushes the agent past its boundaries. The workshop runs the AI Red Teaming Agent in the cloud: it registers a Foundry agent target, generates a prohibited-actions taxonomy, wires up agentic evaluators, and submits a run with attack strategies.
from azure.ai.projects.models import (
AzureAIAgentTarget,
AgentTaxonomyInput,
EvaluationTaxonomy,
RiskCategory,
)
# Target descriptor referenced by both the taxonomy and the run.
target = AzureAIAgentTarget(name=agent.name, version=agent.version)
# Foundry generates the attack-prompt taxonomy from the agent's tools & instructions.
taxonomy = project_client.beta.evaluation_taxonomies.create(
agent.name,
EvaluationTaxonomy(
description="Taxonomy for banking agent red teaming",
taxonomy_input=AgentTaxonomyInput(
risk_categories=[RiskCategory.PROHIBITED_ACTIONS],
target=target,
),
),
)
taxonomy_file_id = taxonomy.id
# The red team groups one or more runs and wires up the built-in agentic evaluators.
red_team = openai_client.evals.create(
name="Red Team — Banking Agent Safety",
data_source_config={"type": "azure_ai_source", "scenario": "red_team"},
testing_criteria=[
{
"type": "azure_ai_evaluator",
"name": "Prohibited Actions",
"evaluator_name": "builtin.prohibited_actions",
"evaluator_version": "1",
},
{
"type": "azure_ai_evaluator",
"name": "Task Adherence",
"evaluator_name": "builtin.task_adherence",
"evaluator_version": "1",
"initialization_parameters": {"deployment_name": model_deployment},
},
{
"type": "azure_ai_evaluator",
"name": "Sensitive Data Leakage",
"evaluator_name": "builtin.sensitive_data_leakage",
"evaluator_version": "1",
},
],
)
# Create the red-team run — it executes server-side in Foundry.
eval_run = openai_client.evals.runs.create(
eval_id=red_team.id,
name="Banking Agent Red Team Run",
data_source={
"type": "azure_ai_red_team",
"item_generation_params": {
"type": "red_team_taxonomy",
"attack_strategies": ["Flip", "Base64", "IndirectJailbreak"],
"num_turns": 5,
"source": {"type": "file_id", "id": taxonomy_file_id},
},
"target": target.as_dict(),
},
)
print(f"Run created: {eval_run.id} status={eval_run.status}")
print(" View in Foundry → Build → Evaluations → Red team")
The strategies (Flip, Base64, IndirectJailbreak) select input transformations and multi-turn depth; they test whether the agent resists manipulation, stays on task, and does not leak data. Two things matter for an honest write-up:
- Review the taxonomy before scanning. Confirm the generated prohibited behaviors reflect your policy. Run red teaming against an isolated target with synthetic accounts and no irreversible side effects.
- Distinguish a security result from an infrastructure failure. A run that ends in failed is incomplete coverage, not a passing scan. Capture the run status honestly, and only present an attack-success-rate scorecard from a completed run.
Red-team run: Capture the run status and, when the run completes, the scorecard in Foundry → Build → Evaluations → Red team. If the run failed, caption it as incomplete coverage rather than a pass.
Step 8: Prove the fix and move to production monitoring
For the sample, the correction is a tightened prompt plus a deterministic amount/approval check in the transaction path—so the agent is not expected to reason its way out of receiving or emitting the wrong amount. After a fix, compare the two versions across the same evaluation set:
| Signal | Before the fix | After the fix |
|---|
| Request success | 100% | 100% |
| Correct refund amount | Fail | Pass |
| Approval threshold honored | Fail | Pass |
| Tool-call accuracy | Fail | Pass |
The operational success rate does not change. The business outcome does.
Once you know how to detect the failure, the next question is whether it is happening elsewhere. The Foundry Monitor experience brings operational metrics, evaluation results, and red-team results together for the selected agent; Application Insights adds deeper investigation across runs, models, tools, tokens, and errors. For this refund agent, I would monitor at least:
- Run success rate and model or tool errors
- P50 and P95 end-to-end latency
- Input and output token consumption
- Refund-amount and approval-threshold pass rate
- Tool-call-accuracy distribution
- Human escalation and customer correction rate
- Safety and adversarial-testing findings
An alert should lead to an operational action:
| Signal | Example condition | Action |
|---|
| Wrong-amount refund | Any run proposes an amount outside the approved range | Block the version and notify the finance/policy owner |
| Quality regression | Tool-call-accuracy pass rate falls below the release threshold | Stop promotion or roll back |
| Token anomaly | Tokens per run rise materially above baseline | Inspect context growth and repeated tool calls |
| Latency | P95 exceeds the agreed service objective | Inspect model, tool, and throttling spans |
| Safety | A scheduled red-team or production safety check fails | Route to the security and responsible-AI process |
A KQL query like the tool-failure-rate example can become the signal for an Azure Monitor log alert. Configure the rule to trigger when the query returns results in the evaluation window, and route the notification through the customer's approved action group.
Continuous evaluation is the bridge between deployment and learning
A one-time evaluation protects one release. It does not protect the agent indefinitely. Production traffic changes, tools change, and models and prompts are revised. This makes observability a continuous engineering loop:
- Observe production behavior.
- Diagnose representative traces.
- Evaluate the failure mode.
- Add that failure to the regression dataset.
- Correct the prompt, tool, policy, or model configuration.
- Compare the candidate with the approved baseline.
- Promote only when the required quality, safety, operational, and business gates pass.
Microsoft Foundry supports recurring and continuous evaluation. When using live production traces, sample intentionally—random sampling gives coverage, while targeted or intelligent sampling can prioritize unusual, high-risk, failed, expensive, or low-quality interactions. Evaluation calls and telemetry have cost, privacy, and retention implications, so the sampling strategy should reflect business risk.
Security, privacy, and cost are part of observability design
Tracing can capture prompts, model outputs, tool arguments, and tool results. That visibility is valuable, but it creates responsibility:
- Keep message-content recording disabled unless there is an approved need.
- Never place credentials, tokens, or secrets in prompts or span attributes.
- Avoid storing personal data when a pseudonymous transaction identifier is sufficient.
- Redact or minimize sensitive content before telemetry is emitted.
- Apply Azure RBAC to Application Insights and Log Analytics; set retention by environment and data classification.
- Monitor ingestion and evaluation cost, not only model-token cost.
- Validate preview capabilities against production requirements.
There is also a practical instrumentation lesson: do not add every available value to every span. Capture the dimensions needed to investigate reliability, behavior, quality, security, cost, and business outcomes. More telemetry is not automatically better telemetry.
What the customer gained
The most valuable outcome was not another dashboard. It was a shared way for developers, platform engineers, finance/policy owners, security teams, and business stakeholders to discuss the same agent run with evidence.
The developer could see the exact execution path. The platform team could see latency, errors, and token consumption. The policy owner could see the refund amount and whether the approval threshold held. The security team could verify how sensitive telemetry was handled. The business sponsor could see whether customers were refunded correctly.
The original request had a green check before the investigation, and it had a green check afterward. The difference was that after the fix, we could prove the refund amount was correct.
Final takeaway
An agent should not be considered healthy merely because it responds. A production-ready observability practice should be able to answer:
- Did the agent complete the request?
- What model, tool, and path did it use, and with what arguments?
- Was its action relevant, grounded, safe, and correct?
- Did it comply with the approved business process?
- Can the team detect a regression before it affects more users?
- Can the team prove that the remediation improved the outcome?
In the agentic era, observability is not the dashboard at the end of deployment. It is the evidence system connecting design decisions, production behavior, governance controls, and measurable business value.
References
Contributors:
This article is maintained by Microsoft. It was originally written by the following contributors.
Gaurav Bhardwaj | Senior Cloud Solution Architect
Log in
