Observability and tracing

An agent run is rarely a single model call. The path from a user question to a final answer can branch across tools, retries, nested calls, and long prompts. When the answer is wrong, slow, or expensive, you need to see each step in order—not guess from the final text alone.

Microsoft Foundry observability records what happened during agent runs: inputs and outputs, tool usage, retries, latency, token use, and errors. Tracing is the detailed record of one run. You inspect traces in the Microsoft Foundry portal and in the Application Insights resource linked to your Foundry project. Concept 8 introduced measuring quality with evaluations; tracing shows why a run behaved the way it did before you score it.

• Tracing is generally available for prompt agents and hosted agents.
• Tracing for workflow agents and external agents is in preview.
• Foundry follows OpenTelemetry semantic conventions so trace shape stays consistent across supported tools and integrations.
• Telemetry is stored in Azure Monitor Application Insights; additional Application Insights pricing may apply based on data volume and your retention settings.

Think of a trace as the full story of one request through your agent. A span is one operation inside that story—such as invoking the agent, calling a tool, or running a model step. Spans can nest, so you see which tool ran inside which agent step. An attribute is a named piece of metadata on a trace or span, such as parameters, timing, or custom labels you add for search.

Multi-agent setups use spans such as invoke_agent, invoke_workflow, execute_tool, and plan, with child spans for nested work. Microsoft contributes multi-agent conventions with Cisco Outshift; they build on OpenTelemetry GenAI agent spans and W3C Trace Context. The OpenTelemetry GenAI semantic conventions themselves have Development status and may change in future releases.

• Use consistent span attributes across agents and tools so queries stay simple.
• Correlate evaluation run IDs with trace data when you want quality and performance in one view.
• Redact secrets, credentials, and personal data from prompts, tool arguments, and attributes before they reach telemetry.

Tracing is off by default. No trace data is collected until a Foundry Account Owner or Foundry Owner connects telemetry. When you connect an Application Insights resource to the project, Foundry begins collecting traces for agents in that project.

• Sign in to the Microsoft Foundry portal and turn on the New Foundry toggle.
• Open your Foundry project. In the left navigation, select Agents, then select Traces at the top.
• On the right, select Connect to create a new Application Insights resource or connect an existing one.
• If Connect is missing, select Manage, then Project details, open the Connected resources tab, select Add connection, choose Application Insights, and finish the wizard.
• Assign the Log Analytics Reader role on the connected Application Insights resource so you can query traces in the portal. If Log Analytics tables are protected, also assign Privileged Monitoring Data Reader.
• Run the agent once, open the Traces view, and confirm a new trace appears. Ingestion can take a few minutes; refresh if needed.

The recommended starting point is server-side tracing. After Application Insights is connected, Foundry can log traces for common agent and workflow scenarios without code changes. Traces typically show up within minutes for agents hosted in Foundry.

To stop new collection, disconnect or remove the Application Insights connection. Previously collected data remains under your Application Insights and Log Analytics retention settings—not a separate Foundry-only store.

Client-side tracing is marked preview in Microsoft documentation. You install OpenTelemetry packages, opt in to GenAI tracing (for example AZURE_EXPERIMENTAL_ENABLE_GENAI_TRACING for Python), and export to Application Insights using the project's connection string or Entra-based ingestion. Include agent_reference with name and id in responses.create when you need the portal to tie spans to a specific agent.

• Content recording captures message bodies and tool arguments; keep it off in production unless policy allows it.
• For Entra-authenticated ingestion (preview), set Auth type to Project managed identity when you connect Application Insights and assign Monitoring Metrics Publisher to identities that emit telemetry.
• Trace context propagation links client spans with server spans; disable it only when compliance rules forbid sharing trace identifiers.

Imagine a prompt agent for an online shop that answers order questions and calls a refund tool. A customer asks, "Was my return approved?" The agent replies with the wrong status. In the Foundry project, open the agent, select Traces, and open the trace for that conversation.

• Walk the span tree top to bottom: user input span, invoke_agent, then execute_tool for the refund lookup.
• Compare gen_ai.tool.call.arguments and gen_ai.tool.call.result on the tool span with the final answer span.
• If latency spiked, check duration bars on nested spans to see whether the model or the tool was slow.
• Open Trace Replay (preview), switch between Trajectories and User tabs, and use Playthrough to step through the run.
• Add a thumbs-down annotation (preview) or log end-user feedback (preview) on the trace so the team can filter similar failures later.

Single traces help with one failure. The Agent Monitoring Dashboard (preview) summarizes production traffic on an agent's Monitor tab: token usage, latency, run success rate, evaluation metrics, and red teaming results when configured. Retention and billing for monitoring data follow your Application Insights configuration.

On the Traces page you can also view Conversation results: search by Response ID or Trace ID, open a Conversation ID, and review history, token use per run, ordered tool steps, and inputs and outputs. That pairs with trace replay when you need both a chat view and a span hierarchy.

• Convert traces into evaluation datasets from the Data Generation tab (Create dataset > From traces) or by selecting traces and choosing Add to dataset.
• Intelligent sampling (when you set max_samples from 1 through 1,000) picks a diverse subset instead of dumping every low-signal trace into a dataset.
• Annotate traces in the portal with thumbs up or thumbs down (preview), or emit gen_ai.evaluation.result events from your app for end-user feedback (preview).
• Link trace IDs to evaluation runs so Concept 14 evaluators can score the same production behavior you already captured.

Traces can hold user prompts, model outputs, tool arguments, and system instructions. Project members with Log Analytics Reader on the connected Application Insights resource can read that data, which may include customer content. Treat traces like production logs: limit who gets read access, and decide what to record before you enable tracing broadly.

• Do not store secrets, credentials, or tokens in prompts, tool arguments, or span attributes.
• Redact or minimize personal data before it enters telemetry.
• Restrict access to sensitive content in traces (preview) routes GenAI message attributes to a dedicated AppGenAIContent table you can mark Protected and gate with Privileged Monitoring Data Reader.
• Disable tracing by disconnecting Application Insights when you no longer want new collection; old data follows Application Insights retention.

Sampling and archive behavior for stored telemetry follow Application Insights and Log Analytics settings—not a fixed period stated in Foundry docs. Tracing is available in regions where Foundry is supported. Watch ingestion delay: client-side spans often appear in the portal after roughly two to five minutes, and Application Insights may need thirty to ninety seconds before trace-to-dataset jobs see new spans.

• No traces in the portal: confirm Application Insights is connected, generate new agent traffic, wait a few minutes, and refresh.
• Authorization errors when opening traces: assign Log Analytics Reader on Application Insights; add Privileged Monitoring Data Reader if tables are protected.
• Client-side spans missing: verify GenAI tracing is enabled, packages are installed, and the connection string or Entra ingestion roles are correct.
• Evaluators see empty conversations: ensure invoke_agent spans include gen_ai.input.messages and gen_ai.output.messages per GenAI conventions.
• Enabling content recording in production without review: message bodies and tool args may violate privacy rules—keep recording for development unless policy allows it.

Next, use Concept 14 to run evaluations and red teaming on the datasets and traffic patterns you surfaced here. Keep tracing enabled in non-production projects while you iterate on tools and prompts from Concepts 7 and 12, then apply the same connection and access model before production rollout.

• Start with the platform overview of agent tracing, OpenTelemetry usage, and multi-agent span names. Agent tracing overview
• Follow portal steps to connect Application Insights and verify traces appear. Set up tracing in Microsoft Foundry
• Read default-off behavior, storage location, and customer responsibilities for trace data. Tracing and data handling
• Use the combined checklist when traces or trace-based evaluations fail. Troubleshoot evaluation and observability issues