Measure and improve quality

Evaluation is how you measure the quality, safety, and reliability of a model or agent while you build it and after it is live. In Microsoft Foundry you can score answers that already sit in a test dataset, run fresh answers from a model or agent against test inputs, or score interactions captured from production traces. An evaluation applies one or more evaluators to that data. Evaluators return metrics and, when supported, short explanations so you can compare versions, spot weak spots, and decide whether the agent is ready to deploy.

Concept 7 showed how to wire tools and knowledge into a prompt agent. This concept answers the next question: how do you know the agent actually helps customers instead of guessing? Playground chats feel convincing, but they are a small sample. Evaluation turns "it looks fine" into numbers you can repeat on every change.

• Score stored query–response pairs without calling the model again.
• Generate new responses from test inputs, then score them.
• Pull production traces and evaluate real conversations or turns.
• Simulate multi-turn behavior before deployment when you need scripted scenarios.

Most evaluation workflows in Foundry follow the same rhythm. You decide what slice of behavior matters, pick metrics that match that goal, run the job, then act on the results. You can run once during development, hook runs into a deployment pipeline, or set up recurring evaluation on production traffic when you are ready for that level of monitoring.

• Choose what to evaluate: a single turn, a full conversation, an existing dataset, generated responses, simulated conversations, or production traces.
• Select evaluators: quality and safety metrics such as task completion, relevance, groundedness, similarity, or content safety.
• Run the evaluation: on demand during development, in continuous integration, or on a schedule for live agents.
• Review and improve: read aggregate metrics and individual failures, compare versions, and fix patterns that keep failing.

Before your first run, check permissions for evaluation workflows, which regions and rate limits apply, and any virtual network requirements listed in the evaluation documentation. Missing any of those shows up as blocked buttons or failed jobs rather than bad scores.

Choose the evaluation unit before you pick evaluators. Turn-level evaluation scores one response at a time. That fits debugging a single answer, tool choice on one step, or a row in a query–response dataset. Conversation-level evaluation scores a complete multi-turn interaction and can measure outcomes that span several steps, such as finishing a booking. Each evaluator supports specific levels, so a metric that works on one turn might not apply to a whole thread.

Start from built-in evaluators, or create a custom evaluator when you need your own rubric. Common built-in quality names you will see include task completion, relevance, groundedness, and similarity, plus content safety checks where individual-turn scope applies. Use the set that matches what "good" means for your app, not every checkbox on the list.

Prompt Optimizer in the Microsoft Foundry portal rewrites an agent's system instructions using prompt-engineering practices. It restructures and clarifies text, shows paragraph-level reasoning for each change, and lets you add suggestions and optimize again until you are satisfied. Apply the result with one click when you accept it.

• Sign in to Microsoft Foundry with the New Foundry toggle on.
• Go to Build > Agents and open the agent you want to improve.
• In the agent configuration panel, open the Instructions section.
• Select the pencil with sparkle icon next to the Instructions header to open the Prompt Optimizer dialog.
• Enter a description or optional suggestion, select Optimize, review purple-highlighted changes, then select Use prompt to copy the text into Instructions.

Prompt Optimizer works on text instructions only. It does not rewrite tool definitions or knowledge sources. Results are not stored if you close the dialog without selecting Use prompt, and the optimizer does not keep version history—copy your old instructions first if you might revert. The feature is available only when your Foundry project is hosted in a supported region; in other regions the Optimize control does not appear. After you apply changes, test in the playground, then run a full evaluation on your own dataset to confirm the rewrite helps your cases, not just generic best practices.

When you need proof on real tasks, use the optimization wizard on a prompt agent's Optimize tab. The agent optimizer evaluates a chosen agent version, generates alternative configurations called candidates, scores them with the same dataset and evaluators, and ranks them against your baseline. For prompt agents it can improve instructions, function-calling tool descriptions, and model selection. Other tool types are not supported for tool-description optimization. Function-calling tools run on the client side, so the optimizer can refine descriptions that guide calls but cannot execute those tools during tool-description optimization.

• Open your project, select Agents, choose the prompt agent, then select the Optimize tab and Create optimization run.
• Target step: pick the baseline agent version, optimization model, maximum candidates, and evaluation model; optionally turn on Compare across models.
• Dataset step: generate from agent traces, pick a registered dataset, or upload JSONL; confirm columns match what your evaluators require because the wizard does not map columns.
• Criteria step: select built-in evaluators, an existing custom evaluator, or create a custom rubric evaluator.
• Review step: verify settings, read the estimated cost range (modeled, not a spending cap), select Submit, then compare scores when the run completes.

Imagine a prompt agent that answers order status and return policy questions for a small online shop. You already connected file search to your policy PDF in Concept 6 and added a function-calling tool in Concept 7 that looks up orders by ID. You want returns explained correctly and order lookups only when the shopper gives a valid ID.

• Build a JSONL dataset with representative rows: happy-path status checks, missing order ID, angry tone, and off-topic chit-chat you want refused.
• Run a turn-level evaluation with relevance and groundedness on policy answers, plus agent evaluators such as tool selection and tool call success on order rows.
• Note the baseline scores, then use Prompt Optimizer with a suggestion like "refuse off-topic questions and ask for order ID before lookup."
• Test in the playground, then start an optimization wizard run on the same dataset with task completion and groundedness as criteria.
• If a candidate beats the baseline on metrics that matter and token usage stays acceptable, select Promote candidate to create a new agent version; otherwise keep the baseline and fix the dataset or evaluators first.

If every candidate scores lower than the baseline, the docs advise keeping the current agent and revising the dataset, evaluators, or optimization settings before you run again. A prettier prompt in the playground is not a win until the numbers move on the tasks you care about.

Treat the version you started from as the baseline. Run the same evaluation after each meaningful change and compare aggregates plus failing rows. The agent optimizer computes a composite score between 0.0 and 1.0 by rescaling each evaluator's raw score, flipping any metric configured as lower-is-better, then averaging across evaluators and tasks. The star marks the highest-scoring candidate, but you still choose whether to promote it.

• Skipping evaluators that match your goal and chasing a high fluency score while task completion stays flat.
• Using a tiny or unrealistic dataset, then over-trusting a composite score that does not reflect live shoppers.
• Closing Prompt Optimizer without selecting Use prompt and losing the rewrite.
• Promoting a candidate with better scores but much higher token usage without checking the Token usage view on the completed run.
• Running optimization against a dataset that triggers real external API calls; use test endpoints or mocks when tools would charge or change production data.

You now have a loop: define good, measure with evaluators, improve instructions with Prompt Optimizer or the optimization wizard, and keep a baseline until scores prove the change. Concept 9 picks up architecture and rollout—how projects, deployments, and environments fit together so you can ship the version you validated instead of leaving it in the playground.

• Full evaluation workflow, approaches, and components in Foundry. Evaluation in Microsoft Foundry
• Step-by-step use of Prompt Optimizer in the portal playground. Optimize agent prompts by using Prompt Optimizer
• Portal wizard to optimize a prompt agent with a dataset and evaluators. Quickstart: Optimize a prompt agent
• How the agent optimizer scores baselines, generates candidates, and ranks results. What is the agent optimizer?