Microsoft Foundry: Models, Agents and Evaluation

Microsoft Foundry is Microsoft’s current AI app and agent platform for discovering models, building agents, attaching tools and knowledge, evaluating behavior, tracing production interactions, and governing AI resources.

Foundry is best understood as a lifecycle platform rather than one AI service. It brings models, prompt or hosted agents, tools, evaluation, observability, identity, networking, and policy into one management plane so teams can move from prototype to governed production.

Foundry Models is the model-selection layer

Microsoft Foundry Models provides a catalog with models from Microsoft, OpenAI, Anthropic, Meta, Mistral, DeepSeek, and other providers.

Teams can compare model capabilities, deployment options, regional availability, fine-tuning support, and model lifecycle before choosing a production target.

The right model should be selected from measured workload quality, latency, cost, modality, context, and safety requirements rather than from brand preference.

Model lifecycle also matters because deprecation or regional availability can affect long-term application support.

Foundry’s large catalog makes curation more important, not less. Teams should define approved model families, regions, deployment types, and evaluation requirements rather than letting every project choose independently.

The model catalog should be treated as an engineering input with lifecycle and governance implications, not as a menu where the newest model automatically wins.

Deployment type also matters. Managed compute and serverless-style options can differ in availability, control, scaling, and operational responsibility.

A model that benchmarks well but cannot meet regional, throughput, or networking requirements is not a viable production choice.

Prompt agents are the lowest-operations agent path

Foundry prompt agents are declarative: developers define instructions, select a model, and attach tools while Foundry hosts the agent runtime.

This path is appropriate when the application does not require custom orchestration code or unusual runtime dependencies.

The simplicity reduces infrastructure ownership and can speed experimentation or controlled business automation.

The tradeoff is less direct control than a fully code-defined agent, so choose the path from actual customization requirements.

Prompt agents work well when instructions and tools can express the behavior clearly and the team values managed lifecycle over custom runtime control.

Start simple and move to hosted code only when requirements demand it. Owning the agent loop adds flexibility and creates more deployment, testing, and support responsibility.

Prompt agents are also easier to standardize across teams because instructions, tools, model, and identity can be governed through the platform without every project building its own runtime.

Use this simplicity as an architecture advantage when custom code would add no business value.

Hosted agents provide full-code flexibility

Hosted agents allow teams to bring their own code, framework, or container and run it on Foundry-managed infrastructure.

That model fits applications using custom orchestration, complex dependencies, or frameworks such as Microsoft Agent Framework, LangGraph, or Semantic Kernel.

Foundry still provides managed endpoint, scaling, identity, and observability around the agent.

The architectural question is how much agent loop and runtime behavior the application genuinely needs to own.

Hosted agents should still expose clear versioning and health behavior. A containerized agent can be technically deployable and operationally opaque if the team cannot tell which code, model, or tool configuration served a request.

Use CI/CD, immutable versions, and deployment telemetry so agent-runtime flexibility does not undermine reproducibility.

Full code can bring custom memory, multi-agent orchestration, specialized libraries, or existing application frameworks into the runtime.

That flexibility should be justified by requirements because every dependency expands the testing, security, patching, and support surface.

Tools and knowledge make agents useful beyond conversation

Agents become operationally valuable when they can search approved information, call business functions, use MCP servers, or interact with other tools.

Tool contracts should be narrow, permissioned, and observable because an agent with write capability can create real business impact.

Knowledge sources should preserve user authorization and source traceability instead of becoming a shortcut around normal data controls.

The internal agentic operations material provides broader context for why tool-enabled agents need stronger governance than chat-only systems.

Tool design should expose business actions at the right level. A narrow ‘approve invoice’ function is safer and easier to evaluate than giving the model unrestricted database or shell access.

Knowledge integrations should also track source and freshness so the agent can be debugged when an answer comes from obsolete or incomplete enterprise content.

Evaluation turns model and agent quality into measurable gates

Foundry can evaluate models, agents, datasets, and traces using built-in or custom evaluators.

Current agent evaluators cover intent resolution, task adherence, task completion, navigation efficiency, tool selection, tool input accuracy, tool output use, tool success, quality, and safety dimensions.

Use representative datasets and thresholds before release, then compare versions when instructions, model, tools, or knowledge change.

Evaluation should become part of CI/CD for important agents rather than a one-time prelaunch exercise.

Keep evaluator versions and judge models pinned when comparing releases, otherwise score changes can reflect the evaluator instead of the agent.

Use task-specific rubrics for business outcomes alongside generic relevance or safety metrics. A customer-service agent and an engineering agent need different definitions of successful completion.

Observability explains what the agent actually did

Foundry observability links tracing, monitoring, evaluation, and Application Insights so teams can inspect the path from user request through reasoning, tool calls, and final result.

This is especially important for agents because the final answer can look reasonable even when the internal tool sequence was inefficient or unsafe.

Production support should preserve agent version, model, tool calls, errors, latency, and evaluation context.

The objective is not maximum logging; it is enough evidence to explain and improve behavior.

Trace evaluation can be useful after deployment because it measures real interactions rather than only curated test prompts.

Combine production traces with a stable regression dataset so the team sees both day-to-day drift and controlled before/after comparisons.

Governance and security sit in the same control plane

Foundry uses Microsoft Entra identity, RBAC, networking, content controls, and Azure Policy to govern models, agents, and tools.

A production platform should separate builder, operator, and administrative permissions and control which agents can access sensitive data or high-impact tools.

Network isolation and policy can help organizations keep enterprise AI inside approved boundaries.

Deterministic authorization remains essential even when content or safety filters are also present.

Central governance becomes more valuable as organizations deploy many agents. Teams need an inventory of owners, models, tools, data connections, identities, and approved environments.

The control plane can help create consistent boundaries while project-level settings still handle application-specific requirements.

Separate application, platform, and governance ownership

Application developers own agent instructions, tool contracts, data boundaries, and behavior tests. Platform operators own deployment environments, telemetry, capacity, incident response, and rollback. Security and risk owners define permitted data, identity policies, audit retention, and escalation rules.

These are responsibilities rather than fixed job titles. A small team may combine roles, but every production agent still needs an accountable owner and an explicit release decision. MLOps practices help teams turn evaluations, operational metrics, and versioned assets into repeatable release controls.

When an agent fails, the team should know who can change the prompt, retrieval source, model, tool permissions, or acceptance thresholds. No one should silently weaken a safety requirement merely because a demonstration improved.

Build Foundry systems as a lifecycle, not a demo

A production Foundry project should have model selection evidence, tool and data ownership, evaluation datasets, deployment controls, tracing, incident response, and a clear migration plan for model or agent changes.

Prototype speed is useful, but the platform’s enterprise value comes from making AI behavior measurable and governable after launch.

The strongest learning path is to build one agent, evaluate it, deploy it, trace it, change one component, and prove the new version is better.

Add a deliberate regression to the final lab—change the model, tool description, or instructions—then use evaluation and traces to prove the impact before redeploying.

That exercise demonstrates the core Foundry value proposition: models and agents can evolve quickly without forcing the organization to give up engineering discipline.

Document every production agent with owner, purpose, model, tool set, knowledge sources, evaluation thresholds, data classification, and rollback path.

That inventory becomes essential when dozens of agents exist and the organization needs to understand where one model or tool change could create broad impact.

Use approval and rollback gates for high-impact agents just as you would for other production software. A model or instruction change can alter business behavior without changing the surrounding application code.

That is why evaluation, tracing, and versioning should be release requirements rather than optional observability features.

For final practice, create one prompt agent with a knowledge source and one controlled tool, then build a regression dataset that tests normal tasks, ambiguous inputs, unsafe requests, and tool failure.

Deploy a second version with one deliberate change and compare task completion, tool selection, safety, latency, and trace behavior before promotion. This makes evaluation an engineering gate rather than a presentation metric.

Document the production owner, identity, network boundary, approved model, tool permissions, evaluation thresholds, and rollback version. That operational record is what turns a Foundry prototype into a supportable enterprise service.

img