Part 2 of a series. Part 1, Grounding Copilot Studio Agents with Azure AI Search and Foundry IQ, defined five grounding patterns. This article gives you a reproducible way to compare them without mixing latency boundaries, billing units, or quality signals.
Reference implementation, not a production template. This article is for architects, agent developers, and platform owners comparing retrieval patterns across Microsoft Copilot Studio, Microsoft Foundry, and Azure AI Search. It assumes familiarity with agent grounding, basic latency evaluation, and cloud-cost concepts. The sample measurements are intended for learning and experimentation. Before production use, apply the Azure Well-Architected Framework and your organization’s reliability, security, cost, data-governance, and operational requirements.
The wrong first question
A single latency chart can make the wrong retrieval architecture look like the obvious winner when its rows measure different boundaries, billing units, or quality signals. This article gives architects and agent developers a reproducible decision system that separates those variables, applies explicit quality gates, and preserves the evidence behind each recommendation.
Five ways to ground the same agent. The question is not which is fastest, but which comparisons are legitimate.
The obvious next question is “which one is fastest?” It is also the wrong first question.
Why? A five-pattern p95 chart looks decisive, but it often compares different experiments. Some paths include the Copilot Studio front door and a Direct Line network hop; others stop at the deployed-agent boundary. Some run a model on every turn; others return a document link. Some consume Copilot Credits; others incur Azure token charges. Put those values on one axis and the chart becomes a category error, not a decision.
This post presents an evidence system, not a universal winner. The worked example identifies the Hosted retrieval modes that clear this reference project’s quality gate; the broader five-pattern comparison remains incomplete until front-door latency, isolated-load evidence, confidence intervals, and billed-cost reconciliation are available. The result is a defensible decision process, not a product ranking.
What you will learn
- Separate evidence correctly: controlled experiments, load tests, and production telemetry answer different questions.
- Measure what users experience: keep front-door, deployed-agent, and component boundaries distinct.
- Compare cost honestly: report Azure token charges and Copilot Credits in separate lanes.
- Turn results into a release decision: apply deterministic quality gates, preserve provenance, and leave missing evidence visible.
Three classes of evidence that do not mix
The first discipline is to stop treating three different kinds of measurement as one.
| Evidence class | What it is | What it answers | What it must never do |
| Controlled experiment | Sequential cases run against a recorded manifest | “Which pattern/config is better, all else equal?” | Be blended with concurrent load rows |
| Load / capacity test | Concurrent traffic with a named profile and saturation signals | “How does it behave under N users?” | Be merged into sequential latency percentiles |
| Production telemetry | Observed user traffic in Application Insights, Foundry, Azure AI Search, and Azure Monitor | “What is actually happening in production?” | Be reconstructed into controlled percentiles |
Controlled experiments, load tests, and production telemetry each answer a different question. Blending them is the fastest way to make a benchmark misleading.
A latency value is only “measured” when a committed result records its manifest, sample count, environment, and measurement boundary. Anything else, including the friendly ~1–2 s and ~10–14 s labels from Part 1, is an illustration, not a finding. Keeping these classes separate is not bureaucratic; it is what stops a load-test spike from masquerading as a typical response time.
Rule 1: Measure the boundary you actually ship
Every pattern is timed at an explicit measurement boundary, and the boundaries are not interchangeable.
Pattern key: A is the Copilot Studio standard-harness search path; A2 is the GitHub Copilot harness path; B is the connected Foundry agent path; C is the deterministic lookup path; and Hosted is the Foundry hosted-agent path. Part 1 contains the full architecture and setup details.
- A, A2, and C are measured through the Copilot Studio front door over Direct Line. The timer starts when the harness sends the question and stops when the final activity arrives, capturing the complete client-visible path. It does not isolate time spent in Copilot Studio’s internal generative-answer, knowledge, REST-tool, or connected-agent steps. Every front-door lane uses the same selected model, so measured differences reflect the harness and retrieval path rather than a model change.
- B and Hosted are timed at the deployed-agent boundary: gpt-5-mini retrieves evidence, synthesizes the answer, and receives a quality grade. This boundary is useful when native evaluation cannot grade the external-agent hop from the Copilot Studio front door, but it does not represent the complete user-visible path.
That leads to the single most important rule in the whole system:
Never subtract one boundary from another and call the difference “Copilot Studio overhead.” Different boundaries are different experiments.
In production, B and Hosted still sit behind the front door too, so the user-facing latency is roughly the deployed p50 plus the same front-door overhead the A/C paths pay. We say that in words; we do not fabricate it with subtraction.
The adapter records client_wall_time_ms as measured. When the service does not expose an internal elapsed-time value, service_elapsed_time_ms is recorded as unavailable with reason NOT_EXPOSED. Treating the value as unavailable preserves the distinction between a metric that was not measured and a genuine zero-millisecond result.
A third boundary sits inside both. At the component boundary a single adapter call is timed on its own: a classic search, a knowledge-base retrieve, the deterministic lookup, or the Foundry agent and its MCP tool. The three boundaries nest, and each one answers a question the others cannot. Component runs identify where time and failures originate. The deployed-agent boundary is where the answer is synthesized and graded. Only a complete front-door run describes what the employee actually experiences.
Evidence flows outward across three boundaries: component, deployed agent, and Copilot Studio front door. Subtracting boundaries creates a result no experiment measured.
Rule 2: Two cost lanes that never add up
Cost comparisons are misleading when architectures are billed in different units.
| Lane | Patterns | Unit | Source |
| Azure per-token USD | B, Hosted (local, token-instrumented) | USD per invocation from service-reported input/cached/output tokens × a dated retail pricing profile | SDK final usage × pricing profile |
| Copilot Credits | A, A2, C, and the B/Hosted front doors | Copilot Credits consumed per interaction, the common currency across Copilot Studio capabilities | Power Platform admin center (Copilot Credit consumption reporting) |
A null per-token cost on a Copilot Studio front-door run is correct: that path is metered in Copilot Credits rather than Azure retail token dollars. Some Copilot Studio capabilities calculate Credits from token volume, but the result is still a Credit charge, and the front door does not expose a per-request token count that belongs on the Azure-dollar axis. Estimate each lane with its own meter, reconcile Azure tokens in Azure Cost Management and Credits in the Power Platform admin center, and never add the two units together.
Microsoft draws the same line we do. The published Copilot Credit rates cover the language models Copilot Studio provides and explicitly exclude bring-your-own-model configurations, including Foundry Models, which are billed separately. That is precisely Pattern B and the Hosted front door: the connected Foundry model’s tokens land on the Azure lane in addition to the Copilot Credits the front door consumes. Two lanes, stated separately.
Two cost lanes, never summed. Copilot Studio meters Copilot Credits by agent activity; Foundry model usage is priced by tokens. There is no exchange rate between these axes, so the workbench renders separate charts instead of a false total.
One more reason the Credits lane cannot be token-priced: in Copilot Studio the harness and the model are two different things. The harness is the runtime that decides when to call the model and which tools to invoke; the model is the engine you select for reasoning and generation. Patterns A, C, and the B/Hosted front doors run on the standard harness; Pattern A2 runs on the GitHub Copilot harness.
Because the model is selectable, we pin the same one on every front door and record both halves in provenance as
Neither harness belongs on the Azure per-token cost axis. Copilot Credits are the common currency for Copilot Studio capabilities, but the two harnesses differ in acquisition model, when metering starts, and what the metering covers. The standard harness uses Copilot Studio licensing or pay-as-you-go options for published agent activity. The GitHub Copilot harness uses usage-based billing that includes model tokens, tools, and harness activity. Compare quality within a platform first, record the selected harness and model in provenance, and treat a harness or model change as a confounding variable in cross-platform comparisons.
Rule 3: Deterministic gates first, model judges second
Quality is graded in two tiers, and the order matters.
- Deterministic graders are the release gates: does the answer cite the expected policy IDs, refuse when it should, and, for locator queries, return the exact source URL? These are code, not opinions, so they are reproducible and auditable.
- Model-based judges (relevance, groundedness, task adherence via the Foundry evaluators) are supplemental. They are calibrated against a small reviewed gold set and reported next to the deterministic results, never on top of them.
def citation_pass(answer_text: str, expected_policy_ids: list[str]) -> bool:
return all(pid in answer_text for pid in expected_policy_ids)
When a judge disagrees with the gold set, the gold set wins. That single rule keeps a chatty model from grading its own homework.
One contract to make runs comparable
The three rules are enforced by a normalized contract: ExperimentManifest, CaseResult, and AggregateReport, all strict, versioned (schema 1.0) Pydantic models. The manifest carries the provenance that makes a comparison legitimate:
“schema_version”: “1.0”,
“dataset_version”: “copilot-hr-policy-release-v2”,
“corpus_fingerprint”: “
“index_fingerprint”: “
“model_deployment”: “gpt-5-mini:2025-08-07”,
“pricing_profile”: “azure-gpt-5-mini-global-standard-cache-aware:2025-08-01”,
“measurement_boundary_class”: “copilot_studio_direct_line”,
“warmups”: 3,
“repetitions”: 35
}
The comparison surface fails closed: if two runs differ in dataset, corpus, index, model, retrieval mode, or measurement boundary, the tool refuses to rank them instead of drawing a tidy, misleading bar chart. Retrieval references keep stable source IDs and omit policy content by default, so nothing sensitive lands in an artifact.
That surface is a small read-only app, the Benchmark Workbench, included in the repository. It runs no experiments and calls no Azure services. It reads the committed manifests and reports and renders them under the rules in this post, which is why it can refuse a comparison rather than produce one. You can run it locally with scripts/app.sh start.
This is the app declining to answer. Two runs, incompatible scope, no ranking, and by design.
Running it is a two-line CLI. The offline smoke validates the contract with fixtures and no Azure spend:
–manifest experiments/manifests/synthetic-direct-search.json
–cases experiments/datasets/synthetic-migration-smoke.json
–fixture-responses experiments/datasets/synthetic-direct-search-responses.json
–output-dir experiments/reports/synthetic-direct-search
Fixture wall-times are contract checks, not Azure performance evidence, and the reports label them that way.
A worked example: three ways to retrieve, measured
The worked example produces one release-ready conclusion within the Hosted retrieval lane: tool and context-agentic clear this reference project’s 85% deterministic-quality gate; context-semantic does not. The broader five-pattern comparison does not support a universal ranking because front-door latency, isolated-load evidence, confidence intervals, and billed-cost reconciliation remain incomplete. The section shows what the available evidence supports without mixing comparison surfaces.
The Hosted runtime supports three retrieval modes. All three were tested against the same synthetic HR corpus and versioned release set, with 35 timed invocations per mode after declared warmups. The run used gpt-5-mini and the dated azure-gpt-5-mini-global-standard-cache-aware:2025-08-01 pricing profile. Deterministic quality was evaluated on seven release cases. Each mode also passed two security probes, one prompt-injection case and one secret-disclosure case. That small probe is not a security assurance and does not replace a production security assessment.
| Hosted retrieval mode | Retrieval type | Deterministic quality (7-case set) | p95 latency (35 runs) | Est. model cost / answer |
| tool (custom @tool hybrid search) | Classic search | 100% (7/7) | ~25.8 s | ~USD 0.0033 |
| context-semantic (built-in context provider) | Classic search | 71.4% (5/7) | ~38.2 s | ~USD 0.0031 |
| context-agentic (context provider over the Foundry IQ knowledge base) | Agentic retrieval | 100% (7/7) | ~34.7 s | ~USD 0.0032 |
Two caveats travel with the table. First, the deterministic-quality result comes from seven cases: context-semantic misses two, while tool and context-agentic pass all seven. Expand the dataset before treating that gap as stable. Second, 35 samples are not enough for a reliable extreme-tail estimate. tool has a p95 of 25.8 seconds and a p99 of 32.4 seconds, but the p99 is driven by one request. Each percentile is reported with its sample count and a small-sample warning. Without confidence intervals and a larger sample, treat these tail figures as directional evidence, not precise estimates.
Read the release gates below at the boundary each one was written for. The ≤ 30 s p95 gate is a Copilot Studio client target and does not transfer to this deployed-agent lane. The ≥ 85% deterministic-quality gate does apply here, and only tool and context-agentic clear it; context-semantic, at 71.4%, does not.
Provenance. These figures come from the committed publication bundle (decision-system-20260811, clean commit 9c94215): 35 timed invocations per mode with declared warmups, priced against the azure-gpt-5-mini-global-standard-cache-aware:2025-08-01 profile. Each mode publishes its manifest, per-case JSONL, aggregate report, and evaluation summary under experiments/reports/decision-system-20260811/hosted/. If you cannot point a number back to a committed manifest and report, it does not belong in the post.
Why the pricing profile is dated 2025-08-01. That date is not the day we ran the experiment, and the rate was not entered manually. It is the effectiveStartDate Azure publishes for these meters. The committed profile pins the GPT-5 Mini, Global Standard, East US rates used for this experiment: $0.25 per 1 million uncached input tokens, $0.025 per 1 million cached input tokens, and $2.00 per 1 million output tokens. The profile records the source query and meter identifiers used to verify those rates against the public Azure Retail Prices API.
The linked API is live, requires no authentication, and lets you verify the current published rates directly. Rates can change, and your invoice still depends on region, deployment type, batch versus standard processing, and negotiated terms. Reprice against the current API response before using these figures in a business case.
The same bundle measures the other four patterns too, but at different boundaries. That is exactly why Rule 1 keeps them off the chart above.
Patterns A, A2, and C ran through the Copilot Studio front door over Direct Line, scoring native-evaluation quality of 85.7%, 85.7%, and 71.4% on the seven-case set. All three were pinned to the same selected model, so the remaining difference is the harness: A and C on the standard harness, A2 on the GitHub Copilot harness.
Question 1: What happened at the deployed-agent boundary? Pattern B and the Hosted runtime each scored 71.4% on the seven-case aggregate using gpt-5-mini. Native front-door evaluation cannot grade the external-agent hop. Question 2: Which retrieval modes cleared the Hosted-lane quality gate? In the separate retrieval-mode experiment, tool and context-agentic each scored 100% and cleared the 85% deterministic-quality gate; context-semantic did not. These experiments use different comparison surfaces and must not be combined into one ranking. The workbench reports missing front-door A2 latency, isolated-load evidence, and billed-cost reconciliation as unavailable rather than zero.
Measuring the Hosted front door itself makes the boundary vivid. The same deployed gpt-5-mini agent, reached through Copilot Studio over Direct Line on the same selected front-door model, comes back with an end-to-end p95 of about 55 s and an 86.7% success rate (two errors, four timeouts) across 45 invocations, published as its own bundle (copilot-front-door-hosted-45-20260818). That lane is exploratory, not release-ready, because it fails both the ≤ 30 s p95 and ≥ 99% success gates below. We report it at its own boundary and never subtract it from the deployed number; the release story stays the token-priced gpt-5-mini lane above. It is a clean reminder that the front door, including its harness, its selected model, and its network hop, is part of what you ship, and it is not free.
Release gates before claims
A run becomes a recommendation only when it clears an explicit gate profile, read at the boundary it was written for. These are project acceptance criteria for this reference assistant, not Microsoft service guarantees, and you should set your own with your product and operations owners:
| Gate | Threshold (this reference project) |
| Deterministic quality | ≥ 85% |
| Copilot Studio client p95 | ≤ 30 s |
| Success rate | ≥ 99% |
| Security pass rate (prompt-injection, secret-disclosure) | 100% |
| Estimated variable model cost | ≤ USD 0.05 / request |
The workbench produces no recommendation when front-door evidence, category-level quality, confidence intervals, explicit outcome counts, security results, or variable-cost evidence is missing. A Pareto-optimal point without those gates is labeled exploratory, not a release call. That is the publication gate in software form: the system would rather say “not yet” than ship a confident wrong answer.
Being on the Pareto frontier is not the same as being releasable. The blockers are shown, not hidden.
Reuse the native tools; do not rebuild them
The evidence system complements Microsoft-native observability rather than replacing it. Application Insights and Azure Managed Grafana support production traces, dashboards, and alerts; Microsoft Foundry provides agent monitoring and cloud evaluation; Azure AI Search exposes retrieval-capacity and throttling signals; Azure Load Testing supplies concurrent-load evidence; and Azure Cost Management provides billed-cost data. Some agentic-retrieval capabilities are generally available through the 2026-04-01 REST API, while the Azure portal and Microsoft Foundry portal provide preview access to the full feature set. Some agent-level monitoring experiences may also be in preview or vary by region. Check the linked documentation for current availability.
The workbench normalizes just enough to make a cross-product decision, then deep-links out to those systems for root-cause work. It does not copy their detail views or pretend to be a trace explorer.
What this does not claim (yet)
Three evidence gaps block a broader release recommendation: category-level quality, confidence intervals, and explicit failure counts are not represented in the aggregate contract; connected-production and isolated-load evidence are pending; and billed-cost reconciliation against Azure Cost Management is pending until a declared allocation rule can attribute shared resources. Until those gaps close, the workbench keeps the affected comparisons exploratory.
None of these gaps are fudged into the charts. Missing stays missing, and the affected comparison remains exploratory until the evidence is complete.
Try it yourself
You have a few ways to go deeper, depending on where you are:
- Start with the architecture. If you have not read Part 1, Grounding Copilot Studio Agents with Azure AI Search and Foundry IQ defines the five patterns this post benchmarks.
- Clone the repo and run the offline contract smoke. Everything described here (the contracts, deterministic graders, CLI, and workbench) lives in foundry-copilot-hr-policy-knowledge. The fixture-backed smoke needs no Azure spend and prints the normalized manifest, case, and aggregate reports on your own machine.
- Open the evidence in the workbench. Run scripts/app.sh start and browse to http://127.0.0.1:5174. It reads the committed bundles with Azure calls disabled, so you can inspect every figure in this post, and watch the Compare view refuse an invalid ranking, without spending anything. See the Benchmark Workbench guide.
- Benchmark your own agents. Stand up five non-production Copilot Studio agents, one per pattern, pin the same model on each, then follow the Direct Line benchmarking guide so every measurement exposes exactly one retrieval path.
- Go deeper on the method in Microsoft Learn: Azure AI Search agentic retrieval, Foundry IQ, and Microsoft Agent Framework.
Start here: clone the repository and run the offline contract smoke. It requires no Azure spend and shows how the manifests, gates, and reports fit together. Next, adapt one manifest to your architecture and collect a controlled baseline before you run load tests or interpret production telemetry. In the comments, share which evidence class carries the most weight in your architecture reviews—and which evidence your team struggles most to collect.
References
Series
- Part 1: Grounding Copilot Studio Agents with Azure AI Search and Foundry IQ
- Azure Well-Architected Framework
Microsoft Copilot Studio
- Microsoft Copilot Studio overview
- Choose a harness: GitHub Copilot, standard, and Copilot chat
- Copilot Credits billing rates and management
- Standard harness licensing
- Manage Copilot Studio credits and capacity (Power Platform admin center)
- Copilot Studio analytics overview
- Direct Line channel
Microsoft Foundry
- Microsoft Foundry overview (Agent Service, evaluation, tracing)
- Agent evaluators (relevance, groundedness, task adherence)
- Agent tracing and monitoring
- Microsoft Foundry Agent Monitoring dashboard
Retrieval, observability, cost, and load testing
- Azure AI Search agentic retrieval overview and availability
- Model Context Protocol (MCP)
- Application Insights Agent details view
- Azure Monitor and Log Analytics (KQL diagnostics)
- Azure Load Testing (capacity lane)
- Azure Cost Management (billed-cost reconciliation)
- Azure Retail Prices API (live, unauthenticated rate lookup)
- Azure OpenAI pricing
- OpenTelemetry GenAI semantic conventions
- Locust (load generation)
Reference implementation
- Repository root
- Benchmark Workbench guide (the decision app)
- Benchmark decision system
- Pattern setup and benchmark guide
- Pattern A2 wiring (Copilot Studio + Foundry IQ)
- Pattern B (Foundry Agent Service + MCP)
- Native evaluation
- Load testing (kept separate from controlled runs)
- End-to-end setup
- Infrastructure (all Azure resources)
Ask HR is a development and learning sample. Before production deployment, apply authentication, monitoring, data governance, content safety, compliance, reliability, and cost-management controls that match your organization’s requirements and the Azure Well-Architected Framework.