RAG on Azure: Grounding and Retrieval

Retrieval-augmented generation on Azure increasingly centers on Azure AI Search and Microsoft Foundry. Azure AI Search supports classic RAG and agentic retrieval, while Foundry IQ supports permission-aware knowledge bases when the connected knowledge sources, access metadata and query-time user identity are configured to enforce those permissions. The AI-103 exam is the current Microsoft role target for developers building AI apps and agents that use these retrieval patterns.

A good Azure RAG system is not defined by the foundation model alone. Source quality, chunking, vectorization, semantic and keyword retrieval, access control, query rewriting, grounding, citations, evaluation, and observability determine whether the system can be trusted.

Start with authoritative content and ownership

Before indexing anything, identify which sources are authoritative, who owns them, how often they change, and what user populations are allowed to read them. A stale internal policy or duplicate product manual can create incorrect grounded answers even when search and generation are technically healthy. Include source identifiers, effective dates, tenant or department metadata, and security labels before documents reach the index. RAG quality begins with information governance.

Add a lifecycle state to each source: draft, approved, superseded, expired, or deleted. The retriever should avoid documents that are technically present but no longer authoritative. For high-value policies, connect indexing to the content-management workflow so approval or retirement changes the search corpus automatically. RAG becomes more trustworthy when knowledge governance and retrieval state are synchronized rather than maintained by separate manual processes.

Content preparation determines retrieval quality

Chunking should preserve meaningful sections, tables, lists, definitions, and procedures. Too-small chunks lose context; too-large chunks reduce precision and consume more model context. Azure AI Search pipelines can combine parsing, enrichment, chunking, and vectorization so content preparation is repeatable. Evaluate chunk strategy with real questions rather than deciding from document size alone.

Tables and semi-structured documents need extra care because a generic text splitter can separate headers from values or lose relationships between rows. Preserve enough structure for the search system to retrieve interpretable evidence. Test representative PDFs, Office documents, web pages, and internal knowledge formats before scaling ingestion. Parsing quality is often the hidden bottleneck behind supposedly weak embeddings.

Hybrid search is often stronger than vector-only search

Vector search is strong for semantic similarity while keyword search remains useful for exact identifiers, codes, names, and rare technical terms. Azure AI Search can combine lexical and vector retrieval and apply semantic ranking to improve the final source set. Use a benchmark that contains paraphrased questions and exact-identifier questions so each retrieval mode has a fair test. Do not assume an embedding similarity score alone represents business relevance.

Query-time weighting should follow the workload. Exact error codes and product identifiers may need stronger lexical influence, while conversational questions may benefit more from semantic similarity. Semantic ranking can improve ordering after candidate retrieval, but it cannot recover a document that was excluded by a bad filter or never indexed. Treat retrieval as a pipeline whose stages can fail independently.

Index schema design also affects hybrid quality. Store fields that need exact matching, filtering, faceting, full-text search, and vector similarity in forms appropriate to each operation. One giant unstructured text field makes it harder to combine semantic and deterministic constraints. Good RAG search begins with a search index designed for the query types the application will actually receive.

Agentic retrieval helps with complex conversational questions

Azure AI Search now supports agentic retrieval that can rewrite or decompose complex user questions into focused subqueries, execute them in parallel, semantically rank results, and return grounding data with references. This is useful when one user request spans multiple documents, concepts, or implicit context. Agentic retrieval adds model calls, latency, and orchestration, so measure whether the improved recall and grounding justify the extra complexity. Simple FAQ-style workloads can still be better served by classic RAG.

Subquery planning is especially useful for comparison, synthesis, and questions with implicit assumptions. A query such as ‘Which plan changed and how does that affect customers in Germany?’ may need separate retrieval for the old plan, new plan, geography, and effective date. Trace the subqueries during evaluation so the team can see whether the agentic retriever misunderstood the question before blaming the model response.

Keep query-planning telemetry during testing. If the system decomposes one user question into four subqueries, reviewers should be able to see those subqueries and the sources returned for each. That trace makes it possible to distinguish a bad decomposition from a bad ranker or bad model answer. Without it, added agentic intelligence can make the retrieval layer harder to debug than classic RAG.

Permission-aware retrieval is a security requirement

Foundry IQ and Azure AI Search support knowledge bases with source-specific access controls, but document permissions are not automatically enforced for every connector. Some sources require permission metadata to be ingested and the end-user identity to be supplied on the retrieval request; omitting that identity or configuring an unsupported source can expose results that the application should have filtered. A search result must be filtered according to the caller’s actual permissions before it becomes model context. Document-level access controls, metadata filters, identity integration, and network boundaries should work together. The model must not decide whether a user is allowed to see content merely because that content would answer the question well.

Use the caller’s identity to enforce document-level or metadata-based permissions at retrieval time. Post-generation redaction is weaker because sensitive content may already have entered the model context or logs. Test with two users who ask the same question and should receive different evidence sets. Security is proven when the unauthorized document never becomes eligible for retrieval.

Grounding needs citations and source traceability

The application should keep document IDs, section references, or URLs with retrieved chunks and expose citations when users need to verify important claims. Citations are not useful if they point to a whole document that never contained the claimed evidence. Evaluate citation precision as well as answer quality. When a source changes, the team should be able to identify which answers or tests depend on the updated knowledge.

Citations also help support teams debug content problems. When a user flags a wrong answer, the operator can determine whether the cited source itself was wrong, the retriever selected the wrong source, or the model misinterpreted correct evidence. Store source version and index timestamp so a later document update does not make an old trace impossible to reconstruct.

Evaluation should separate retrieval and generation failures

Build retrieval-only tests that check whether the expected evidence appears in the result set before evaluating the final generated response. Then test groundedness, completeness, answer usefulness, refusal behavior, and citation correctness after generation. If the right source was never retrieved, prompt changes to the model are unlikely to fix the problem. If the right source was retrieved and ignored, the generation prompt or model behavior is the more likely target.

Use gold queries with expected documents and expected answer points. Retrieval metrics can check whether the evidence was found; generation metrics can check whether the answer used that evidence correctly. This separation is especially important during embedding or ranking changes because a different retrieval set may shift answer quality even when the model and prompt are unchanged.

AI-901 and AI-300 mark useful role boundaries

The AI-901 exam is the foundational Microsoft AI path. The AI-300 exam goes deeper into MLOps and GenAIOps lifecycle operations. AI-103 sits at the application-and-agent engineering layer where retrieval, grounding, models, tools, and user experience are implemented. Use adjacent certifications to understand fundamentals and production operations without turning one RAG article into the entire Microsoft AI catalog.

AI-103 candidates should be comfortable implementing the retrieval path and integrating it with the application or agent. AI-300 becomes relevant when the main responsibility shifts toward production evaluation, deployment, monitoring, governance, and lifecycle of the AI system. The distinction is useful on teams where one engineer builds the RAG behavior and another owns the release and operational controls around it.

Design RAG as a maintained production system

Production RAG needs source ownership, re-indexing or deletion workflow, access control, retrieval evaluation, generation evaluation, tracing, monitoring, and a release process for changing embeddings or ranking. The strongest Azure design makes the evidence pipeline observable enough that a bad answer can be traced back to source, retrieval, or generation rather than treated as an unexplained model failure.

Keep retrieval configuration versioned: index schema, chunking strategy, embedding model, semantic configuration, filters, ranking profile, prompt, and generator model. A production regression can come from any of those components. Versioning lets the team compare changes and roll back selectively instead of treating the entire RAG system as one opaque application release. Include deletion and re-index SLAs in the operating model. When a policy expires or a user loses access, the old content should stop appearing within a defined period. Monitoring should detect failed indexing jobs and stale sources just as application monitoring detects failed API calls. Knowledge freshness is part of service reliability for any system that claims to answer from enterprise truth.

A final end-to-end evaluation should use two identities with different permissions, several question types, and one intentionally stale document. Verify retrieval, citations, authorization, refusal, and source freshness before promoting a change.

img