Databricks Data and AI Skills
Data and AI skills on Databricks make more sense when learned through one useful data product than through a list of tools or certification names. A team might collect customer-support events, produce an operational dashboard, and also build a retrieval-augmented assistant that answers questions about published support procedures. Those are different outputs, but both depend on the same foundational decisions: trustworthy sources, predictable transformations, controlled access and observable changes.
Consider a fictional support operation with three inputs: daily ticket events, a maintained procedures library and a small table of approved product codes. The goal is to produce an accurate backlog report for managers and a knowledge assistant for support agents. This project illustrates how data engineering, analytics, governance and generative AI contribute distinct skills to one outcome. It is an illustrative architecture, not a requirement for any individual Databricks certification.
For the ticket stream, define what one record represents. Is it the creation of a ticket, a status change or the current ticket state? A report that counts all change events as open tickets will be wrong even if the Spark transformation runs successfully. Decide on a stable event identifier, ticket identifier, event time and source-system sequence before aggregating anything.
For the procedures library, define the authority of the source: published documents only, current revision identifiers, document owner and permitted audience. A generative-AI assistant should not infer that a draft procedure is official simply because it happens to be present in a storage location. The source contract is therefore different from the event contract, even though both become managed datasets.
Write explicit acceptance checks. A ticket event without an identifier cannot join reliably. A procedure without a revision or owner cannot be defended as current. These are examples of conditions that need manual policy choices; automated acceptance rules should reflect those decisions instead of inventing them.
Land raw events and source-document metadata without silently rewriting their meaning. Preserve ingestion time separately from event time. When a delayed event arrives, the system needs to decide whether it changes the current ticket state, which historical report windows need recomputation, and how to keep a replay from creating duplicate business activity.
Use a small synthetic set with ticket T-101 opened, assigned, resolved and reopened in that order. Deliver one of those events late and replay another event twice. The expected report should be defined before running the pipeline; otherwise a plausible final count can conceal non-idempotent processing. The key skill is distinguishing the arrival of records from the business state they represent.
Databricks’ Delta MERGE documentation explains an important failure mode: if multiple source rows ambiguously match one target row, an update can fail. Deduplication and choice of business key therefore matter before implementing an upsert. A successful write is not enough unless the result preserves the intended event semantics.
Create a curated ticket-state table keyed by ticket ID and containing the latest approved state, priority, owner team and timestamps relevant to the report. Build a separate procedures table keyed by document and revision, with cleaned text and an explicit publish status. Do not force these into one giant dataset merely because they share a platform.
From the ticket-state table, produce an analytics view such as open tickets by team and age bucket. Validate it with hand-calculated expected counts for the synthetic example. From the procedures table, create the retrieval corpus for the assistant. Validate that only published versions enter the retrieval index, that obsolete versions can be removed or excluded, and that the original source remains traceable.
Those distinct consumption patterns illustrate why “one data and AI platform” does not mean every workload should use the same table shape, latency target or evaluation metric. The dashboard needs reliable aggregation. The assistant needs accurate document retrieval and appropriately grounded answers.
Suppose managers can inspect individual ticket details but support agents should see only their own team’s procedures. A schema permission alone does not prove the application behaves correctly. Determine which Databricks identity reads which governed tables, what filtered or masked data appears in each output, and whether the serving component can reach data its users must not see.
Run two synthetic user profiles against both consumers. The manager’s report should show authorized aggregate information; the agent’s assistant should retrieve only approved procedure content. Test a restricted title as a negative case. If the retriever provides restricted text to the model, the error occurs before answer generation and must be fixed at the access or retrieval boundary.
The Unity Catalog governance guide addresses the underlying catalog and table controls. For this end-to-end project, the additional skill is testing whether the complete reporting and application path preserves those controls through intermediate assets and services.
For analytics, specify the expected output row count, the business definition of “open,” the accepted lateness window and the tolerance for a delayed refresh. A report that runs quickly but double-counts reopened tickets is not a quality success. Monitor completeness, duplicate handling, late-event corrections and freshness alongside runtime.
For the assistant, create answerable questions with source passages and unanswerable questions whose evidence is absent. Check whether the expected passage is retrieved, whether the answer is faithful to it, whether the assistant admits uncertainty, and whether a denied user is prevented from receiving a protected procedure. Separate retrieval measures from answer measures: a confident model response cannot compensate for a missing authoritative source.
Current Databricks documentation describes MLflow-based agent evaluation with tracing, scorers and evaluation datasets. Those tools can help make an AI test repeatable, but human reviewers should still inspect cases involving policy interpretation, ambiguous instructions or sensitive authorization boundaries.
| Product | Primary correctness check | Operational failure to rehearse |
|---|---|---|
| Ticket dashboard | Expected ticket state and counts at a stated as-of time | Late status event changes the prior total |
| Procedure assistant | Authorized, current source passage supports the answer | Index retains an obsolete or restricted document |
| Shared data platform | Sources, lineage and permissions can be explained | Source schema changes without a downstream contract update |
Keep transformations, tests and deployment definitions in reviewed source control. A change to the ticket-state logic should trigger report-validation tests; a changed document extractor should trigger retrieval and answer evaluations. The two pipelines may share infrastructure, but their approval conditions are different.
Rehearse a nonproduction release with a deliberately malformed ticket event and an out-of-date procedure. A reliable pipeline should either reject or quarantine invalid business records under a documented policy; the assistant should exclude unpublished procedures and report when no authorized evidence is available. Databricks’ pipeline expectations documentation describes warn, drop and fail behaviors, with different consequences for downstream data and metrics. Choose among them based on business consequence, not as an automatic preference.
Define rollback evidence for each consumer. The report owner needs to know which data revision and definition produced a published metric. The assistant owner needs to know which retrieval index, document revisions and model or prompt version produced a response. An identical dashboard screenshot or fluent answer does not prove the underlying provenance remained the same.
Different practitioners will discover different gaps. Someone who struggles with joins, late events and job orchestration needs more data-engineering practice. Someone who can publish governed datasets but cannot reproduce a meaningful AI evaluation needs application-evaluation depth. Someone whose feature works only under their personal workspace permissions needs deployment and governance practice before taking on a larger service.
The Databricks certification catalog spans data analysis, data engineering, machine learning, generative AI, context engineering and Spark development. It can help a learner choose a direction, but the catalog is not a substitute for understanding the system they intend to build. The certifications-by-role article discusses credential fit; this worked example concerns the complementary skills and handoffs required to ship one product.
A useful completion criterion is simple: another engineer can reconstruct both outputs from documented sources, demonstrate that the same inputs produce the expected results, trace a failed run, and show that permissions are enforced. When those claims are backed by test data and observed behavior, the project has taught real data-and-AI skills beyond the names of the platform’s individual features.