Amazon AWS Certified Generative AI Developer - Professional AIP-C01 Practice Test 46

Question 450

An FM judge approves all outputs from a similar model. Which evidence is needed for an accurate comparison?

Use one judge from the same model family as its own final approver with no calibration.
Treat a 10/10 judge score as proof of human approval when no reviewer participated.
Calibrate judge against human-labeled reference cases and monitor disagreement.
Remove disagreement cases from the calibration set so judge accuracy appears higher.

Explanation

Correct Answer

Calibrate FM-judge decisions on human-labeled accepted/rejected examples and monitor disagreement, especially with a related model family.

Incorrect Answers

Correlated failure modes can produce optimistic scores.

An automated score is not independent human evaluation.

Selective omission conceals the important error distribution.

Question 451

A human reviewer sees only successful model replies. What should be added to this test program before release?

Stratify human review across edge/failure cases and sample populations.
Give reviewers only the model's fluent final messages without failed tool states.
Sample only the highest-confidence outputs to minimize disagreement.
Exclude edge populations because their cases slow down manual QA.

Explanation

Correct Answer

Stratify human review across successes, timeouts, denied requests and protected populations; show context and outcome evidence for each.

Incorrect Answers

The review systematically excludes important failure modes.

Confidence-based cherry-picking inflates apparent quality.

Unrepresentative evaluation misses concentrated harms.

Question 452

A retrieval system has high recall but low precision at five. Which evaluation design most reliably measures the actual task risk?

Evaluate ranked precision, recall and answer-context usefulness separately.
Report only corpus-wide recall and assume top-five search results are high precision.
Use k-NN similarity scores as substitutes for labeled relevance judgments.
Increase k without measuring context budget, ranking quality or response latency.

Explanation

Correct Answer

Measure recall, precision at k and usefulness of the returned evidence to answer the task; retrieving many relevant passages can still have poor early ranking.

Incorrect Answers

A system can find a relevant document somewhere while top results are mostly noise.

Numerical proximity is not verified relevance.

More candidates may worsen precision and cost.

Question 453

A knowledge base returns the right document but wrong revision. Which evidence is needed for an accurate comparison?

Give full marks when the document ID matches even if the passage is an obsolete version.
Score only the text embedding similarity without comparing effective dates.
Include current-source/version correctness in retrieval evaluation.
Drop revision metadata from the benchmark to simplify evaluation.

Explanation

Correct Answer

Check that retrieved source revision is effective/approved for the query date, not merely that the correct document title appeared.

Incorrect Answers

A superseded policy can produce incorrect current answers.

Semantic similarity does not establish source currency.

The test cannot detect version-specific retrieval errors afterward.

Question 454

An agent completes tasks but makes unnecessary tool calls. What should be added to this test program before release?

Track task success, authorized tool efficiency and costs jointly.
Measure only final success text while ignoring duplicated external writes and tool costs.
Treat a higher tool-call count as inherently better reasoning.
Minimize tool calls to zero even when live facts or transactions are required.

Explanation

Correct Answer

Track verified task completion, number and authorization of tool calls, error recovery and cost together; unnecessary calls are a quality defect.

Incorrect Answers

A model can claim success despite wasteful or unsafe operations.

Repeated calls may indicate loops or poor planning.

Avoiding necessary tools destroys task correctness.

Question 455

A tool performs a write and model says it failed. Which evaluation design most reliably measures the actual task risk?

Mark the workflow failed because the model said 'sorry' after the backend committed successfully.
Mark every tool invocation successful as soon as it was requested.
Ignore the operation ID and rely on user-visible prose alone.
Check authoritative service outcome as part of evaluation rather than narration.

Explanation

Correct Answer

Inspect the downstream authoritative write receipt and stored state, not just model narration, when scoring transaction success or failure.

Incorrect Answers

Natural-language uncertainty can disagree with actual side effects.

A model-generated tool request is only an intention; a missing successful execution receipt must not be presented as a completed business transaction.

The evaluator cannot verify whether a real write occurred.

Question 456

A multi-step workflow hides partial retries. Which evidence is needed for an accurate comparison?

Retry the entire workflow with new IDs whenever a timeout occurs.
Evaluate state transitions and idempotent recovery from interrupted sessions.
Evaluate only the final happy path after resetting the state between attempts.
Suppress intermediate tool errors and count the final answer as a complete workflow.

Explanation

Correct Answer

Test interruption and retry state transitions with stable operation IDs, idempotency and reconciliation of partial commits.

Incorrect Answers

Already-committed steps can be duplicated.

This misses persistence and recovery failures.

Hidden partial failures can invalidate business state.

Question 457

An evaluation team compares two eligible foundation-model versions for the same business workflow. They differ in inference charges, answer quality and final response duration. Which TWO like-for-like measures show which model serves accepted tasks more efficiently?

Use radically different prompt lengths for the two models and compare price per API call without workload normalization.
Measure cost per quality-accepted business task under the same representative inputs and acceptance criteria.
Count raw output tokens alone without checking whether the task was completed correctly.
Choose solely by advertised model parameter count because larger models always have better task accuracy.
Measure end-to-end completion latency for the same request distribution and concurrency.

Explanation

Correct Answer

Quality-adjusted unit cost captures actual productive results, not just billing for generated text.

End-to-end completion time includes useful final output, not merely first-token arrival or isolated inference speed.

Incorrect Answers

Different input distributions and price-per-call comparisons confound model choice with workload size.

Raw tokens can be cheap but factually wrong, creating expensive downstream remediation.

Parameter count cannot substitute for a held-out evaluation against the actual domain task.

Question 458

An A/B test gets more clicks but less policy accuracy. Which evaluation design most reliably measures the actual task risk?

Select the higher-click model even when it cites incorrect policy clauses.
Require factual quality guardrails alongside business metrics.
Use the same biased internal test audience for both but claim all customer populations are covered.
Evaluate answer tone only and assume persuasion proves source correctness.

Explanation

Correct Answer

A/B success needs factual acceptance thresholds as hard gates alongside clicks/conversion, especially for policy and regulated answers.

Incorrect Answers

Business engagement cannot override material factual errors.

Generalizability requires representative users and cases.

Plausible prose may hide hallucinated policy statements.

Question 459

A canary rollout has no rollback threshold. Which evidence is needed for an accurate comparison?

Wait until a customer reports a serious harm before deciding rollback criteria.
Use only HTTP uptime as the acceptance/rollback measure.
Define explicit material error/latency/safety triggers before promotion.
Change the thresholds after reviewing failed canary outcomes until the release passes.

Explanation

Correct Answer

Define quantitative rollback thresholds for materially wrong answers, unsafe content, latency and backend failures before canary deployment.

Incorrect Answers

A canary needs preauthorized stop conditions.

Content can be wrong while the service remains available.

Post-hoc threshold manipulation undermines independent acceptance.

Question 460

A model update meets general benchmark but fails tenant-specific cases. What should be added to this test program before release?

Approve the model solely because it improved on a public benchmark unrelated to tenant workflows.
Remove failing tenant cases from the evaluation set because their number is small.
Assume the new model ID preserves all old prompt/tool behavior automatically.
Block promotion pending use-case-specific regression evidence.

Explanation

Correct Answer

Use tenant-specific held-out scenarios and expected responses in addition to general benchmarks; fail promotion when important local business cases regress.

Incorrect Answers

General scores cannot guarantee specialized business correctness.

Critical localized regressions should not be hidden.

Version changes require integration and case regression checks.

Pass Your Amazon AWS Certified Generative AI Developer - Professional AIP-C01 Exam Easy!

Question 450

Explanation

Correct Answer

Incorrect Answers

Question 451

Explanation

Correct Answer

Incorrect Answers

Question 452

Explanation

Correct Answer

Incorrect Answers

Question 453

Explanation

Correct Answer

Incorrect Answers

Question 454

Explanation

Correct Answer

Incorrect Answers

Question 455

Explanation

Correct Answer

Incorrect Answers

Question 456

Explanation

Correct Answer

Incorrect Answers

Question 457

Explanation

Correct Answer

Incorrect Answers

Question 458

Explanation

Correct Answer

Incorrect Answers

Question 459

Explanation

Correct Answer

Incorrect Answers

Question 460

Explanation

Correct Answer

Incorrect Answers

AWS Certified Generative AI Developer - Professional AIP-C01 Practice Tests