Amazon AWS Certified Generative AI Developer - Professional AIP-C01 Practice Test 47

Question 460

A model update meets general benchmark but fails tenant-specific cases. What should be added to this test program before release?

Approve the model solely because it improved on a public benchmark unrelated to tenant workflows.
Remove failing tenant cases from the evaluation set because their number is small.
Assume the new model ID preserves all old prompt/tool behavior automatically.
Block promotion pending use-case-specific regression evidence.

Explanation

Correct Answer

Use tenant-specific held-out scenarios and expected responses in addition to general benchmarks; fail promotion when important local business cases regress.

Incorrect Answers

General scores cannot guarantee specialized business correctness.

Critical localized regressions should not be hidden.

Version changes require integration and case regression checks.

Question 461

A test suite checks only syntactically valid JSON. Which evaluation design most reliably measures the actual task risk?

Accept a parseable JSON object with an unsupported financial status code.
Include allowed values, references and business-rule consistency assertions.
Test only that required keys are present, ignoring whether they contradict each other.
Use a model-generated success flag as proof a referenced account exists.

Explanation

Correct Answer

Assert permitted enum values, cross-field business rules and referenced authoritative records after JSON syntax/schema validation.

Incorrect Answers

Parsing does not enforce allowed business semantics.

A structurally valid record can be logically impossible.

Model text is not an authoritative database lookup.

Question 462

A model refuses legitimate requests after stronger guardrails. Which evidence is needed for an accurate comparison?

Optimize only for catching harmful prompts and disregard every harmless request denied.
Remove all guardrails to reduce false refusals without checking harmful outputs.
Conclude a more restrictive prompt is always safer without validating user harm.
Evaluate false-refusal rate as part of safety quality.

Explanation

Correct Answer

Measure legitimate-question false refusal/overblock rate alongside prohibited-content capture using representative allowed inputs.

Incorrect Answers

A safety configuration can become unusable through excessive false positives.

This creates unbounded false negatives.

Policy usability depends on both allowed and disallowed case performance.

Question 463

A prompt injection test cannot tell if tool action actually ran. What should be added to this test program before release?

Declare the injection harmless because the final message omitted the unauthorized command.
Inspect downstream execution receipts rather than only generated text.
Count model intent to use a tool as a successful prohibited transaction.
Inspect only retrieved document text and skip backend authorization logs.

Explanation

Correct Answer

Test actual downstream tool execution receipts, caller identity and state change under injected instructions—not just whether model text mentions an action.

Incorrect Answers

The tool might still have executed before the final response.

A toolUse proposal and actual backend execution are different.

Outcome evidence is necessary for an action-based injection test.

Question 464

A new RAG index changes chunking strategy. Which evaluation design most reliably measures the actual task risk?

Benchmark retrieval and answer support with source-aligned gold cases.
Approve the new index solely because it stores the same number of vectors as before.
Use unrelated benchmark queries so before/after scores cannot be compared.
Measure only retrieval latency, not whether effective clauses are supported in the answer.

Explanation

Correct Answer

Benchmark changed chunk strategy with labeled source spans, retrieval hits and claim entailment in final answers before cutover.

Incorrect Answers

Vector count does not establish evidence coverage or context preservation.

A changed test set confounds the indexing effect.

Faster retrieval can still return unusable evidence.

Question 465

An application evaluates only English responses. Which evidence is needed for an accurate comparison?

Use a perfect English benchmark as proof all multilingual user requests are supported.
Use language/locality-specific reference sets for intended users.
Translate all reference answers automatically and never verify technical terms with qualified speakers.
Exclude low-resource languages to increase overall acceptance rate.

Explanation

Correct Answer

Evaluate intended locales with labeled language-specific test questions, document versions and error categories instead of an English-only score.

Incorrect Answers

Language/locale differences can change retrieval and generation quality.

Translation errors may corrupt the gold benchmark.

That hides performance for real user populations.

Question 466

A production agent receives no user feedback. What should be added to this test program before release?

Treat raw thumbs-up counts as sufficient evidence the model's facts are accurate.
Implement authenticated rating/correction channel with reviewer ownership.
Allow any anonymous visitor to overwrite production training labels directly.
Suppress negative feedback because it makes the dashboard look worse.

Explanation

Correct Answer

Collect authenticated customer feedback and correction evidence with accountable review ownership, avoiding anonymous or untraceable overwrites.

Incorrect Answers

Ratings may reward fluency rather than correctness.

Unauthenticated writes can poison evaluation and feedback sets.

Missing reports prevent systematic defect detection.

Question 467

A model output rating is not linked to source revision. Which evaluation design most reliably measures the actual task risk?

Store only a star rating without the answer or source versions used.
Attach prompt/model/document version metadata to feedback items.
Use the latest current policy document to explain an old response without knowing the old revision.
Group feedback only by user display name without a request correlation key.

Explanation

Correct Answer

Attach model/prompt, indexed document/source revision and request IDs to feedback so a negative rating is replayable and attributable.

Incorrect Answers

The team cannot reproduce or fix the specific failure.

The older answer may have been based on different source content.

Multiple model versions and sessions become indistinguishable.

Question 468

An agent must obey an explicit termination constraint. Which evidence is needed for an accurate comparison?

Let the agent continue indefinitely until it chooses to stop itself.
Mark task success when max steps is reached even if required work is unfinished.
Restart interrupted transactions under new IDs with no persisted state.
Test max-steps/time/cost stops and recoverable error status.

Explanation

Correct Answer

Test max-steps, time and cost stopping conditions and safe resumable failure status for long-running agents, including tool timeout scenarios.

Incorrect Answers

Unbounded loops can exceed cost and tool quotas.

A budget stop is not a completed business task.

Recovery can duplicate effects or lose progress.

Question 469

An automated evaluation can't establish harmful behavior in rare scenarios. What should be added to this test program before release?

Use a single automated FM judge's low risk score as final proof for every severe scenario.
Remove rare adverse events from evaluation because confidence intervals are wide.
Treat a policy-violating action as harmless because it produced well-formatted text.
Route high-risk cases for qualified human review and document judgment.

Explanation

Correct Answer

Escalate rare but high-consequence failures to qualified human adjudication and record uncertainty when automatic scores lack valid evidence.

Incorrect Answers

An automated judge may miss rare harm modes.

Small-sample uncertainty calls for caution, not suppressing failures.

Action consequence differs from output formatting.

Question 470

A monitoring report hides confidence intervals and sample size. Which evaluation design most reliably measures the actual task risk?

Publish a 99% success number without the number of evaluated cases.
Combine populations with different task difficulty into one denominator with no stratification.
Report valid denominators, uncertainty and tested population per metric.
Claim the reported proportion is exact true-world performance with zero uncertainty.

Explanation

Correct Answer

Report each metric's real sample size, denominator, tested cohort and uncertainty interval; scores without those details are easily misinterpreted.

Incorrect Answers

One error in a tiny sample is not comparable to a large controlled test.

An aggregate can hide important subgroup failures.

Empirical estimates have sampling error and distribution limits.

Pass Your Amazon AWS Certified Generative AI Developer - Professional AIP-C01 Exam Easy!

Question 460

Explanation

Correct Answer

Incorrect Answers

Question 461

Explanation

Correct Answer

Incorrect Answers

Question 462

Explanation

Correct Answer

Incorrect Answers

Question 463

Explanation

Correct Answer

Incorrect Answers

Question 464

Explanation

Correct Answer

Incorrect Answers

Question 465

Explanation

Correct Answer

Incorrect Answers

Question 466

Explanation

Correct Answer

Incorrect Answers

Question 467

Explanation

Correct Answer

Incorrect Answers

Question 468

Explanation

Correct Answer

Incorrect Answers

Question 469

Explanation

Correct Answer

Incorrect Answers

Question 470

Explanation

Correct Answer

Incorrect Answers

AWS Certified Generative AI Developer - Professional AIP-C01 Practice Tests