Your invoice extraction uses tool use with strict JSON schemas. JSON syntax errors never occur, but 12% of extractions fail semantic validation--for example, line Item amounts don't extracted total, or vendor IDs don't match valid formats. These failures currently route to manual review. What's the most effective approach to reduce manual review volume while m accuracy?
Answer(s): C
A: Retry the extraction up to 3 times when validation fails, accepting the first result that passes validation. Incorrect.Retries without targeted feedback often repeat the same mistakes and don’t reliably fix semantic inconsistencies.B: Implement post-processing logic that automatically corrects common errors, such as recalculating totals from line items when sums don't match. Incorrect.While useful for specific cases, this is narrow and brittle, and doesn’t address broader validation failures like incorrect IDs.C: When validation fails, make a follow-up request with the document, extraction, and validation errors for model correction. Correct.This provides targeted feedback , enabling the model to fix specific issues, significantly reducing manual review while maintaining accuracy.D: Add stricter schema constraints with detailed field descriptions to prevent the model from generating invalid values initially. Incorrect.Schema improvements help upfront, but they cannot fully prevent semantic errors like mismatched totals.
Your team is extracting structured data from 50,000 legacy legal contracts under a two-week deadline. Initial testing with 500 sample documents shows 82% pass JSON schema first attempt, while the remaining 18% fall due to diverse issues—missing required fields, malformed dates, and incorrectly identified parties. Documents that fail typically need refinements targeting their specific failure modes before extraction succeeds. Which batch processing strategy is the most cost-efficient while still meeting the deadline?
Answer(s): B
A: Split documents into 10 sequential batches of 5,000 each, analysing results and refining prompts between batches to improve extraction quality progressively. Incorrect.This introduces unnecessary sequential delays and reduces throughput, risking the deadline.B: Submit all 50,000 documents via batch API, then submit failed extractions in successive batches— refining prompts between each batch—until all documents pass validation. Correct.This maximizes throughput and parallelism upfront , ensuring the deadline is met. Then it uses targeted iterative refinement only on failures , making it cost-efficient while handling diverse failure modes effectively.C: Use the real-time API for all 50,000 documents since the batch API's 24-hour processing window creates unacceptable deadline risk. Incorrect.This is unnecessarily expensive and not required given batch processing capabilities.D: Process 2,000 sample documents via real-time API to identify failure patterns and refine prompts, then batch process all 50,000 with the optimized prompts. Incorrect.While proactive, this assumes failure patterns generalize well, which the scenario suggests they don’t—since failures are diverse and require case-specific refinements .
Your extraction pipeline processes contracts that frequently include amendments. When a contract contains both original terms and later amendments (e.g., original clause specifies "30-day payment terms" while Amendment 1 changes this to "45 days"), the model inconsistently extracts one value or the other with no indication of which applies. What's the most effective approach to improve extraction accuracy for documents with amendments?
A: Preprocess documents with a classifier that identifies and removes superseded sections before the main extraction step. Incorrect.This is brittle and risky—accurately identifying and removing superseded clauses is complex and may lead to loss of important context.B: Implement post-extraction validation using pattern matching to detect amendments and flag those extractions for manual review. Incorrect.This increases manual review but does not improve extraction accuracy or resolve ambiguity.C: Redesign the schema so amended fields capture multiple values, each with source location and effective date. Correct.This preserves both original and amended values with context , enabling accurate interpretation and avoiding ambiguity about which value applies.D: Add prompt instructions to always extract the most recent amendment value and ignore superseded original terms. Incorrect.This relies on model judgment, which is inconsistent, and loses traceability of how values changed over time.
Your system must extract event details from calendar invitations and output JSON that strictly conforms to a schema with fields for title, date, time, location, and attendees. Downstream reject any malformed or non-conformant JSON. What approach provides the most reliable schema compliance?
Answer(s): A
A: Define a tool with your target schema as input parameters and have Claude call it with the extracted data. Correct.Tool use enforces strict schema compliance at generation time , ensuring valid, structured JSON that downstream systems can reliably consume.B: Pre-fill Claude's response with an opening brace to force JSON output, then complete and parse the response. Incorrect.This is a fragile workaround and does not guarantee valid or schema-compliant JSON.C: Append instructions like "Output only valid JSON matching the schema exactly" and implement retry logic to re-prompt when JSON parsing fails. Incorrect.Helpful but not reliable—models can still produce malformed or non-conformant JSON.D: Include detailed JSON formatting instructions and the target schema in your prompt, then parse Claude's text response as JSON. Incorrect.Prompt-based formatting alone cannot guarantee strict compliance, especially in edge cases.
Your schema includes a skills: string[] field. Production monitoring reveals three consistency issues: (1) compound phrases like "Python and SQL" are sometimes kept as one entry, sometimes split; (2) implied but unstated skills occasionally appear in extractions; (3) similar documents produce wildly different array lengths (5-10 vs 40+ entries). Your prompt currently says "Extract skills mentioned." What's the most effective improvement?
Answer(s): D
A: Add constraints: "Extract 10–20 skills maximum, one skill per entry, only explicitly named skills." Incorrect.This enforces limits but is arbitrary and may exclude valid skills or still leave ambiguity in how to split phrases.B: Add post-extraction normalization that maps skills to a canonical taxonomy and deduplicates similar entries. Incorrect.Helpful downstream, but it does not fix inconsistent extraction behavior at the source .C: Enrich the schema to capture extraction metadata. Incorrect.Adds complexity but does not directly address inconsistency in skill identification and formatting.D: Add few-shot examples demonstrating compound phrase handling, explicit mention criteria, and appropriate entry granularity. Correct.Examples directly guide the model on how to split, what to include, and the expected level of detail , addressing all three issues effectively.
Your pipeline uses a tool called extract_metadata with a JSON schema for paper details. You've also defined lookup_citations and verify_doi tools for enrichment. During testing, you notice that when users include requests like "extract the metadata and tell me how cited it is," Claude sometimes calls lookup_citations first, which fails because it needs the DOI that extract_metadata would provide. What's the most effective way to ensure structured metadata extraction happens first?
A: Set tool choice to ("type": "tool", "name": "extract_metadata") and process the enrichment requests in subsequent turns after receiving the extracted metadata. Correct.This enforces the correct execution order , ensuring required data (like DOI) is available before dependent tools are called.B: Set tool choice to "any" so Claude must use a tool, combined with system prompt instructions prioritizing extract_metadata. Incorrect.This does not guarantee ordering—Claude may still choose the wrong tool first.C: Set tool choice to ("type": "tool", "name": "extract_metadata") for every API call in the pipeline, ensuring Claude always extracts metadata before any enrichment can occur. Incorrect.This is too rigid and prevents legitimate use of other tools in later steps.D: Set tool choice to "auto" and reorder the tool definitions so extract_metadata appears first in the tools array, since Claude prioritizes earlier-listed tools. Incorrect.Tool ordering does not reliably control selection or execution order.
Your system has been operating with 100% human review for 3 months. Analysis shows that extractions with model confidence >90% have 97% accuracy overall. To reduce reviewer workload, you plan to automate high-confidence extractions. Before deploying, what validation step is most critical?
A: Verify that 97% accuracy meets requirements for all downstream systems that consume the extracted data. Incorrect.Important, but it doesn’t ensure the confidence signal is reliable across different cases —it only checks overall acceptability.B: Analyze accuracy by document type and field to verify high-confidence extractions perform consistently across all segments, not just in aggregate. Correct.Aggregate accuracy can hide weak spots. You need to ensure confidence >90% is trustworthy across all segments , otherwise automation may introduce systematic errors.C: Compare accuracy at different confidence thresholds (85%, 90%, 95%) to find the optimal cutoff that maximizes automation while minimizing errors. Incorrect.Useful for tuning, but only after confirming the confidence signal is consistent and reliable across segments .D: Run a two-week pilot routing 25% of high-confidence extractions directly to downstream systems and monitor error reports. Incorrect.A pilot is valuable, but deploying without validating segment-level reliability first introduces avoidable risk.
Your extraction system uses tool_use with a JSON schema containing 12 fields and detailed descriptions, totaling approximately 2,500 tokens for the complete tool definition. Processing documents under 150K tokens yields 98% accuracy. For documents between 175-190K tokens, accuracy drops to 71%, with information from the final third consistently missed. The model's context window is 200K tokens. What is the most likely cause?
A: Tool definitions consume input context tokens. Combined with system prompts and document content, the total approaches the context limit, degrading end-of-document processing. Correct.The tool schema (~2,500 tokens) plus system prompts and large documents push total input close to the 200K context limit , causing truncation or reduced attention to the final portion—hence missed information in the last third.B: Very long documents exceed the model's effective attention span regardless of context limits, causing accuracy degradation for content farther from the prompt instructions. Incorrect.While attention can vary, the sharp drop near the context boundary strongly indicates a context limit issue , not general attention decay.C: The model distributes attention proportionally across input length, causing fields mentioned only once near the document's end to receive insufficient processing focus. Incorrect.This is a weaker effect and does not explain the consistent failure in the final third tied to document size thresholds .D: Schemas exceeding 8–10 fields increase decision complexity during parameter generation, reducing extraction accuracy independent of document length. Incorrect.Schema size is constant across cases; it does not explain why accuracy drops only for longer documents.
Post your Comments and Discuss Anthropic CCA-F exam prep with other Community members:
Our website is free, but we have to fight against AI bots and content theft. We're sorry for the inconvenience caused by these security measures. You can access the rest of the CCA-F content, but please register or login to continue.