Your system must extract event details from calendar invitations and output JSON that strictly conforms to a schema with fields for title, date, time, location, and attendees. Downstream reject any malformed or non-conformant JSON. What approach provides the most reliable schema compliance?
Answer(s): A
A: Define a tool with your target schema as input parameters and have Claude call it with the extracted data. Correct.Tool use enforces strict schema compliance at generation time , ensuring valid, structured JSON that downstream systems can reliably consume.B: Pre-fill Claude's response with an opening brace to force JSON output, then complete and parse the response. Incorrect.This is a fragile workaround and does not guarantee valid or schema-compliant JSON.C: Append instructions like "Output only valid JSON matching the schema exactly" and implement retry logic to re-prompt when JSON parsing fails. Incorrect.Helpful but not reliable—models can still produce malformed or non-conformant JSON.D: Include detailed JSON formatting instructions and the target schema in your prompt, then parse Claude's text response as JSON. Incorrect.Prompt-based formatting alone cannot guarantee strict compliance, especially in edge cases.
Your schema includes a skills: string[] field. Production monitoring reveals three consistency issues: (1) compound phrases like "Python and SQL" are sometimes kept as one entry, sometimes split; (2) implied but unstated skills occasionally appear in extractions; (3) similar documents produce wildly different array lengths (5-10 vs 40+ entries). Your prompt currently says "Extract skills mentioned." What's the most effective improvement?
Answer(s): D
A: Add constraints: "Extract 10–20 skills maximum, one skill per entry, only explicitly named skills." Incorrect.This enforces limits but is arbitrary and may exclude valid skills or still leave ambiguity in how to split phrases.B: Add post-extraction normalization that maps skills to a canonical taxonomy and deduplicates similar entries. Incorrect.Helpful downstream, but it does not fix inconsistent extraction behavior at the source .C: Enrich the schema to capture extraction metadata. Incorrect.Adds complexity but does not directly address inconsistency in skill identification and formatting.D: Add few-shot examples demonstrating compound phrase handling, explicit mention criteria, and appropriate entry granularity. Correct.Examples directly guide the model on how to split, what to include, and the expected level of detail , addressing all three issues effectively.
Your pipeline uses a tool called extract_metadata with a JSON schema for paper details. You've also defined lookup_citations and verify_doi tools for enrichment. During testing, you notice that when users include requests like "extract the metadata and tell me how cited it is," Claude sometimes calls lookup_citations first, which fails because it needs the DOI that extract_metadata would provide. What's the most effective way to ensure structured metadata extraction happens first?
A: Set tool choice to ("type": "tool", "name": "extract_metadata") and process the enrichment requests in subsequent turns after receiving the extracted metadata. Correct.This enforces the correct execution order , ensuring required data (like DOI) is available before dependent tools are called.B: Set tool choice to "any" so Claude must use a tool, combined with system prompt instructions prioritizing extract_metadata. Incorrect.This does not guarantee ordering—Claude may still choose the wrong tool first.C: Set tool choice to ("type": "tool", "name": "extract_metadata") for every API call in the pipeline, ensuring Claude always extracts metadata before any enrichment can occur. Incorrect.This is too rigid and prevents legitimate use of other tools in later steps.D: Set tool choice to "auto" and reorder the tool definitions so extract_metadata appears first in the tools array, since Claude prioritizes earlier-listed tools. Incorrect.Tool ordering does not reliably control selection or execution order.
Your system has been operating with 100% human review for 3 months. Analysis shows that extractions with model confidence >90% have 97% accuracy overall. To reduce reviewer workload, you plan to automate high-confidence extractions. Before deploying, what validation step is most critical?
Answer(s): B
A: Verify that 97% accuracy meets requirements for all downstream systems that consume the extracted data. Incorrect.Important, but it doesn’t ensure the confidence signal is reliable across different cases —it only checks overall acceptability.B: Analyze accuracy by document type and field to verify high-confidence extractions perform consistently across all segments, not just in aggregate. Correct.Aggregate accuracy can hide weak spots. You need to ensure confidence >90% is trustworthy across all segments , otherwise automation may introduce systematic errors.C: Compare accuracy at different confidence thresholds (85%, 90%, 95%) to find the optimal cutoff that maximizes automation while minimizing errors. Incorrect.Useful for tuning, but only after confirming the confidence signal is consistent and reliable across segments .D: Run a two-week pilot routing 25% of high-confidence extractions directly to downstream systems and monitor error reports. Incorrect.A pilot is valuable, but deploying without validating segment-level reliability first introduces avoidable risk.
Your extraction system uses tool_use with a JSON schema containing 12 fields and detailed descriptions, totaling approximately 2,500 tokens for the complete tool definition. Processing documents under 150K tokens yields 98% accuracy. For documents between 175-190K tokens, accuracy drops to 71%, with information from the final third consistently missed. The model's context window is 200K tokens. What is the most likely cause?
A: Tool definitions consume input context tokens. Combined with system prompts and document content, the total approaches the context limit, degrading end-of-document processing. Correct.The tool schema (~2,500 tokens) plus system prompts and large documents push total input close to the 200K context limit , causing truncation or reduced attention to the final portion—hence missed information in the last third.B: Very long documents exceed the model's effective attention span regardless of context limits, causing accuracy degradation for content farther from the prompt instructions. Incorrect.While attention can vary, the sharp drop near the context boundary strongly indicates a context limit issue , not general attention decay.C: The model distributes attention proportionally across input length, causing fields mentioned only once near the document's end to receive insufficient processing focus. Incorrect.This is a weaker effect and does not explain the consistent failure in the final third tied to document size thresholds .D: Schemas exceeding 8–10 fields increase decision complexity during parameter generation, reducing extraction accuracy independent of document length. Incorrect.Schema size is constant across cases; it does not explain why accuracy drops only for longer documents.
Post your Comments and Discuss Anthropic CCAR-F exam prep with other Community members:
Our website is free, but we have to fight against AI bots and content theft. We're sorry for the inconvenience caused by these security measures. You can access the rest of the CCAR-F content, but please register or login to continue.