AI returns valid JSON but wrong data: how do you catch it?

AI can return JSON that parses cleanly, contains every expected field, and uses the correct data types while still being factually wrong. An incorrect totalAmount is still a number; the wrong customer's customerId is still a valid string. Schema validation is therefore only the first gate. Detecting bad data requires business rules, cross-field checks, source verification, and an explicit policy for cases where evidence is insufficient.

A practical design separates validation into layers: JSON/schema → semantic & business rules → source verification → risk-based decision. Data should trigger an action only after passing the controls appropriate to that action's risk. Creating a low-impact ticket may tolerate simpler checks; payments, customer updates, and access changes should require stronger evidence and fail-closed behavior.

Suppose an AI system reads an invoice and returns invoiceNumber, vendor, currency, subtotal, tax, and total. JSON Schema can require total to be a number, restrict currency to an allowed enum, require specific fields, and reject unexpected properties. But if the document says VND 12,500,000 and the model extracts VND 12,800,000, the payload can still satisfy that schema.

This is a dangerous failure mode because nothing crashes. The workflow receives a valid object and keeps moving. If downstream automation creates a purchase order, updates a CRM, or calls a payment API, a semantic error becomes a business error. JSON Schema defines assertions over an instance, such as types, enums, and constraints; it is not a mechanism for proving that a model-derived value matches the source document.

Layer 1 — Syntax and schema. Parse JSON, constrain types, required fields, enums, basic ranges, and additionalProperties where appropriate. A payload that fails here stops immediately.

Layer 2 — Semantic and business rules. Check domain conditions that a generic schema cannot know: endDate >= startDate, invoice totals reconcile under the organization's rounding rules, customer IDs exist, status transitions are allowed, and sender domains match registered partners.

Layer 3 — Source verification. Critical fields need provenance. Store a sourceDocumentId, page or extraction region, message ID, or evidence span that a validator can trace back to the source. If supporting evidence cannot be found, a high model confidence score is not proof.

Layer 4 — Decision gate. Combine the signals into accept, retry, quarantine, or reject. The gate should reflect the downstream action's risk rather than a single universal confidence threshold.

Prefer deterministic validators wherever possible. If AI extracts quantity, unitPrice, taxRate, and total, recompute the total in code using the application's accounting rules. If it returns a customerCode, look it up in master data. If the output includes an effective date, validate it against policy using the application's canonical timezone.

Cross-field validation is especially useful because many errors emerge only when fields are considered together. paymentStatus = paid with no transaction ID, action = close_ticket when the ticket is not in a closable state, or a currency/value combination that violates domain rules are signals that a shape-only check can miss. Version these rules alongside business logic, give failures explicit error codes, and cover them with tests.

The practical rule is simple: the more directly a fact can be checked against a system of record, the less you should rely on the model to validate it. LLMs are useful for interpreting ambiguous input; databases and deterministic code are better for confirming facts your organization already knows.

A robust design preserves enough provenance to answer: which source, location, and version supports this value? For email, that may be the original message ID and actual sender headers; for PDFs, a document ID plus page or OCR region; for RAG, an approved source ID or chunk.

Validators can then check high-impact fields against evidence. A contract number should be supported by the approved document; a vendor name should map to a real vendor ID; an email identity should be checked against the original header rather than a display name. When the source does not contain a value, the contract should support an explicit unknown or not_found outcome instead of forcing the model to guess a syntactically valid answer.

Microsoft Foundry separates quality signals such as groundedness, relevance, and completeness because response quality cannot be inferred from output shape alone. The same idea applies to extraction pipelines at field level: important values need evidence or ground truth, not just well-formed JSON.

Avoid reducing validation to one boolean. Production pipelines benefit from classifying the reason for failure: schema violation, missing evidence, business-rule conflict, unavailable source, or ambiguous extraction. That classification determines whether a retry can actually help. Re-running the same input will not fix a master-data conflict; it only costs more and may produce a different answer that is harder to audit.

Auto-accept can be appropriate when required fields pass schema, deterministic rules, and evidence requirements. For recoverable OCR or parsing failures, retry with a limit and an idempotency key. If sources conflict or a critical field has no evidence, quarantine or reject the record and do not call the downstream action. Human review can be one risk-policy branch where needed, but it does not need to be a mandatory stage in every workflow.

NIST AI RMF notes that validity and reliability in deployed AI systems are commonly assessed through ongoing testing and monitoring, with controls reflecting the potential impact of failures. That is why the same model may reasonably have lighter gates for email classification and stronger gates for permission changes or financial transactions.

Build a ground-truth dataset from inputs representative of production: clean documents, poor scans, multilingual text, multiple templates, missing fields, similar-looking values, and edge cases that previously caused failures. For each sample, record expected values for the fields that matter.

Track at least schema pass rate, exact or normalized match for critical fields, false-accept rate, false-reject rate, missing-evidence rate, and retry/quarantine rate. For amounts and dates, use domain-aware comparators rather than generic text similarity. Segment metrics by source or template so a healthy overall average does not hide one failing data group.

Run this evaluation in CI whenever prompts, models, OCR, schemas, or business rules change, and continue production sampling and monitoring after deployment. Microsoft Foundry similarly treats evaluation, monitoring, and tracing as complementary capabilities for detecting inaccurate, ungrounded, or degrading model behavior over time.

Can JSON Schema detect an incorrectly extracted amount?

Not if the wrong amount still satisfies the declared type and constraints. Recompute it from line items when possible or compare it with evidence in the source document.

Should we trust a confidence field returned by the model?

Not as the only proof. Confidence can be a routing signal after calibration with evaluations, but critical fields still need source evidence and deterministic checks.

Is a second AI call enough to verify the first one?

No. A second call may expose disagreement, but it can repeat the same mistake. If the fact exists in a database, API, or source document, verify it directly there.

When should the workflow retry?

Retry when the failure is plausibly transient or a different strategy can add information, such as a different OCR path. Do not retry indefinitely on business-rule conflicts or missing source data.

Does every output need human review?

No. Low-risk output can auto-pass when it clears the required gates. Human review can be a control for critical cases where policy requires it; the essential rule is that automation must not act when acceptance conditions are unmet.