AI returns valid JSON but wrong data: how do you catch it?
Schema-valid JSON does not prove AI-extracted values are correct. Learn how to add business rules, source verification, provenance, and quality gates before automation acts.
AI can return JSON that parses cleanly, contains every expected field, and uses the correct data types while still being factually wrong. An incorrect totalAmount is still a number; the wrong customer's customerId is still a valid string. Schema validation is therefore only the first gate. Detecting bad data requires business rules, cross-field checks, source verification, and an explicit policy for cases where evidence is insufficient.
A practical design separates validation into layers: JSON/schema → semantic & business rules → source verification → risk-based decision. Data should trigger an action only after passing the controls appropriate to that action's risk. Creating a low-impact ticket may tolerate simpler checks; payments, customer updates, and access changes should require stronger evidence and fail-closed behavior.
Problem
Correct structure does not mean correct facts
Structured output makes responses easier to consume, but it does not independently verify what the values mean.
Suppose an AI system reads an invoice and returns invoiceNumber, vendor, currency, subtotal, tax, and total. JSON Schema can require total to be a number, restrict currency to an allowed enum, require specific fields, and reject unexpected properties. But if the document says VND 12,500,000 and the model extracts VND 12,800,000, the payload can still satisfy that schema.
This is a dangerous failure mode because nothing crashes. The workflow receives a valid object and keeps moving. If downstream automation creates a purchase order, updates a CRM, or calls a payment API, a semantic error becomes a business error. JSON Schema defines assertions over an instance, such as types, enums, and constraints; it is not a mechanism for proving that a model-derived value matches the source document.
