What hallucinations are
We speak of hallucinations when a language model produces statements that sound plausible but are not true. A model gives an invoice number that does not appear in the document, refers to a rule that does not exist or answers a customer question with an invented delivery time.
This is not a rare exception but a property of the technology that you have to plan for. If you build AI into your workflows, you should not hope that errors will not happen, but design the system so that errors are caught before they cause damage.
Why language models make things up
A language model generates text by choosing the most likely continuation word by word. It has no built-in knowledge of whether a statement is true. If information is missing, it fills the gap with something that would typically appear at that point.
- The required detail is not in the document, but the model is asked to provide it anyway.
- The task is worded openly, for example “answer the question” instead of “answer only from these documents”.
- The document is hard to read, for example a blurred scan or handwriting.
- The question concerns knowledge the model cannot have, such as current stock levels.
The remedies follow almost automatically from these causes: the model has to be tied to sources, its output has to be checked, and it has to be allowed to say that it does not know something.
Safeguards built into the setup
Source grounding
The most effective protection is to let the model work exclusively with the material that belongs to the case: the document, the email, the data from your system. Answers from an internal assistant should cite where the information was found, so that every statement can be verified.
Mandatory fields and plausibility checks
Extracted data is not simply accepted but checked against rules. Does the sum of the line items match the invoice total? Does the supplier exist in your master data? Is the tax rate correct? Is there a purchase order number? Checks like these are classic program logic and therefore not prone to hallucinations.
Flagging uncertainty
A good system is allowed to say that it is not sure. If a detail is missing or two sources contradict each other, the case is flagged and passed to a person. A system that asks when in doubt is better than one that guesses.
Matching against existing data
Wherever possible, details are matched against data already held in your systems: purchase order, delivery note, customer records, price list. Anything that does not match is not silently corrected but made visible.
The human in the loop
“Human in the loop” means that at defined points a person decides before the workflow continues. This is not a sign of distrust in the technology but a division of labour: the system handles the routine, the person takes care of whatever needs judgement.
To keep approval from becoming a formality, the case should be presented so that the review is quick: with the original document, the system's suggestion and a brief explanation of why the case is being referred. If you approve everything blindly, you lose the protection that approval is meant to provide.
How to set thresholds
A threshold determines when a case goes for approval. It can depend on an amount, on the type of case or on how confident the system is. A proven approach is to start cautiously and only relax the thresholds once you have observed the results for a while.
- Amounts: invoices below a set sum go through, above it a person approves. You decide where the line is.
- New partners: the first case with a new supplier or customer always goes for approval.
- Discrepancies: every difference between invoice, purchase order and delivery note is referred.
- Content: complaints, cancellations and legal questions always go to people.
- Uncertainty: if the system cannot reliably read a mandatory detail, the case is flagged.
At the start it can make sense to generate all results as drafts only and pass nothing on automatically. Only once you see how reliably the system handles your cases do you gradually release more.
Audit log and spot checks
Every automatic step should be recorded with a timestamp and source. This lets you check afterwards what the system did and on what basis. That helps with queries, with troubleshooting and with audits by third parties.
In addition to approvals, regular spot checks on the cases that went through automatically are worthwhile. If errors turn up, the rules are tightened. This way the system improves over time instead of quietly getting worse, for example when a supplier switches to a new invoice layout.
Conclusion
Hallucinations cannot be ruled out completely, but they can be kept well under control. Source grounding, validation rules, flagged uncertainty, sensible thresholds, genuine approvals and a complete audit log turn a language model into a system your team can trust. With every offer, do not just ask what the AI can do, but also what happens when it gets something wrong.