Document Extraction: LLM vs OCR

OCR is not obsolete. LLMs are not magic. For most business files, the best setup is a hybrid with confidence routing: machines handle the easy fields, people check the shaky ones.

The only way to know your mix is to test on your own documents. This post gives you the comparison and a test plan you can run in a week.

What intelligent document processing with LLMs covers

Intelligent document processing (IDP) turns files into structured data. Invoices, purchase orders, bank statements, contracts, claims, IDs, lab reports. The output is fields in a system: vendor, date, total, line items, policy number.

Three jobs sit inside that:

Read. Turn pixels into text. This is OCR.
Understand. Work out which text is the invoice number and which is the PO number. This is extraction.
Check. Decide if the answer is trustworthy enough to post without a person.

Older systems did job 1 well and job 2 with templates and rules. LLMs do job 2 well without templates. Neither does job 3 automatically. Most pipelines fail on job 3, not job 1 or 2.

OCR strengths and limits

Classic OCR engines and cloud document services read what is on the page. The output is literal.

Strengths

Fast and cheap per page.
Mostly deterministic. The same page gives the same text.
Returns word-level confidence scores and positions.
Does not invent text that is not there.

Limits

It reads text but does not understand it. You still need templates, rules, or a trained model to find the “invoice total.”
New vendor layout, new template.
Struggles with messy scans, skew, stamps, and handwriting.

TableFlow’s comparison makes the same split, though it is a vendor and sells a hybrid product. Treat its framing as a view, not a measurement: OCR is literal, fast, and cheap but needs templates, while LLMs handle varied layouts but can hallucinate and cost more.

Where documents are fixed and clean (a tax form, a standard ID), template OCR is hard to beat on cost and predictability.

LLM strengths and limits

A vision-capable LLM reads the page image, or the OCR text, and returns the fields you ask for as structured JSON. You describe a schema. You do not draw boxes.

Strengths

Handles new layouts without a new template.
Reads context. It can tell a ship-to address from a bill-to address.
Normalizes output: dates to one format, currency to numbers.
Works across languages and mixed formats in one pipeline.

Limits

It can produce a confident wrong answer. That is the main risk.
Cost and latency per page are higher.
Output can vary between runs unless you pin settings and test.
Confidence is not built in. A model saying “I’m sure” is not a calibrated score.

The hallucination point is not theoretical. A June 2025 paper on OCR hallucinations in multimodal LLMs built a benchmark (KIE-HVQA) on identity cards and invoices with simulated real-world degradation. Its argument: under blur or ambiguity, models lean on language priors instead of what is in the image, and produce plausible text. For a field like a total or an account number, plausible and wrong is the worst outcome, because it looks fine.

Benchmarks also disagree by document type. The OmniDocBench authors, who built a benchmark across nine document sources, evaluated pipeline-based methods and end-to-end vision-language models and found different strengths and weaknesses depending on document type. That is the honest summary. No single winner. I’d ignore any vendor chart that does not say which documents it used.

So, is OCR obsolete?

No. OCR as a reading step is still useful because it is cheap, fast, and gives you positions and confidence you can use. What is obsolete is the template-per-vendor approach to the “understand” step.

Hybrid pipelines

A hybrid uses each tool for what it is good at. Common patterns:

Pattern	How it works	Use when
OCR then LLM	OCR produces text and positions. The LLM maps text to a schema.	Clean to medium scans, many layouts
LLM reads the image, OCR cross-checks	LLM extracts from the image. OCR text confirms each value appears on the page.	High-stakes numbers
Router	Cheap rules for known formats. LLM for unknown ones.	Mixed stream with a few big vendors
Two-pass	Two different methods extract. Disagreements go to a person.	Money or compliance fields

The OCR cross-check is the one I like most. If the LLM returns a total of 4,180.00, search the OCR text for that string. If it is not on the page, the value is suspect. Cheap, deterministic, catches many invented values.

A 2025 paper on a hybrid OCR-LLM framework for copy-heavy documents shows what routing can buy in the right setting. On identity documents across four formats, it reported F1 of 1.0 at 0.97 seconds for structured files and 0.997 at 0.6 seconds for image inputs with PaddleOCR, and a 54x speedup over naive multimodal methods. Read the limits: one document family, highly repetitive, and the speed gain is against a naive baseline. Do not expect those numbers on your invoices. The useful point is the design idea: pick the method by document format instead of using one method for everything.

Confidence routing to a reviewer

This is the part most teams skip, and it decides whether the project saves money.

The idea: every extracted field gets a confidence signal. High confidence posts automatically. Low confidence goes to a person, who sees the field and the spot on the page, and fixes it.

Where the confidence comes from:

OCR confidence for the word or field. Microsoft’s guidance describes field and word confidence scores and says to use them to accept automatically or flag for human review where accuracy is critical.
Cross-checks. Value found in the OCR text. Total equals the sum of line items. Date parses. Vendor exists in the vendor master.
Agreement. Two methods or two runs return the same answer.
Model self-report, only as a weak extra signal, never alone.

How to set thresholds. AWS’s Textract guidance says the right threshold depends on the use case: as low as 50 percent for archival work, and 90 percent or higher for business processes with financial decisions. AWS’s human review workflow example goes further and sets different thresholds by field: 99 percent for one key, 90 percent for the rest. Do the same. A wrong notes field costs nothing. A wrong bank account costs a lot.

A routing rule I’d start with:

Auto-accept if confidence is high AND every cross-check passes.
Review if confidence is middling OR one cross-check fails. Show only the flagged field, with the page region highlighted.
Full review if several fields fail or the file looks unreadable.
Always review the fields that move money, until the data says you can stop.

Then measure the auto-accept error rate with a weekly random sample. If the auto-accepted fields are wrong more than your tolerance, tighten the threshold.

Why field-level routing matters: say a document has 15 fields and each is right 97 percent of the time. The chance that all 15 are right is 0.97 to the 15th power, about 63 percent. Roughly 37 percent of documents have at least one error. (That is my arithmetic, assuming independent errors.) A “97 percent accurate” system is not a 97 percent hands-off system.

How to run your own bake-off

Do this before you pick a vendor or write a line of pipeline code.

Step 1: Build the test set. Pull 150 to 300 real documents. Include the ugly ones: skewed scans, faxes, phone photos, stamps, handwriting, odd vendors. If your real mix is 30 percent poor scans, your test set should be too. A test set of clean PDFs will flatter every option.

Step 2: Label the truth. For each document, record the correct value for each field you care about. Have two people label a sample and compare. Disagreement between your own labelers tells you the ceiling.

Step 3: Pick the contenders. At minimum: your current method, OCR plus rules or a trained model, an LLM on the image, and a hybrid with cross-checks.

Step 4: Score by field, not by page. Character accuracy hides the problem. A page can be 99 percent right on characters and wrong on the total. Score each field as right or wrong, then report per-field accuracy, document-level “all fields right” rate, and a count of confident errors (wrong and high confidence). Confident errors are the dangerous ones.

Step 5: Test the routing. For each contender, sweep the threshold. At each setting record the share auto-accepted, the error rate among those, and the share sent to review. Plot it. You are looking for the point where auto-accepted errors drop to your tolerance.

Step 6: Cost it. Machine cost plus reviewer cost (next section).

Step 7: Rerun on a fresh batch. Hold out 50 documents you never tuned on. The score on those is the real one. See the AI agent evaluation guide for how to keep a test set honest over time.

Cost per document

Reviewer time is usually the biggest line. Machine cost per page is small next to a person’s minutes.

Here is a worked example. Every number is made up to show the math. Use your own rates.

Assume 15 fields per document, a reviewer who fixes a flagged field in 20 seconds, a loaded cost of $30 per hour (about $0.17 per field), and a perfect flag for every wrong field.

Setup	Machine cost per doc	Field error rate (made up)	Fields reviewed per doc	Review cost per doc	Total
OCR with templates	$0.005	8%	1.2	$0.20	$0.205
LLM only	$0.06	3%	0.45	$0.075	$0.135
Hybrid with cross-checks	$0.03	3%	0.45	$0.075	$0.105

The pattern matters more than the figures. The cheapest machine is not the cheapest pipeline, because errors turn into review time. And a pipeline that does not catch its own errors costs far more than any of these, because bad data reaches your ERP and someone cleans it up later. For the business case around that, see AI agents for back-office ROI.

Leave out one cost at your peril: setup and maintenance. Template OCR needs new work for each new layout. LLM pipelines need prompt and schema upkeep, plus a regression test when the model changes. Put both in your estimate.

If your documents are invoices, the follow-on question is what happens to the exceptions, such as a missing PO or a price mismatch. I cover that in AI agents for accounts payable exceptions. For contracts and other long-form files, extraction has different failure modes, covered in AI agents for legal document review.

Bottom Line

Do not pick OCR or LLM on faith. Pick the pipeline that has the lowest confident-error rate on your own ugly documents.

Start with OCR plus an LLM, add cross-checks, and route low-confidence fields to a person.

Next step: pull 200 real documents this week and run the bake-off.

FAQ

Is OCR obsolete? No. OCR is still a cheap, fast way to read text and gives you positions and confidence scores. What is fading is the template-per-layout approach to understanding which text is which field. LLMs are better at that part.

How accurate is LLM extraction? It depends on your documents, so I won’t give you a single number. Published results vary by document type, quality, and how accuracy is scored. Run your own test and score by field, not by page. Pay special attention to confident wrong answers.

When is a hybrid pipeline better? When your documents vary in layout and quality, or when a wrong value is expensive. OCR gives positions and a check that a value really appears on the page. The LLM handles layout variety. Together they catch more errors than either alone.

How do I route low-confidence fields? Combine signals: OCR confidence, cross-checks like totals matching line items, and agreement between methods. Set stricter thresholds for fields that move money. Send only the flagged field and the page region to a reviewer, and sample auto-accepted fields each week.

How do I test on my documents? Collect 150 to 300 real files, including bad scans. Label the correct field values. Run each candidate pipeline, score per field, sweep the confidence threshold, and cost the review time. Confirm on a fresh held-out batch.

Three Ways to Work With TMA

Need an agent built? We deploy production AI agents in your infrastructure. Working pilot. Real data. Measurable ROI. → Schedule Demo

Want to co-build a product? We’re not a dev agency. We’re co-builders. Shared cost. Shared upside. → Partner with Us

Want to join the Guild? Ship pilots, earn bounties, share profit. Community + equity + path to exit. → Become an AI Architect

Document Extraction: LLM vs OCR for Business Files