A language model wrote the numbers. A script checked them. Then we checked the script.
Why a number checker
Our /report product carries the results of recent macro events: CPI, payrolls, Fed decisions. A language model reads the coverage and writes one line per event. Agents that buy the line usually do not reopen the release. They take the number as given.
That is where language models are weakest. A number can be the forecast instead of the result, last month’s figure instead of this month’s, a figure from the next line of a table, or the right figure in the wrong unit. Each looks perfectly fine in a sentence.
So since October 17, 2026, every result passes a source-matching check before it is stored. Each number must appear on the official release page, near the name of the indicator, in the same unit. A number that fails is removed and we get an alert. If the official page cannot be fetched, nothing is stored.
Did the checker actually work? We had only tested it on the cases we had imagined.
How we tested it
We took the 16 official releases the checker could have seen between August 27 and November 6, 2026:
- BEA: two GDP releases and three personal income and outlays (PCE) releases
- BLS: three employment reports, two CPI releases, one PPI release and one JOLTS release
- Census: one retail sales release
- Federal Reserve: two FOMC statements
- Bank of Japan: one policy statement
For those releases we had 19 result lines: 15 that we stored and 4 drafts the checker had already rejected. Together they contain 56 numbers. We did not ask the model again. We re-ran the checker on the same text against the same official pages.
We tested in two directions. First, does it keep the numbers that are correct? Second, does it catch numbers that are wrong? For the second test we changed one number at a time, for example adding 0.3 to a percentage or 25 to a basis-point move, and checked whether the checker removed it.
What we found
Of the 56 numbers, the checker removed 15. Ten of them were correct official figures:
- the September FOMC move, “+25bp to 3.75–4.00%”, in two stored lines (six numbers)
- Q3 GDP, 3.4%
- Q3 core PCE prices, 2.5%
- August core PCE, 3.0%
- retail sales excluding autos, +0.9%
The other five sat in clauses about forecasts or the Fed’s 2% target, which an official release cannot confirm.
Eight of the ten correct figures were in results stored before the check began on October 17, so nothing changed for them. The other two were removed in live use: Q3 GDP 3.4% and core PCE 2.5%. From October 30 to November 5, our Q3 GDP result was served with only the PCE price line. No incorrect official figure was served. We have logged this on our incident record.
In the other direction, we planted 35 wrong values. The checker removed 32. Three got through.
Four reasons it rejected correct numbers
PDF text breaks numbers apart. We read BEA’s GDP release from its PDF. When we extracted the text, “3.4 percent” came out as “3.”, “4”, “p” and “ercent”. There was nothing left to match.
A trailing zero. The PCE release said “3.0 percent”. Our matcher mishandled the “.0” and could not match it to “3.0%”.
Basis points versus percent. Our line said “+25bp”. The FOMC statement says the Committee raised the target range “by 0.25 percent to 3.75 to 4 percent”. The checker did not know that 25 basis points is 0.25 percentage point. The clause failed, and “3.75–4.00%” went out with it.
Tables, not sentences. In the Census release, retail sales excluding autos appear as a table row (“Total (excl. motor vehicle & parts)”), not in a sentence. The checker read only sentences.
None of these were judgment calls. All four were about reading the page.
The holes on the other side
A number from the next sentence. We changed PPI core to “+0.8%”. The checker accepted it, because 0.8 appeared in a nearby sentence about services. Core figures must now sit in the same sentence or table row as the word that defines them.
Last period’s figure. Releases quote the previous period for comparison, as in “following an average monthly gain of 45,000”. A number like that could vouch for a wrong current figure. Numbers introduced by “compared with”, “following” or “after” no longer count as a match.
Numbers we never checked. The other two leaks were in our October FOMC line, “3-3/4 to 4 percent in 10-2 vote”. We changed the vote to “10-9” and the fraction to “3-10/4”. Nothing objected, because counts, votes and fractions were not checked at all. They are checked now.
The checker failed in both directions. It deleted correct figures and let wrong ones through.
We had tested it only against what we expected to go wrong.
After the fix
We fixed PDF and table parsing, basis-point matching, the trailing zero and the three holes on November 7, added a regression test for each, and ran the same back-test again.
- No correct official figure was rejected because of a parsing error.
- Six numbers are still removed. Four are forecasts or expectations, one is the Fed’s 2% target, and one is a correct figure: July PCE inflation of 3.7% shares a clause with “+0.1pp vs forecast”, and the checker removes whole clauses.
- We planted 45 wrong values with a sign or unit. All 45 were removed. We planted 7 wrong counts, votes and fractions. All 7 were removed. That makes 52 of 52.
The total is higher than before because more numbers now survive the check, so there are more of them to corrupt.
We did not rewrite past records. A wrong result is corrected in a separate, append-only correction file, and the original line stays.
The same week brought a separate incident. On November 7, our check removed “average hourly earnings +3.1% y/y” from the October jobs result and stored “+165,000” as “+165, 000”. One paid response carried the incomplete line before we corrected it at 13:41 KST. That incident has its own entry on the record.
What it still can’t do
- The Bank of Japan publishes its English policy statement only as a PDF, with no HTML version. We recorded the October 30 BOJ result as unverified rather than guess.
- The Labor Department’s weekly jobless claims PDF is replaced every week, so we could not back-test past weeks. From November 8 we archive each release on our side. The Census retail PDF we check is also the “current” file, which is replaced every month.
- Numbers without a sign or unit are checked only when they are counts, votes or fractions. Other bare numbers pass unchecked.
- The checker removes clauses, not single numbers, so one bad comparison can take a correct figure with it.
- Sixteen releases over three months is a small sample. A release format we have not seen can break the parser again.
Three lessons for agent builders
Test the checker in both directions. A checker tested only on bad data will quietly delete good data. A checker tested only on good data will wave bad data through. Ours did both. Replay real past inputs, then plant errors in them.
Parse the source the way it is published. Most of our false rejections came from PDFs, tables and the same number written in another unit (“+25bp” against “0.25 percent”). The model had the number right. Our reading of the page was wrong.
When the check fails, withhold and say so. We would rather serve a shorter line than a wrong number, and we would rather publish the incident than hide it. Keep the original, record the correction separately, and let the buyer see what changed.
Official releases used in the back-test: BEA GDP Q2 second estimate and Q3 advance estimate; personal income and outlays for July, August and September; BLS employment situation for September 4, October 2 and November 6; CPI for September 11 and October 14; PPI, October 15; JOLTS, November 3; Census retail sales; FOMC statements of September 16 and October 28; BOJ statement, October 30.
Incidents and corrections: asia-pulse.com/transparency. What we publish: aggregate counts, the four causes and the fixes. What we keep private: the test harness and the stored result lines.
Cited with attribution to Asia Pulse.
Since September 26, 2026, Asia Pulse has recorded a daily fingerprint of every paid output on X Layer: 0xb759Ea689FF49A6781232198fcAdD6fdfE0680cd