Same filing, same rules, two models: one passed our pre-set test and one did not
In short. We asked AI models to play investor personas with a written rule and read Japanese annual securities reports with key facts changed. The test was fixed before the runs. Claude Haiku 4.5 did not meet it. Claude Sonnet, on the same cells under the same rule, did. Most of the gap sits in how each model reads the unchanged filing. Both Claude models cited nearly every fact by its exact source row. Simulated personas only: not real investors, not a forecast, not investment advice.
What we did
Five annual securities reports (有価証券報告書) of large listed Japanese companies, here Company A to Company E. For each, a fixed evidence pack from Kabuken's MCP server, which serves data taken from EDINET. No prices, no news, no company names in the test packs.
Twenty personas read each pack. Each persona has a short written rule: a passive index holder, a dividend-focused holder, an active manager, and so on. Each picks one action (hold, add, trim, exit or no reaction) and cites every fact it relied on by its exact source row.
Then we changed key facts in the pack, strongly and consistently: a large fall in revenue, a loss, lower equity, a deep dividend cut. For one company we did the opposite, a return to profit. The question: do personas whose rule depends on those facts move in the direction their rule implies, while passive personas stay put?
We wrote the test down before running it: which personas should move, in which direction, and how many had to do so for the result to count. We did not change it afterwards.
Result one: Claude Haiku did not pass
With Claude Haiku 4.5 (extended thinking on), the test was not met. In the first experiment, which made small single-fact edits, it was not met for any filing. In the second, with strong consistent changes, fewer than half of the personas expected to move did so in the expected direction. Our rule called that "mixed".
Result two: Claude Sonnet, same cells, same rule, did pass
We then ran Claude Sonnet (extended thinking on) on exactly the same 400 persona-and-condition cells. Same packs, same persona files, same rule. This time the test was met. A clear majority of the expected movers moved in the direction their rule implies, none moved the opposite way, and the personas that should stay put stayed put.
It is tempting to say Sonnet understands shocks better. We do not read it that way. After the strong changes both models mostly chose to trim. The larger difference was in what they chose before anything was changed.
Citations held up
Nearly every fact either Claude model cited matched its source row exactly. The models differed in discipline: Claude Sonnet cited almost every number it used in its reasoning, while Claude Haiku left noticeably more numbers uncited. Gemma 4 26B, tried on a small pilot, cited less reliably, so we did not use it further. Passive personas stayed passive with both Claude models.
What this does not show
- It does not show what real holders do. The personas are simulated.
- It is not a prediction, a ranking or investment advice. Kabuken has no stock prices.
- Five large companies only. Smaller companies, realistic news or other models may behave differently.
- The changes were strong and synthetic, which is easier than real filings.
- We removed names and codes but not the size of the numbers, so a model might still recognise a very large company. We did not test that.
- In the second experiment each cell was repeated only twice, and Sonnet's repeats were nearly identical, so its effective sample is smaller than it looks.
- Sampling settings were not controlled; repeats are independent samples.
- In the first experiment, an explanatory note field reached the personas in garbled form. We found it later, fixed it for the following runs and did not redo the first.
Takeaway
The honest takeaway is narrow. On strong, synthetic changes to large-company filings, the same personas moved as their rules imply with one model and mostly did not with the other, and the difference sits mostly in the baseline reading. The part we would build on is the evidence-checking: a model that cites each fact by an exact source row can be checked against the filing.
What's next
We are exploring collaborative use: for example, teams reading the same filing and comparing their readings side by side. This is exploration only. There is no date and no promise, and nothing described here is a product feature today.
Read the full write-up
Method, findings and limits: https://labs.intellics.ai/2611-crowd-lab/en/?utm_source=blog&utm_medium=referral&utm_campaign=2026-11-11-crowd-lab&utm_content=lab_link
Source: EDINET (Financial Services Agency, Japan), Public Data License 1.0. Processed by Kabuken. A Kabuken experiment by Intellics K.K.