Intellics
Two trays on one platform: a sparse tray of loose tokens and a full tray of tagged tokens

In our test, web search found the single figures. It could not screen a sector.

In short. We asked twelve multi-company questions about Japanese listed companies to two assistants from the same model family: Assistant A with web search, Assistant B with Kabuken's MCP server and no web search. With web search, 237 of 245 scored figures matched the filing. But Assistant A read no annual securities report (有価証券報告書) itself, and none of its figures carried a filing document ID: they came from third-party sites, news and results summaries. With Kabuken, 253 of 253 matched, and each figure carried the EDINET document ID of its filing. The largest gap was in sector screens. This is a small test with real limits, and we list them below.

The difference we saw is not whether one figure is right. It is whether the reader can open the exact filing behind each figure, and whether a whole sector can be listed at all.

Why a third round

In round one, three assistants each gave five headline figures for ten large companies. All 150 figures matched the filing.

In round two, one assistant with web search answered harder single-company questions: segments, prior years, smaller companies, the latest period, and Japanese prompts. 40 of 45 items were correct and it fabricated nothing. But it read no 有報 itself for any of its 48 figures. They came from earnings releases (決算短信), company releases, news and data sites, and it flagged most of them as unverified.

So single-company lookups did not separate a web-search assistant from a filing database. Round three asks for what an analyst often wants instead: a table across several companies.

We did not test an assistant that reads the filing itself. Other tests suggest a current model reads single figures well when it has the filing. Our question was what happens when the assistant only has web search.

What we asked

Ten prompts in English: three sector screens, three comparisons of five to seven named companies, two five-year series, and two leverage or margin checks. Two more prompts repeat one screen and one comparison in Japanese. A screen reads, for example: pharmaceutical companies with ROE of 15% or more and total liabilities at most half of shareholders' equity, top ten by ROE, with the source document for each row. The prompts name companies and securities codes only. They do not mention EDINET or Kabuken.

Diagram: 12 prompts (10 EN + 2 JA) go to Assistant A with web search and to Assistant B with Kabuken MCP and no web search; correct answers fixed before any answer was read and checked against original EDINET XBRL; both are scored with the same rules and rounding tolerance

We fixed the correct answers before we read any answer. We built the answer key with Kabuken's tools, which favours Assistant B. So we then checked the key independently against the original EDINET XBRL files, with no Kabuken code on that side: all 364 values in 64 filings matched exactly. Both assistants were scored with the same rules. A figure counts as correct when it equals the filed value within the rounding of the precision it was written in.

Two parallel lanes from one start block to one shared checking arch

Results

Assistant A: web search Assistant B: Kabuken MCP
Scored figures that match the filing 237 of 245 253 of 253
Figures that differ from the filing 6 (5 asked, 1 extra) 0
Filed figures not given 2 0
Fabricated figures 0 0
Figures with a filing document ID 0 253
Rankings in the right order 6 of 6 6 of 6
Screen recall (four screens) 0, 0.33, 0, 0.33 1, 1, 1, 1 (indicative, see below)
Japanese prompts fully answered 0 of 2 2 of 2

Diagram: Assistant A, web search: 237 of 245 figures match, no filing read itself, 0 figures with a document ID, screens: 1 of 3 found twice, no list twice. Assistant B, Kabuken MCP: 253 of 253 figures match, 253 figures with an EDINET document ID, screens: every listed company found, indicative only

Where the gap was: screens

Assistant A said it could not enumerate a sector from the web. It wrote that the web has no sector-wide ranking of these ratios and that it could not open the annual reports of the roughly 100 companies in the sector. That is a limit of web search, not of the model: an assistant that only searches has no list to enumerate from. For the pharmaceutical screen it confirmed one of the three passing companies, in English and in Japanese. For transport equipment and for retail it gave no list. In the transport-equipment answer it wrote in bold "Verified: no company passes both criteria", then narrowed this to the few names it had checked. Kabuken's screen lists five companies in that sector that pass.

Assistant B used Kabuken's screen tool and listed every company on the list of correct answers, in all four screens.

One caution belongs here, not in a footnote. For the screens, the list of passing companies came from Kabuken's own screen. Assistant B used that same screen, so its full recall partly shows that it read the tool correctly. Read the screen numbers as indicative, not as proof. The figures in each listed row were checked against the original filings.

Where the figures came from

Pie chart: Assistant A, source of each figure (248 transcribed, 3 of them not scored): data aggregator sites 144, own arithmetic 41, earnings releases 32, news 29, company releases 2

Pie chart: Assistant B, source of each figure (253): read from Kabuken tool output with document ID 207, own arithmetic on those sourced inputs 46

Assistant A's figures came from third-party sites, news and results summaries, plus its own arithmetic. Some of those sites are themselves built from the 有報, but Assistant A read no filing itself and gave no filing document ID. In all twelve answers it said that it had not checked its figures against the 有報. Assistant B read 207 figures directly from Kabuken's tool output and computed 46 from those sourced inputs. Every figure carries the document ID of its filing, so anyone can open the same filing on EDINET. Ten of those figures cite an amended report (訂正有価証券報告書) for one company. Assistant B pointed this out itself.

A tray of square tiles, each with a small seal in its corner, and one stamp beside it

What web search did well

Assistant A put every comparison in the right order (6 of 6). It invented no figure, and it said plainly where it was unsure. Four of its five differing asked figures were rounding or truncation slips of less than ¥100 million, on figures of hundreds of billions or trillions of yen. For example, it wrote Toyota's revenue as ¥50,684.9 billion against a filed ¥50,684,952 million. The fifth was a different value: NEC's equity ratio as 47.3% against a filed 49.2%. The Japanese prompts were weaker: in the Japanese comparison, two filed equity ratios were left unconfirmed and one was wrong (9 of 12).

What this does not show

  • Screens are partly circular. The list of passing companies came from Kabuken's own screen (see above).
  • Different harness. Assistant A ran once through an API, with a cap of 15 searches per prompt. Assistant B was run by an agent session that could retry and cross-check. Its two Japanese answers reused the tool results of the English runs.
  • Seven days apart. Assistant B ran seven days after Assistant A, against the same fixed answers.
  • One model family. Both assistants are from the same model family. The difference shows the data path, not model skill.
  • Small sample. Twelve prompts. The named companies are large companies with a March year-end.
  • Definitions differ on screens. Screen ROE is Kabuken's own calculation on average equity, not the ROE the company reports. Assistant B said so in each screen answer. A web user rarely gets that basis, so the screens also compare definitions.
  • Assistant B's gap. On screens, Assistant B left the securities-code column blank, because the tool returned EDINET codes.
  • Not investment advice. Kabuken has no stock prices.

Takeaway

For one figure about one company, web search did well in our rounds. For a table across a sector, web search gave the assistant nothing to enumerate from, and no figure carried a filing document ID. In this run, the filing database answered the same tasks completely, with a document ID for each figure that anyone can check on EDINET.

When to use Kabuken

Kabuken answers from the filings that Japanese listed companies submit to EDINET. Use it to:

  • Screen a whole sector from your AI assistant on filed figures and list the companies that pass. Web search has no list of a sector's companies to work through.
  • Ask what the annual securities report (有価証券報告書) says about officers and board candidates at the AGM, female managers, male childcare leave, the gender pay gap, major shareholders, owner categories, policy shareholdings, segments and notes, or find any single figure in a filing step by step.
  • Check and cite each figure. It comes with the EDINET document ID of its filing, so you can open it.

For one headline figure of a large company, web search usually gets it right.

The tools behind these jobs

Sector screens use screen_companies. Officers and human capital come from get_company_financials (include: ["officers"], ["human_capital"]). Top holders, owner categories and policy shareholdings come from get_major_shareholders. Segments come from get_segment_breakdown. Any single fact, including the notes, comes from get_financial_facts, one level per call or by a query on labels and concept names. Field reference: https://kabuken.intellics.ai/docs/en/api/tools/?utm_source=blog&utm_medium=referral&utm_campaign=2026-11-16-verify&utm_content=docs_tools

Kabuken: https://kabuken.intellics.ai/?utm_source=blog&utm_medium=referral&utm_campaign=2026-11-16-verify&utm_content=kabuken_home

Source: EDINET (Financial Services Agency, Japan), Public Data License 1.0. Processed by Kabuken. A Kabuken experiment by Intellics K.K.