ResearchPractical AIProjectsOpen SourceAbout
Research/The AI Could Read the Answer. The Search Pipeline Had to Find It First
RetrievalAI EvaluationKnowledge BasesOriginal Research

The AI Could Read the Answer. The Search Pipeline Had to Find It First

I tested a chatbot on 96 questions about a fictional knowledge base. Giving it the whole parent article after search raised accuracy from 78% to 89%; giving it the answer sections directly reached 99%.

Bart Porcincula · Published

Finding

On 96 questions over a synthetic HTML knowledge base, Claude Haiku 5.5 scored 78% with heading-based retrieval and 89% when the top three parent articles were included; the paired difference was 10.4 percentage points (95% bootstrap interval 3 to 19). Given gold sections, it scored 99%.

Synthetic knowledge-base retrieval pipeline comparison (November 2026)

96 questions over 58 fictional HTML articles, using one Haiku 5.5 configuration. Pipeline stages were frozen after a retrieval-only assessment and a 10-question pilot. Scores were checked by automated scorers after an author audit and corrections.

MeasurementValueNote
Raw HTML, fixed chunks58%96 questions
Cleaned text, fixed chunks70%96 questions
Heading chunks with path78%96 questions
Heading chunks plus parent articles89%96 questions; paired improvement 10.4 points, 95% interval 3 to 19
Gold sections supplied99%96 questions; diagnostic, not a deployable search pipeline
Entire knowledge base in prompt94%48 questions, not directly comparable to the 96-question percentages

The short version

A knowledge-base chatbot can only answer from the information its search finds. In this test, improving what search delivered mattered more than changing the model. A pipeline that retrieved heading-based chunks scored 78% on 96 questions. Expanding the top three distinct hits to their full parent articles raised that to 89%. When the correct sections were supplied directly, the same model scored 99%.

The knowledge base and its policies were fictional. This is one model and one synthetic dataset, not a promise of the same improvement on a real company's documents.

What I tested

I built 58 fictional HTML articles and 96 questions, including direct lookups, tables, qualifications, conflicting versions and questions the documents could not answer. Claude Haiku 5.5 answered using different versions of the same retrieval pipeline: search found relevant text, then the model used that text to answer.

The baseline used raw HTML and fixed-size chunks. Later versions cleaned the HTML, split by headings and added heading paths. The parent-expansion version searched the heading chunks, then supplied the full articles containing the top three distinct hits. Each stage changes the context the model receives; the comparison does not isolate every design choice independently.

Results

Material given to the modelAccuracyQuestions
Raw HTML, fixed chunks58%96
Cleaned text, fixed chunks70%96
Heading chunks and paths78%96
Heading chunks, expanded to parent articles89%96
Correct sections supplied directly99%96
Entire knowledge base in the prompt94%48

Parent expansion improved accuracy by 10.4 points; a paired bootstrap interval ran from 3 to 19 points. It helped with qualifications: the qualifier question class rose from 40% to 93%. Supplying the whole knowledge base used much more context; its apparent advantage over retrieval on the 48-question subset was not statistically clear.

Why a correct answer can be missing

Some fixed-size chunks ended mid-sentence. In several failures the model said the source did not contain enough information instead of inventing an answer. That is safer than fabrication, but it still leaves the user without an answer. A separate diagnostic that supplied the correct sections reached 99%, suggesting retrieval was a major source of errors in this particular test.

A second small test compared ways to update the index after articles changed. On 20 questions, replacing all chunks for each changed article scored 85%, versus 70% for a naive chunk update; the difference was concentrated in questions about deleted articles. That sample is too small to treat the difference as a general estimate.

Limits

The corpus was synthetic, cleaner and more consistent than a real site. The main arms used one model and one run each, with one repeat of the cleaned-text arm. The author audited about 30 answers, corrected two scorer defects and rescored the cached answers; this was not an independent audit. No reranker or managed search service was tested. The combined country and superseded-document filter was not tested on top of the parent-expansion arm.

The practical lesson is to inspect what search actually gives the model and test missing qualifications, stale documents and unanswered questions before assuming the model itself needs replacing.

Cite this research

Bart Porcincula. "The AI Could Read the Answer. The Search Pipeline Had to Find It First." Published 2026-11-13; updated 2026-11-13. https://porsync.com/research/knowledge-base-ai-retrieval-parent-article-expansion

Permanent article URL