CiteVQA: Benchmarking Evidence Attribution for Trustworthy Document Intelligence

Today’s paper addresses the evaluation of Multimodal Large Language Models (MLLMs) in document understanding tasks. Specifically, the paper tackles the problem of answer-only evaluations in Document Visual Question Answering (Doc-VQA). In high-stakes domains like law, finance, or medicine, it is critical that models not only provide the correct answer but also ground it in the precise source material. The paper introduces CiteVQA, a comprehensive benchmark designed to evaluate both final answer accuracy and visual evidence attribution simultaneously.

Method Overview

The method constructs a large-scale, automated benchmark by synergizing document parsing tools with MLLM agents. The main idea is to generate complex question-answer pairs while rigorously tracing the exact location of the supporting evidence within the document. Instead of relying on expensive and inconsistent manual annotation, an automated pipeline processes large sets of documents, extracts connected pieces of information, and formulates verifiable questions alongside element-level bounding-box citations. This ultimately results in a dataset of 1,897 questions across 711 diverse multi-page documents.

The process begins by linking multiple documents based on semantic similarity to create realistic, cross-document reasoning scenarios. Next, an intelligent MLLM agent navigates the parsed layout of these documents to collect an “Evidence Package,” which includes text, tables, images, and their exact visual coordinates. Using distilled templates derived from real-world datasets, the system then synthesizes realistic queries that can be answered using the collected evidence package.

To ensure high quality and filter out common-knowledge guesses, the paper implements strict quality control steps. This includes an ablation procedure where evidence elements are individually masked; if masking a specific element prevents a strong model from answering correctly, that element is flagged as crucial evidence. Finally, the paper proposes a new evaluation metric called Strict Attributed Accuracy (SAA). SAA credits a prediction only if the model produces both the correct textual response and accurately cites the precise visual region, ensuring reliable traceability.

Results

The evaluation of 20 state-of-the-art MLLMs revealed a widespread issue termed “Attribution Hallucination,” where models frequently output the correct answer while citing entirely wrong locations within the document.

There is a significant performance gap between model tiers. The strongest closed-source system (Gemini-3.1-Pro-Preview) achieved an SAA of 76.0, whereas the strongest open-source model reached an SAA of only 22.5. Furthermore, multi-document scenarios proved significantly more challenging for evidence localization than single-document tasks, causing a stark drop in localization metrics even when answer correctness remained stable. Smaller open-source models struggled severely, often scoring below 10.0 on the SAA metric, highlighting the unreliability of deploying them in domains requiring strict source verification.

Conclusion

In summary, the paper highlights a critical flaw in current Doc-VQA evaluation frameworks and bridges this gap with CiteVQA. By transitioning from answer-only scoring to joint answer-evidence verification, the benchmark exposes significant hallucination vulnerabilities in modern models. The introduction of the Strict Attributed Accuracy metric and a scalable dataset generation pipeline provides the necessary tooling for advancing more trustworthy, transparent, and verifiable document intelligence systems.

For more information please consult the full paper.

Congrats to the authors for their work!

Ma, Dongsheng, et al. “CiteVQA: Benchmarking Evidence Attribution for Trustworthy Document Intelligence.” arXiv preprint arXiv:2605.12882 (2026).