EY re-envisions RAG around multimodal knowledge graphs to improve accuracy
Retrieval-augmented generation is a standard way to ground large language models in enterprise information, but new research from EY, the business name of Ernst & Young LLP, says most implementations overlook a lot of useful data.
Conventional RAG systems are built mainly to retrieve text. Enterprise documents, however, often place critical facts in charts, tables, engineering diagrams, equations and images. EY has developed a multimodal RAG framework that retrieves those materials alongside text and connects them through a knowledge graph, producing answers that are more complete, contextualized and easier to verify.
The approach doesn’t affect the underlying LLM, but rather changes how enterprise content is prepared, indexed, related and supplied to the model at inference time. The work grew out of limitations EY encountered in client projects, said Dipanjan Sengupta (pictured), EY Global Delivery Services Consulting distinguished technologist and AI engineering leader.
“RAG works well for textual content,” he said, “but in many industries, a lot of information is in illustrative content as well.”
For example, an industrial company may keep essential specifications in engineering drawings, while a life sciences company may rely heavily on graphs. EY’s methodology links illustrations back to relevant text, resulting in a “manyfold” increase in accuracy with improved response narratives, Sengupta said.
Multiple pipelines
The framework begins by separating textual and illustrative content into different ingestion pipelines. Text is segmented and enriched through keyword extraction and named-entity resolution, a process that links records or name variations that represent the same real-world entity. Illustrations are assigned descriptive metadata using existing captions, nearby text, bounding-box analysis, optical character recognition and descriptions generated by a language model.
The system stores text and illustrations in separate vector indexes, narrowing the search space and allowing a query to target text, images or both. At the same time, each content element becomes a node in a knowledge graph with weighted relationships connecting relevant passages and illustrations.
“We establish relationships between the various illustrations and the text segments,” Sengupta said. “We stored the information in a graph database where we had individual nodes for illustrations and for text and rules for establishing the relationships between the two.”
The white paper describes three methods for building those relationships: deterministic keyword matching, semantic similarity based on embeddings, and machine learning-based inference for implicit associations. A “gleaning” process looks for missing links, resolves ambiguous entities and identifies related information across documents. The result is a heterogeneous graph that can support multi-hop reasoning without requiring the LLM to infer every connection.
Retrieval also occurs in stages. The system performs a similarity search against the appropriate modality-specific index. It then uses the resulting identifiers to traverse neighboring nodes in the knowledge graph. The search can remain local, broaden into graph communities or combine both approaches. A multimodal re-ranker orders the retrieved passages and illustrations before inserting them into the LLM prompt.
“We are not just relying on vector search but also enhancing the content space to other modes of information,” Sengupta said. “That gives us a larger narrative.”
‘Far better decisions’
The configuration process differs from that of many RAG deployments. A compliance application may favor narrow, deterministic retrieval, while a research application may benefit from broader semantic exploration.
EY argues that chunking methods, embedding models, relationship-building techniques, reranking strategies and retrieval scope should be configurable rather than hard-coded. Flexibility is one reason Sengupta favors an enterprise platform rather than isolated RAG projects. Common services for ingestion, security, governance and retrieval can be reused while individual business units tune the system for their requirements.
The framework could become more important as enterprises deploy artificial intelligence agents. Those need current, domain-specific information to make decisions and select actions, and weak retrieval can propagate errors throughout an automated workflow.
“When RAG is enriched with multimodal capability, we find that the agents are far better equipped to make decisions,” Sengupta said. RAG can also improve the path an agentic workflow takes.
EY’s paper presents the method as a foundation for more transparent and domain-aware enterprise RAG, but it does not publish comparative benchmarks or quantify the accuracy gains. The evidence so far is client experience and the presumption that an architecture that retrieves connected visual and textual evidence should produce richer answers than text-only search.
Sengupta rejected the idea that expanding model context windows will make RAG unnecessary. Larger windows may reduce the need to split documents into small chunks, he said, but they do not solve the problem of finding the most relevant evidence. Without focused retrieval, an LLM is still searching for “that proverbial needle in the haystack.”
Photo: EY
Support our mission to keep content open and free by engaging with theCUBE community. Join theCUBE’s Alumni Trust Network, where technology leaders connect, share intelligence and create opportunities.
15M+ viewers of theCUBE videos, powering conversations across AI, cloud, cybersecurity and more** 11.4k+ theCUBE alumni**— Connect with more than 11,400 tech and business leaders shaping the future through a unique trusted-based network.
About SiliconANGLE Media
theCUBE AIand theCUBE SuperStudios — with flagship locations in Silicon Valley and the New York Stock Exchange — SiliconANGLE Media operates at the intersection of media, technology and AI.
Founded by tech visionaries John Furrier and Dave Vellante, SiliconANGLE Media has built a dynamic ecosystem of industry-leading digital media brands that reach 15+ million elite tech professionals. Our new proprietary theCUBE AI Video Cloud is breaking ground in audience interaction, leveraging theCUBEai.com neural network to help technology companies make data-driven decisions and stay at the forefront of industry conversations.