The Five Questions Your Supply Chain AI Cannot Answer by Design
A risk manager asks: if TechChip Inc fails, which products are affected and how severe is the cascade? The RAG system returns three documents. They mention TechChip. They mention several components. The answer looks reasonable. What it cannot contain is the correct answer, because the correct answer requires tracing a path through a graph: supplier to component to factory to product to customer, weighted by hop distance and severity. That path does not exist in any document. It exists in the topology of the supply chain itself, and the RAG system was never built to read topology.
This is not a hallucination. Nothing was fabricated. The system retrieved real documents accurately. The failure is architectural: five categories of questions that supply chain operators ask every day are, in the precise technical sense, structurally unreachable for single-pass vector retrieval. Not hard. Not inaccurate. Unreachable, the way a car cannot fly a route that requires altitude.
Grama Chethan, an architect at Siemens Digital Industries Software, published a detailed empirical study in June 2026 testing eight retrieval architectures against 23 supply chain intelligence queries. The headline finding is not a marginal accuracy improvement. Standard RAG answers zero of eleven structural supply chain queries correctly. A dense embedding baseline answers one. The paper's most important architectural contribution, a typed graph planner using nine traversal primitives, achieves an F₁ of 0.632 on the full 23-query benchmark, compared to 0.472 for a hand-coded deterministic system and 0.577 for a ReAct agentic system. The difference between the worst and best LLM-based architecture is not model intelligence. It is the set of computational tools the model was given to work with.
Five Query Categories That Vector Similarity Cannot Reach
The paper formalizes something that practitioners in complex industries already feel but rarely articulate precisely: certain questions require graph computation, and no amount of better embeddings addresses that requirement.
The five formally defined categories are what-if and counterfactual queries (remove this supplier, what breaks?), single point of failure detection (which components have exactly one supplier?), inverse and negative queries (which customers are NOT affected by this disruption?), comparative subgraph analysis (how does Product A's supply chain differ structurally from Product B's?), and risk propagation scoring (which product faces the highest weighted risk exposure right now?). Each fails for a distinct architectural reason:
- Absence blindness: RAG can only find what matches. It cannot represent or reason about what is absent from the graph. A query asking which customers are unaffected requires computing a complement, which requires knowing the full entity set.
- Propagation blindness: Weighted hop-distance scoring across a supply chain requires traversing multiple edges and accumulating a score. No single document contains this computation. It must be executed against the graph structure.
- Topology blindness: Comparing two products' supply chain depth, upstream supplier count, and geographic distribution requires extracting and diffing two subgraphs. Text similarity returns documents about each product. It cannot diff their structural properties.
The empirical evidence for these claims is direct. Standard TF-IDF RAG scored zero correct on eleven original queries. A dense embedding model using all-MiniLM-L6-v2 scored one correct. LightRAG, which extracts 244 entities and 362 relationships from the corpus before indexing, still fails completely on inverse queries and risk propagation in every retrieval mode tested. More graph structure in the index does not solve the computation problem.
Why Smarter Models Do Not Fix This
The paper tested a ReAct agentic system using Claude Haiku 4.5 with four tools: search chunks, lookup entity, get neighbors, and list all entities. It answered five of eleven queries correctly, against zero for standard RAG. That looks like a substantial improvement, and for simple multi-hop queries it is.
But the agentic system failed single point of failure detection by finding only one of fifteen components with a single supplier, a 6.7% recall rate on a question with direct safety implications. It hallucinated suppliers on the what-if counterfactual query. It used 199,148 tokens across eleven queries and averaged 9,480 milliseconds per query. And its apparent success on the inverse query depended entirely on the graph being small enough that listing all entities in a single call returned the complete set. The paper notes explicitly that whether agentic complement computation scales beyond toy-sized entity sets remains an open question.
The agentic system is not reasoning its way to structural answers. It is making do with tools that were not designed for graph computation, and failing specifically where the computation demands graph topology.
The deterministic bespoke handler system, by contrast, answered eleven of eleven original queries correctly. But that result is an artifact of co-design: the eleven queries and eleven handlers were built together, creating a performance ceiling that reflects shared construction, not generalization. When tested on twelve hold-out queries the handlers had not been designed for, performance dropped from an F₁ of 0.574 to 0.379, a 34% relative decline. Each handler required 50 to 200 lines of Python, roughly six intent classifier training phrases, and two to eight hours of engineering time. When a new query category emerges, the approach requires building a new handler from scratch.
What Nine Typed Primitives Change
Architecture 7 in this paper replaces all eleven bespoke handlers with a single LLM query planner that receives the graph schema but not the data, and selects from nine typed traversal primitives. The result is an F₁ of 0.632 on the full benchmark, compared to 0.472 for the bespoke system and 0.577 for the agentic approach. More importantly, the LLM planner improves from F₁ 0.557 on original queries to 0.700 on hold-out queries the system had never seen. The bespoke system degraded. The planner generalized.
The nine primitives are:
- find_nodes: Scan by entity type and attribute filters
- get_node: Single node attribute lookup
- get_neighbors: One-hop traversal with edge type filtering
- shortest_path: Undirected shortest path between two nodes
- subgraph: Multi-hop breadth-first search with direction control
- count_edges: In and out degree by edge type
- set_complement: All nodes of a type minus a specified subset
- filter_edges_by_date: Temporal edge filtering for active vs. expired relationships
- propagate_risk: Weighted hop-distance scoring across a supply chain
The operator vocabulary thesis, which the paper calls its central finding, follows directly: the LLM does not need to be smarter to solve structural queries. It needs the right tools. Given the right primitives, it selects and sequences them correctly without additional instruction.
Architecture 8 adds six graph computation tools: simulate removal for what-if queries, subgraph diff for comparative analysis, aggregate over type for counting queries, betweenness centrality, PageRank, and connected components. The headline F₁ moves from 0.635 to 0.636, which is flat. The paper is explicit that this conceals a qualitative breakthrough rather than contradicting one. Entity-level F₁ scoring penalizes comprehensive correct answers as false positives because a complete aggregation answer must name every entity in a ranking, and the scorer counts additional entity mentions against the system. The paper identifies this as a fundamental limitation of entity-level F₁ for structural queries, and it is the honest limitation a promotional summary would most likely omit.
The Temporal Trap Buried in Your Existing Index
There is a sixth failure mode the paper addresses that does not fit neatly into the structural query taxonomy: temporal blindness. When a supplier relationship expires and a new one begins, standard RAG preserves both text chunks in the index with no mechanism to determine which reflects current reality. A 2023 procurement document and a 2024 contract transition notice are retrieved together and weighted by similarity to the query, not by validity.
The paper's incremental update architecture addresses this with an expire relationship operation that removes the edge from the traversable graph while deliberately preserving the stale text chunk in the corpus. The stale chunk remains, ready to mislead any system using text-based retrieval. This is not a bug in the implementation. It is a demonstration of the structural gap: the graph knows the relationship has expired; the text index does not and cannot.
The paper acknowledges its own temporal model is deliberately simple, using binary active or expired status on edges rather than the overlapping validity intervals and bi-temporal modeling that production supply chain systems require. The gap between a proof-of-concept temporal layer and a production-grade one is real and worth naming.
What This Looks Like in a Real Risk Assessment
The paper's risk propagation scenario is concrete enough to follow. Three risk events have activated: a Thailand flood, a components shortage, and a geopolitical disruption. Each affects one or more suppliers. The question is which products face the highest weighted risk exposure right now, accounting for the number of affected components, the severity of each event, and the hop distance between each event and each product.
Standard RAG returns zero. The agentic system fails because it has no propagation tool, only a neighbor lookup. The deterministic GraphRAG computes the correct answer using a hand-coded handler. Architecture 7, using the propagate_risk primitive, matches that answer on the original query and achieves F₁ 0.860 on risk heatmap queries.
The correct answer is specific: WideBird-X50 carries the highest risk score at 1.19, exposed through four factories and twelve components to all three affected suppliers. RegionalJet, ExecWing, and SkyPatrol score 0.98 each. CargoHawk scores 0.70. NarrowBody scores 0.49. DefenseTech Corp is the sole unaffected customer. A risk manager asking this question against a standard RAG system gets a document about Thailand. A risk manager asking it against Architecture 7 gets a ranked exposure table they can act on.
The cost difference between those two outcomes is not hardware or model size. It is whether the risk propagation primitive exists in the tool library.
What an Operator Vocabulary Strategy Actually Requires
The paper is a single-domain study on a synthetic 46-node knowledge graph. The authors acknowledge this directly, and it matters for how executives read the findings. The LLM architectures were only tested at 46-node scale. The deterministic engine was scale-tested to 1,100 nodes, where aggregation query latency reached 10 milliseconds at the 95th percentile, still well within real-time thresholds. Whether LLM-based planning degrades at larger scales is explicitly stated as an open question.
The findings also rely on a pre-defined schema. The approach requires knowing your supply chain ontology in advance: what types of nodes exist, what types of edges connect them, what metadata each edge carries. This is not a weakness unique to this architecture. It is the prerequisite for any system that reasons structurally rather than superficially. The practical question for an organisation is whether that ontology work has been done, and for most mature supply chain operations, a significant fraction of it already exists in ERP schemas and data dictionaries.
The strategic implication of the operator vocabulary thesis is that the right investment unit is a tool library, not a model. Architecture 7 outperforms a bespoke hand-coded system by providing nine typed graph primitives instead of eleven custom handlers, and it generalizes where the handlers fail. Architecture 8 extends that library with six computation primitives and the LLM adopts them without additional prompting for precisely the query categories where traversal falls short.
When a new analytical question emerges, adding a tool takes hours. Building a handler takes days. At the scale of questions an enterprise risk function generates, that difference compounds.
The organisations that get this right in the next 18 months will not have trained a better model or built a smarter agent. They will have curated a better vocabulary of graph operations and given their existing LLM the words it needs to reason about structure. Every answer they get after that is a function of that library, not of the model behind it.
Agents Applied covers AI research that changes how organisations make decisions. This analysis is based on arXiv:2606.06003v1 by Grama Chethan, Siemens Digital Industries Software, June 2026.