AdaGraphRAG
Adaptive hierarchical Graph-RAG with incremental construction and self-verified retrieval
Overview
GraphRAG answers every query with the same expensive global traversal, re-indexes the full corpus whenever a document is added, and never checks whether its generated answer is actually grounded in the evidence it retrieved. AdaGraphRAG extends Microsoft's GraphRAG (Edge et al., 2024) with three fixes for exactly those gaps: an adaptive router that classifies query complexity and picks the cheapest sufficient retrieval strategy, incremental graph construction that entity-resolves new documents into the existing graph instead of rebuilding it, and a self-verification layer that decomposes generated answers into individual claims and checks each one against retrieved evidence — triggering a targeted second retrieval pass on anything unsupported.
Key features
- Adaptive retrieval router classifying each query and selecting none / local / global / hybrid retrieval instead of always paying for global traversal
- Incremental graph construction — new documents are entity-resolved and merged into the existing graph, re-summarizing only the communities that actually changed
- Self-verification layer decomposing answers into claims and checking each against retrieved evidence, with a second targeted retrieval pass on unsupported claims
- 100% precision distinguishing true from false claims on a test set of deliberately injected false statements
- Adaptive routing evaluated against an always-global baseline on a toy corpus, with methodology, metrics, and limitations documented directly in the repository
- Built under a clean-architecture layout — domain, infrastructure adapters, and orchestration kept separate — with a typer CLI and FastAPI service anticipated as the natural next layer
Architecture & engineering decisions
Domain layer of entities and interfaces with no external dependencies, so retrieval, storage, and LLM adapters can be swapped independently
Adaptive retrieval router (Contribution #1) classifying incoming queries before any retrieval work happens
Incremental graph construction module (Contribution #2) entity-resolving new documents into the existing Neo4j graph and re-summarizing only affected communities
Self-verification layer (Contribution #3) decomposing generated answers into claims and checking each against retrieved evidence, with a second retrieval pass for unsupported claims
Qdrant vector store and Neo4j graph store as pluggable infrastructure adapters behind the domain interfaces
Streamlit UI for indexing and querying, with a typer CLI and FastAPI service scoped as natural next additions given the existing layering
Evaluation is a toy-corpus sanity check (2-3 documents, hand-written queries and injected claims) intended to verify the three design decisions behave as intended, not a benchmark-scale claim about accuracy or cost on real-world corpora.
Challenges & how I solved them
An evidence pool was leaking across claims during self-verification, causing O(n²) prompt growth and 30-50s latency spikes on later claims
Traced the leak to shared state in the verifier, fixed it, and documented it plainly in the evaluation write-up rather than quietly patching it out of the results.
Comparing adaptive routing against an always-global baseline on raw chat-completion call count made adaptive routing look more expensive (8 calls vs. 4)
Flagged the metric as incomplete rather than reporting it as a win or a loss — it excludes embedding calls, and the always-global baseline pays for a full community-embedding batch on every query regardless of complexity, which the latency numbers show even though the call-count metric misses it.
The router occasionally under-classified a directly-answerable query as needing no retrieval, which then sent the self-verification layer chasing evidence for ungrounded claims
Identified this as a router-calibration problem rather than a self-verification bug, and scoped it as future work needing a labeled query set instead of patching around the symptom.
Lessons learned
- Writing the evaluation script surfaced three real bugs (an evidence-pool leak, a non-recursive cache-clear glob, and a Neo4j cartesian-product warning) that the architecture alone hadn't revealed — evaluation is also a debugging tool.
- An incomplete cost metric is worse than no metric if it's reported without its gap stated — the call-count comparison needed the embedding-cost caveat to mean anything.
Stack