Time-Aware Legal RAG

What You'll Learn

• Why ordinary RAG can retrieve a current rule when a historical version governs the question.
• What the 312-pair German statutory QA paper actually tested and what it did not prove.
• How as-of date extraction, version filtering, citations, and abstention fit into a production pipeline.
• Which tests and monitoring controls matter before a legal RAG system is used in consequential work.

What Time-Aware Legal RAG Is Designed to Fix

Time-aware legal RAG exists because legal text is not a static knowledge base. A statute can be amended, repealed, renumbered, or interpreted differently across periods. A question about an incident, contract, filing, or compliance decision may require the version that was in force on the relevant date, not the newest text available today.

Two failure modes are easy to confuse. Post-cutoff staleness occurs when a model applies an old rule after an amendment because the relevant change was outside its training or indexed knowledge. Recency bias runs in the opposite direction. The system finds the newest provision and applies it to a historical fact pattern where an older version governs.

Ordinary RAG helps with freshness, but freshness is not the same as temporal validity. A retriever that ranks documents by semantic similarity and recency can return a current statute, a later amendment notice, and a commentary page that describes the present rule. That may be wrong for a question anchored to a prior date.

The design problem is therefore a constrained retrieval problem. The system needs to extract or receive the relevant fact date, identify the legal provision and jurisdiction, select versions whose in-force intervals contain that date, and show the passages used for the answer. If any of those steps fail, the safe output is a review request rather than a confident conclusion.

What the Cited Paper Actually Tested

The paper behind the old article is Asking For An Old Friend: Diagnosing and Mitigating Temporal Failure Modes in LLM-based Statutory Question Answering by Max Prior, Andreas Schultz, and Matthias Grabmair. It was submitted to arXiv on May 22, 2026 and presented at the International Conference on Artificial Intelligence and Law according to the arXiv record.

The authors built a benchmark of 312 expert-validated, time-sensitive German statutory question-answer pairs. The questions cover three categories: post-cutoff amendment questions, pre-amendment questions, and multi-provision pre-amendment questions. The benchmark is intended to expose whether a model uses the wrong legal version when the question's facts require temporal reasoning.

The paper evaluates five large language models from OpenAI, Anthropic, and DeepSeek under four settings. The settings include vanilla generation, web search, and two retrieval-augmented approaches that enforce temporal validity with fact-date extraction and version filtering. The paper calls those RAG variants RAG-kNN and RAG-ToC.

The abstract reports severe degradation in vanilla post-cutoff settings. It says both RAG approaches substantially improve performance across the question types, while web search produces unstable gains and a marked recency bias on historically anchored tasks. That is a useful research result. It is not a universal production accuracy guarantee.

The old article's claim of 99.2% citation accuracy in production is not supported by the paper's abstract or source record. The paper does not provide a universal accuracy figure that can be used as a product promise. A senior developer should treat the benchmark as evidence for a failure mode and a mitigation direction, not as a certification.

Why Current RAG Can Still Misapply Historical Law

A basic legal RAG pipeline often treats a document as one retrievable object with one embedding and one publication date. That representation is convenient and dangerous. A statute may have several consolidated versions, each with a start date and an end date. An amendment notice may describe a future change. A court decision may interpret a provision without replacing the statutory text.

Semantic similarity does not understand which version governs. A question about a 2017 event may retrieve a 2026 provision because the wording is similar and the later page is better indexed. A reranker may also treat a newer document as more useful because recency is correlated with relevance in ordinary search. In historical legal work, that shortcut becomes a source of error.

The same issue appears after an amendment. A model with a stale parametric memory can answer from the prior rule even when the current corpus contains the new text. Adding the current amendment to the context does not guarantee that the model will override its learned pattern. The system must expose the change and make the time constraint explicit.

The site's context-engineering guide and production-pipeline analysis provide adjacent context. The legal-specific difference is that the context needs an enforceable validity interval, not only a larger prompt.

How As-Of Date Filtering Works

As-of filtering turns temporal validity into a data operation. Each legal provision or document version needs at least a jurisdiction, authority, provision identifier, valid-from date, and valid-to date or open-ended status. The query needs a fact date or an explicit target date. Retrieval should reject versions whose validity interval does not contain that date.

A simple validity rule can be expressed as: retrieve a version when valid_from <= fact_date and either fact_date < valid_to or valid_to is empty. The exact boundary depends on the source's legal convention and time zone. It should be tested with amendment dates, transition provisions, retroactive effect, delayed commencement, and partial provisions.

Date extraction is not a trivial prompt. A question can contain the date of an incident, the date of a filing, the date a contract was signed, the date of a court decision, and a date range describing a statute. The system needs to distinguish those roles. If it cannot, it should ask for clarification or flag the ambiguity.

Filtering should happen before final ranking and answer generation. A post-filter that removes a wrong version after the model has already seen it can still leave the wrong rule in the model's context. The retrieval trace should record the candidate versions, the filter decision, the reason for exclusion, and the passages finally supplied to the generator.

RAG-kNN and RAG-ToC in the Paper

The paper evaluates two retrieval strategies that impose temporal validity. RAG-kNN represents a nearest-neighbor retrieval approach where the relevant legal versions are filtered by the extracted fact date before the model receives the context. RAG-ToC uses a table-of-contents style structure to navigate the statutory material while preserving temporal constraints.

These names should not be turned into marketing labels. The paper's result is that both approaches substantially improve performance across its tested question types relative to the failure modes it measures. It does not show that one architecture is always best, that either architecture solves legal reasoning, or that a production team can copy the diagram without validating its own corpus.

The practical idea is more important than the label. Retrieval should preserve hierarchy, definitions, cross-references, amendment history, and validity dates. The generator should receive enough context to resolve the provision without being flooded by versions that are legally irrelevant to the fact date.

Table-of-contents navigation may help when a legal question requires multiple provisions. Nearest-neighbor retrieval may be efficient for a narrower passage search. Both can fail if the source data has missing dates, inconsistent provision identifiers, broken cross-references, or an authority hierarchy that the index does not represent.

How to Build a Versioned Legal Corpus

The corpus is the difficult part. A vector database does not create legal history. The ingestion layer needs authoritative source acquisition, stable document identifiers, amendment events, effective dates, repeal dates, jurisdiction labels, language, and a method for representing consolidated versions without losing the underlying change history.

The cited paper used buzer.de as raw data because the authors needed historical consolidated German statutes with in-force dates. The paper describes that source as a privately operated resource offering unofficial consolidated versions. That is acceptable as a research input with a disclosed limitation. A production legal service may need official legislation portals, licensed databases, court sources, or a source hierarchy approved by qualified legal professionals.

Ingestion should preserve the original text and a normalized representation. Normalization can improve retrieval, but it should not destroy section numbering, defined terms, footnotes, transitional clauses, or the relationship between a provision and the amendment that changed it. Every transformation needs a reproducible record.

Do not silently merge versions into one current document. That makes a historical query impossible to audit. Store version records separately, link them to a provision lineage, and make the effective interval explicit. When a source conflicts with another source, route the conflict to a review queue instead of choosing the newest page by default.

The site's AI memory systems article is relevant to storage design. Legal version history is not ordinary memory. It is an auditable source layer with authority and temporal semantics.

What the 312-Pair Benchmark Can and Cannot Prove

The benchmark is useful because it isolates a failure that ordinary current-answer tests can miss. A model may answer a modern legal question correctly while failing when the same provision is tested against an older date. The three question categories separate post-cutoff changes from historical-version reasoning and multi-provision reasoning.

But the benchmark is narrow by design. It covers German statutory law, six statutes, synthetically generated questions, five models, and a specific evaluation setup. The source corpus is unofficial and the judge is an LLM validated against human ratings. Those choices do not invalidate the study. They define how far the conclusion can travel.

A production team should add its own evaluation set. Include real queries with dates, amendment transitions, jurisdiction conflicts, cross-references, partially effective rules, and questions where the correct action is to abstain. Keep a held-out set that is not used to tune prompts or retrieval thresholds. Otherwise a rising score may show adaptation to the benchmark rather than better legal reliability.

The paper's result should therefore be stated precisely. In its tested setting, time-filtered RAG improved temporal legal QA over vanilla and web-search alternatives. That is enough to justify engineering the control. It is not enough to promise correct answers to clients or replace legal review.

How to Evaluate Retrieval and Citations

End-to-end answer accuracy is not enough. A system can produce a correct-looking answer for the wrong reason, retrieve a supporting passage but cite the wrong version, or quote the right provision while missing a definition or exception. Evaluation needs separate retrieval, temporal, citation, reasoning, and abstention checks.

Retrieval evaluation asks whether the valid version and necessary provisions were included. Temporal evaluation asks whether the system rejected a later or earlier version when it was inapplicable. Citation evaluation asks whether each material claim is supported by the cited text and whether the citation points to the correct version. Generation evaluation asks whether the answer follows the retrieved law without adding unsupported conclusions.

NIST's Generative AI Profile is useful here because it treats confabulated content and confabulated citations as risks, particularly in consequential applications. A legal RAG evaluator should record the risk of a confidently wrong citation, not only whether the final paragraph resembles a reference answer.

Human review is not a cosmetic final step. Experts should inspect failures that involve ambiguous dates, conflicting sources, transition provisions, or high-impact legal consequences. The evaluation record should retain the prompt, parsed dates, retrieved versions, source spans, model output, judge output, human decision, and remediation.

Production Controls for a Legal RAG System

A production architecture needs controls around the model, not only a better prompt. Start with source governance. Record who approved each source, how often it is updated, how amendments are detected, and what happens when a source is unavailable. Keep an immutable ingestion record so an answer can be reconstructed later.

Next, make temporal validity visible in the interface and API. Show the fact date used, the jurisdiction, the selected version, the effective interval, and any unresolved ambiguity. Do not let the model silently choose a date from a long prompt when the choice changes the legal result.

Then add fail-closed behavior. If no valid version is available, if multiple versions overlap unexpectedly, if a citation span is missing, or if the answer depends on an unverified source, return a review state. A legal assistant can still explain what is missing and suggest the next retrieval step without presenting a legal conclusion as settled.

Security also matters. Legal corpora may contain confidential matter, personal data, privileged material, or sensitive commercial facts. The site's enterprise AI security governance guide and agent hijacking analysis provide related controls. Apply access boundaries, tenant isolation, prompt-injection defenses, output logging, retention limits, and review of tool permissions.

Monitoring, Abstention, and Human Review

Legal RAG systems change when the corpus changes, when a model provider updates a model, when a retriever is re-indexed, and when users discover new query patterns. Monitoring should therefore cover data freshness, version coverage, retrieval failure, citation support, abstention rates, latency, cost, and incidents.

Regression tests should include a provision before and after an amendment. The expected output should change when the fact date crosses the legal transition. Tests should also include a historical question where the newest text is intentionally wrong, a multi-provision question, a conflicting source, and a date that is missing or ambiguous.

Abstention is a product feature, not a failure to be hidden. The system should distinguish “no valid source found,” “source conflict,” “date unresolved,” and “answer requires qualified review.” Those states let a legal professional decide what to do next and create a measurable record of system limits.

The site's planning in the LLM era analysis covers a broader principle that applies here. A system should make its intermediate assumptions visible when those assumptions affect the outcome. In legal QA, the fact date and source version are first-class assumptions.

What a Legal RAG System Should Refuse to Answer

A legal assistant should refuse or escalate when it cannot identify the governing date, jurisdiction, source authority, or applicable provision version. It should also escalate when two authoritative sources conflict, when the retrieved text does not support the requested conclusion, or when the question asks for a personalized legal decision rather than research assistance.

Abstention does not mean returning a blank screen. The system can show the unresolved date, the source versions it found, the missing passage, and the next review step. That creates a useful handoff for a qualified professional and avoids turning uncertainty into a polished paragraph that looks settled.

The refusal policy should be tested like any other feature. Include cases where the current law differs from the historical law, where a transition rule controls, where multiple jurisdictions use similar language, and where the source corpus is incomplete. A system that never abstains is not necessarily helpful. It may simply be hiding its failure rate.

The Bottom Line on Time-Aware Legal RAG

The cited paper identifies a real problem. Legal QA can fail when a model applies an outdated provision after an amendment or a newer provision to a historical fact pattern. Its 312-pair German statutory benchmark reports that time-filtered RAG improved performance across the tested question types, while ordinary web search produced unstable gains and recency bias on historical tasks.

The result does not justify the old article's 99.2% production-accuracy claim or its promise to stop legal hallucinations. The study is a research benchmark built from a specific jurisdiction, source corpus, synthetic questions, models, and judge setup. A production system needs stronger source governance, versioned documents, explicit date handling, citation checks, held-out evaluation, abstention, security controls, and qualified review.

The practical design rule is straightforward. Treat temporal validity as a hard retrieval constraint, not as a suggestion in the prompt. Preserve the legal version lineage. Show the source passages and dates. Refuse or escalate when the evidence is incomplete. Then measure the system against the failures that matter in the jurisdiction and workflow where it will be used.

This article discusses software architecture and research evidence. It is not legal advice and does not replace review by a qualified legal professional.