ReadPaper Blog
Towards Mechanistically Understanding Why Memorized Knowledge Fails to Generalize in Large Language Model Finetuning
This paper studies why large language models fine-tuned on new facts can recall those facts yet fail to use them in downstream reasoning. It formalizes the failure as the Knowing-Using Gap, introduces self-patching to trace where injected knowledge becomes causally usable inside a model, and argues that many failures arise from knowledge-circuit misalignment rather than from the absence of memorized information.
Mission Briefing: Remembering Is Not Using
The paper addresses a central problem in post-training knowledge updates: fine-tuning can make an LLM memorize unseen factual QA pairs without making that knowledge reliably available for multi-hop reasoning. Dai, Rao, Wang, Wang, Liu, and Xiong frame this as a “remembering but not using” failure, where a model can answer an atomic question such as a relation between two entities but fail when the same fact must support a chained query. The motivation is practical because fine-tuning is still a core method for injecting parametric knowledge, even as retrieval-augmented generation and model editing provide alternatives. The authors argue that the key question is not only whether a fact is stored, but when and why it becomes accessible to existing reasoning circuits. This shifts the analysis from aggregate accuracy to the internal training dynamics that connect memorized representations with computation paths used for reasoning.
The Knowing-Using Gap
The paper formalizes the failure mode as the Knowing-Using Gap, defined by both an accuracy gap and a temporal lag between memorization and generalization. Memorization accuracy can saturate early, while performance on generalization tasks remains lower and may emerge much later or not at all. To measure this gap, the authors construct memorization QA tasks from fact triplets and evaluate generalization with two reasoning formats: chaining tasks, which require resolving a bridge entity across two facts, and intersection tasks, which require identifying a shared entity under specified relations while ignoring confounders. The dataset is adapted from STaRK, using the biomedical STaRK-Prime subset and the academic STaRK-MAG subset to test whether the phenomenon holds across domains. Novelty checks show low zero-shot accuracy before fine-tuning, supporting the claim that the injected facts are not already available in the pretrained models.
Self-Patching Technique
The paper’s main methodological contribution is self-patching, an intervention technique designed to locate where newly memorized knowledge is causally usable inside an LLM. Self-patching extends activation patching by copying the hidden-layer representation at an anchor position from a source run into a target run at a target layer, then measuring the resulting change in probability assigned to the correct answer. By scanning layers and fine-tuning checkpoints, the method produces a time-evolving spatial map of knowledge permeation through the model. This matters because ordinary behavioral evaluation can show that a model fails a reasoning question but cannot distinguish between absent knowledge and inaccessible knowledge. Self-patching directly tests whether relocating an internal representation can unlock the correct answer in cases where natural inference fails.
What the Tests Reveal
The experiments support the paper’s knowledge-circuit misalignment hypothesis: memorized representations can be present inside the network yet not routed into computation-effective layers for multi-hop reasoning. After memorization saturates, self-patching identifies internal locations where injected information is retrievable, but the model’s ordinary forward pass may still fail to integrate that information into chaining or intersection computations. Continued fine-tuning sometimes moves usable representations into mid-layer computation, coinciding with the emergence of generalization. In other cases, generalization does not naturally appear because the relevant representations remain stranded after gradients have largely vanished. The paper strengthens this interpretation with causal interventions showing that manually relocating memorized representations yields immediate gains beyond prompting baselines such as chain-of-thought and beyond generic perturbation controls.
Takeaway: Relocate the Knowledge
The paper’s practical implication is that some generalization failures after knowledge injection may be repairable without relearning the facts from scratch. Using the structure revealed by self-patching, the authors design a fixed non-oracle heuristic that recovers 58-75% of oracle headroom in failed generalization cases. This result moves the work from diagnosis toward intervention, suggesting that routing memorized representations into the right computation path can recover a substantial share of lost reasoning performance. The experiments are reported across domains and architectures, which supports the robustness of the Knowing-Using Gap and the knowledge-circuit misalignment account. The release of a specialized Memorization-to-Generalization dataset further positions the paper as a benchmark and diagnostic framework for studying how fine-tuned LLMs convert stored facts into usable reasoning resources.
