ReadPaper Blog
Ideas Have Genomes: Benchmarking Scientific Lineage Reasoning and Lineage-Grounded Idea Generation
The paper introduces IdeaGene-Bench (IG-Bench), a benchmark for testing whether AI systems can reason about how scientific ideas inherit, modify, recombine, or abandon mechanisms from earlier work. Instead of evaluating only retrieval, fluency, or novelty, it uses Idea Genome objects and GenomeDiff alignments to measure lineage reasoning and lineage-grounded proposal generation, revealing that current LLM-based research systems struggle with compositional scientific inheritance.
Source: Ideas Have Genomes: Benchmarking Scientific Lineage Reasoning and Lineage-Grounded Idea Generation
Paper-Centric Chaos
The paper argues that scientific progress cannot be evaluated adequately through paper-level similarity, citation links, or topical retrieval alone. Its central problem is that two papers may share a domain without sharing an inherited mechanism, while two superficially different papers may preserve the same core design idea. The authors define scientific lineage competence as the ability to abstract papers into heritable idea units, trace persistence and change across papers, explain evolutionary dynamics, verify coherent lineages, and generate proposals that plausibly descend from prior work. Their examples distinguish task proximity from mechanism inheritance, such as YOLOv2 modifying YOLO through anchor boxes, batch normalization, and multi-scale training, while DETR shares the object-detection ecology but follows a different architectural lineage. The motivation is that AI research agents can retrieve relevant literature and write plausible proposals while still missing the parent mechanism, repaired limitation, or actual line of descent.
Idea Genomes
The IdeaGene framework represents a paper or proposal as a set of Idea Genome objects, each defined as a minimal, typed, evidence-grounded, lineage-relevant idea structure. Each object has a role type drawn from niche, mechanism, observation, limitation, delta, or claim, along with content, evidence pointers, and optional constraints. This design makes the representation narrower than ordinary summarization: the extracted units must support later comparison, inheritance tracing, and verification. GenomeDiff then aligns Idea Genome objects between predecessor and successor work, recording inheritance, mutation, loss, external import, and novel insertion. The paper also frames these transitions through six operational evolutionary dynamics: mutation, adaptive radiation, hybridization, speciation, niche competition, and isolation. The framework treats the biological metaphor as an operational evaluation layer rather than a grand theory of science, fixing the granularity needed to audit scientific inheritance.
Bench, Not Hype
IG-Bench instantiates the framework with 1,961 golden lineage traces across 10 scientific domains, 1,085 curated Idea Genome objects, and 920 pairwise GenomeDiff records. The benchmark has two complementary evaluations: IG-Exam for closed-form lineage reasoning and IG-Arena for open-ended lineage-grounded idea generation. IG-Exam contains 42 task types and 1,029 instances covering Idea Genome abstraction, inheritance tracing, evolutionary reasoning, and lineage verification. IG-Arena asks systems to generate proposals under controlled information settings, including Question-only, Library, and Lineage conditions. Its Population-Evolution Score (PES) evaluates whether a proposal can be inserted into a specified lineage population by measuring Heredity, Variation, and Selection. This setup makes the benchmark more specific than general scientific QA, embedding evaluation, or ideation preference tests because it asks whether a model can use explicit lineage evidence rather than merely produce relevant scientific prose.
The Bottleneck
The experiments evaluate 14 LLM-based scientists, including direct LLMs, research-agent frameworks, and CLI harnesses, and the results show a substantial compositional bottleneck. On IG-Exam, the strongest system reaches only 27.3% exact accuracy, indicating that current systems often fail at the joint reasoning required by lineage analysis. The paper reports that models may identify local signals correctly while failing to keep parent choice, driver assignment, object fate, and verification flags mutually consistent. This matters because lineage reasoning is not a collection of independent labels; a correct GenomeDiff depends on how inherited mechanisms, repaired limitations, and claimed deltas fit together. In IG-Arena, structured lineage context does not uniformly improve every system, but instead reshuffles rankings. The result suggests that some systems can exploit genome-centric evidence, while others mainly benefit from additional text without achieving coherent lineage use.
Final Punchline
The paper’s broader implication is that plausible research writing and lineage-coherent scientific generation are different capabilities. A generated proposal can sound novel, cite relevant work, and address a recognizable task while failing to inherit the right Idea Genome objects from the lineage it claims to extend. IG-Arena’s PES formalizes this distinction by requiring heredity from appropriate predecessors, meaningful variation relative to nearby work, and selection value for future research. This reframes scientific ideation evaluation around whether a proposal can be placed as a coherent descendant in a population of prior ideas. The benchmark therefore challenges automated research systems to move beyond topical synthesis toward mechanism-level inheritance, mutation, and recombination. Its contribution is not only a dataset but an evaluative lens for asking whether AI systems understand how scientific ideas develop over time.
