ReadPaper Blog
LaMem-VLA: Dual-Latent Memory in Vision-Language-Action Models for Robotic Manipulation
LaMem-VLA addresses a central weakness of many vision-language-action models for robotics: they often predict actions from the current observation and instruction while neglecting task history. The paper proposes a dual-latent memory framework that stores, retrieves, condenses, and injects historical experience as native latent tokens inside the VLA reasoning sequence, improving long-horizon manipulation performance in simulated benchmarks.
Source: LaMem-VLA: Dual-Latent Memory in Vision-Language-Action Models for Robotic Manipulation
Memory That Stays Close
The paper begins from the observation that mainstream vision-language-action models remain biased toward short-horizon decisions because they often operate under a Markovian assumption. In this setting, a robot maps the current visual observation and language instruction to an action chunk without explicitly reasoning about previous state transitions, completed steps, or the current phase of a multi-step task. This limitation matters for robotic manipulation because many tasks require remembering whether an object was moved, whether a subgoal has already been satisfied, or which action should follow from earlier interactions. LaMem-VLA frames this as a temporal short-horizon bias in existing VLA systems such as those built from pretrained vision-language models and policy-learning modules. The paper’s motivation is therefore not simply to add more context, but to make task history participate directly in the same reasoning process that connects perception, instruction understanding, and action generation.
Two Kinds of Forgotten Clues
The paper distinguishes its approach from two common strategies for giving robotic policies access to history. One strategy expands the observation window by concatenating historical frames or using video-like input sequences, which can expose recent visual changes but increases computation and memory as the temporal window grows. Because this window is fixed, it can still discard evidence that lies outside the selected horizon, even when that evidence remains relevant to the task. Another strategy retrieves past trajectories, observations, or action tokens from an external memory bank and uses them as auxiliary context for a downstream policy. LaMem-VLA argues that this keeps memory outside the model’s native latent token space, so historical evidence is not fluidly interleaved with the internal VLA reasoning that resolves visual context, language intent, and action queries.
LaMem-VLA’s Core Idea
The core proposal of LaMem-VLA is to treat historical robotic experience as context-native latent memory rather than as external side information. The framework reconstructs history into compact latent memory tokens that are compatible with the continuous embedding space where modern VLA models already combine visual observations and language instructions. This design allows retrieved memory to be woven directly into the reasoning sequence before action queries are resolved, so past evidence can influence action formation from within the model’s own representation space. The paper emphasizes two complementary forms of memory: short-term memory, which is visually dominant and preserves recent perceptual evidence such as object locations and state changes, and long-term memory, which is semantically dominant and preserves task progress and action continuity. Both forms are ultimately represented as latent tokens, making their distinction a matter of provenance and function rather than incompatible representation formats.
Four Tiny Helpers
LaMem-VLA implements this idea through four coordinated components: a latent memory curator, seeker, condenser, and weaver. The curator organizes past robotic experience into short-term and long-term memory vaults, separating recent visual evidence from longer-horizon semantic and action-continuity evidence. The seeker builds a context-aware query from the current multimodal cognition state, including visual and instruction tokens, and uses it to retrieve task-relevant evidence from both memory vaults. The condenser compresses the retrieved evidence into fixed-length short-term and long-term latent memory tokens, reducing redundancy while preserving compatibility with the VLA embedding space. The weaver then inserts these memory tokens into the reasoning sequence alongside the current observation, instruction, and action queries, producing memory-grounded action queries that condition a diffusion-based action expert to generate future action chunks.
What It Achieved
The paper evaluates LaMem-VLA on simulated robotic manipulation benchmarks, reporting results on LIBERO and SimplerEnv-Bridge. On LIBERO, LaMem-VLA reaches a 97.6% average success rate, outperforming MemoryVLA by 1.1 points, CogACT by 4.4 points, and π0 by 3.5 points on the first four suites. On SimplerEnv-Bridge, it achieves 73.9% average success, exceeding CogACT by 16.6 points and π0 by 4.7 points. These results support the paper’s claim that memory is more effective when represented, retrieved, and consumed within the model’s latent reasoning space rather than attached only as policy-side context. The authors also note an important limitation: the reported empirical validation is currently conducted in simulated environments, with real-world robot experiments left for a future version.
