ReadPaper Blog
Lychee KnowAct-GUIClaw: Personal GUI Assistant with Self-Evolving Memory and Skills
The paper introduces KnowAct-GUIClaw, a personal assistant framework designed to make OpenClaw-style agents more reliable at graphical user interface automation across mobile and desktop environments. It addresses the fragility of standalone GUI agents by combining host-level task understanding, memory-grounded routing, hybrid GUI and shortcut execution, and post-run reflection into reusable skills. The result matters because the framework reports stronger long-horizon GUI task performance, cross-platform adaptability, and transferable gains from memory and skill reuse across different base models.
Source: (none provided)
GUIs Are The Problem
The paper argues that modern personal assistants cannot rely only on clean APIs because many real user tasks happen inside changing graphical interfaces. GUI automation is framed as difficult because screen state is partially observable, actions must be chosen sequentially, and the environment may shift through pop-ups, app transitions, permission dialogs, and login-protected workflows. The authors position OpenClaw as a strong agent framework for complex automation, but identify two central bottlenecks: weak cross-platform GUI interaction support and the absence of a built-in self-evolution mechanism. A naive integration of a standalone GUI agent is described as fragile for long-horizon tasks, especially when a user request spans multiple disjoint applications and intermediate values must be preserved across them. KnowAct-GUIClaw responds by treating GUI work as a coordinated process between a host agent and a lightweight GUI executor, so visual action is invoked when necessary while broader planning, memory, and tool use remain under host control.
Know First, Then Route
The framework’s “Know Deeply, Act Perfectly” paradigm begins with context gathering before interface manipulation. In the Know and Route stages, the host agent uses accumulated interaction experience, task-relevant knowledge, user profiles, feedback, and persistent memory to decompose long-horizon requests and decide which components should handle each subtask. The paper emphasizes memory-grounded routing for distinguishing single-app tasks from cross-app workflows, with structured input and output contracts so extracted information can be carried between subtasks rather than lost in free-text summaries. This design addresses a concrete failure mode in personal assistants: vague user instructions can cause hallucinated app allocation, while underspecified summaries can omit values needed later in the workflow. By making routing depend on stored candidate applications, auxiliary reference context, and a temporary shared data board, KnowAct-GUIClaw turns task decomposition into an explicit information-transfer problem rather than a loose sequence of clicks.
Act With A Fast Path
The Act stage combines visual GUI control with faster non-visual execution paths. The paper argues that many workflows should not be solved by repeated screen exploration when reliable shortcuts exist, including web search tools, Android deep links, system intents, command-line tools, and reusable action sequences. At the same time, it treats such shortcuts as unsafe unless their launch behavior and target page state are validated against the current interface, because a stale shortcut can put the agent in the wrong app state. KnowAct-GUIClaw therefore gives the GUI executor responsibility for screenshot perception, action normalization, device backends, skill validation, and trajectory recording, while the host decides whether visual interaction is truly required. This hybrid GUI-fast path execution is presented as an efficiency mechanism: the agent can commit to a validated skill prefix as a single decision instead of regenerating every low-level action from scratch.
Reflect, Store, Improve
The Reflect stage is the paper’s mechanism for self-evolution from execution experience. Instead of discarding successful and failed trajectories after a run, KnowAct-GUIClaw summarizes traces, attributes useful evidence, and distills them into retrievable memory or parameterized executable skills. The authors describe this as closing the loop from past runs to future decisions: reflected knowledge feeds into memory-grounded routing before decomposition and into the GUI executor during later action. This matters for recurring personal-assistant tasks because repeated navigation, known shortcuts, and common failure patterns can be reused rather than rediscovered. The paper also connects reflection to user personalization, since stored user profiles and feedback improve both task decomposition and tool calls over time.
The Punchline Is The Result
The experimental section reports that KnowAct-GUIClaw improves GUI manipulation efficiency, accuracy, and cross-platform adaptability across Android, iOS, HarmonyOS, and Windows. On the long-horizon MobileWorld benchmark, the paper states that a GUIClaw variant using open-source Kimi-2.6 models achieves 64.1% success on GUI-Only tasks, outperforming the compared agent frameworks and closed-source agentic models named in the excerpt, including Seed-2.0-Pro and GPT-5.5. The results also support the paper’s claim that memory and executable skills are not tied to a single base model: the framework reports improvements of 8.5% with Kimi-2.6 and 16.2% with Qwen3.5-35B-A3B. Additional experiments are organized around MobileWorld, AndroidDaily, ablation and efficiency analysis, case studies, cross-platform checks, and reproducibility. The implication is that personal GUI assistants can become more capable when operational knowledge is treated as a persistent, transferable system component rather than a transient byproduct of one execution trace.
