ReadPaper Blog
Why Can't I Open My Drawer? Mitigating Object-Driven Shortcuts in Zero-Shot Compositional Action Recognition
This paper studies Zero-Shot Compositional Action Recognition, where a model must recognize unseen verb-object pairs made from verbs and objects observed during training. It identifies object-driven shortcuts as a central failure mode: models often infer the verb from the labeled object and training co-occurrence patterns rather than from temporal evidence. The authors propose RCORE, a framework combining Co-occurrence Prior Regularization and Temporal Order Regularization for Composition, and report improved unseen compositional generalization on Sth-com and EK100-com.
The Drawer Problem
The paper frames Zero-Shot Compositional Action Recognition as a test of whether video models can recombine known primitives rather than merely memorize action labels. In this setting, the verb vocabulary and object vocabulary are shared across training and testing, but some verb-object compositions are held out and must be recognized at test time. The central problem is that a model may correctly identify an object such as a drawer while predicting the verb that most often co-occurred with that object during training. The authors call this behavior an object-driven shortcut because the model substitutes object identity and dataset priors for the temporal evidence needed to distinguish actions. This matters because verbs such as opening and closing can involve similar objects but opposite temporal order, so genuine compositional generalization requires motion-sensitive verb representations.
Why The Shortcut Happens
The paper argues that object-driven shortcuts arise from two structural pressures in ZS-CAR data and learning. First, compositional supervision is sparse and skewed because datasets cover only a fraction of the possible verb-object space and concentrate labels on frequent pairs. This creates strong co-occurrence priors, so unseen compositions tend to be pulled toward seen, especially frequent, training combinations. Second, objects are easier to learn than verbs because object categories can often be recognized from static visual appearance, whereas verbs require multi-frame temporal reasoning. The authors connect this asymmetry to broader shortcut-learning behavior: models prefer easier predictive cues when those cues correlate with labels. Their controlled experiments are designed to show that object learning can outpace verb learning and that biased co-occurrence can degrade verb recognition on bias-conflict unseen compositions.
RCORE Steps In
To address the diagnosed failure mode, the paper introduces Robust COmpositional REpresentations, or RCORE. RCORE contains Co-occurrence Prior Regularization, which expands supervision over originally absent compositions and treats frequent co-occurrence priors as hard negatives. This component is meant to prevent the model from letting dominant training pairs monopolize the compositional score space. RCORE also includes Temporal Order Regularization for Composition, which encourages sensitivity to temporal order so that verb features are grounded in dynamics rather than static object appearance. The two components target the paper's two proposed root causes: CPR weakens skewed co-occurrence bias, while TORC counters the easier-object, harder-verb learning imbalance. The method is presented as a training-time framework that can operate with encoder-based vision-language backbones in a fixed verb-object label space.
What The Diagnostics Show
The paper does not rely only on accuracy to diagnose the problem; it proposes shortcut-specific measurements called False Seen Prediction and False Co-occurrence Prediction. False Seen Prediction measures how often an unseen input is incorrectly classified as a seen composition, while False Co-occurrence Prediction focuses on collapse into frequent training co-occurrences. These diagnostics make the shortcut measurable by separating general failure from the particular tendency to reuse observed pairings. The authors report that existing ZS-CAR methods remain biased toward seen compositions even when using modern video-pretrained backbones. They evaluate RCORE on Sth-com and on EK100-com, a ZS-CAR dataset repurposed from EPIC-KITCHENS-100 with more severe compositional sparsity. Across these settings, the paper reports that RCORE reduces FSP and FCP while improving generalization to unseen verb-object compositions.
Takeaway
The broader implication of the paper is that zero-shot compositional action recognition cannot be solved by object recognition plus memorized co-occurrence statistics. Because the test-time challenge is an unseen pairing of known primitives, a robust model must represent verbs in a way that transfers across objects and resists dataset frequency bias. The paper positions temporal grounding as essential for this transfer, especially for verb pairs whose distinction depends on sequence direction or motion evidence. Its open-world evaluation protocol on Sth-com considers all possible verb-object pairs rather than requiring test-set ground-truth labels to restrict the search space. By combining shortcut diagnostics with RCORE's regularizers, the study offers both a way to measure the failure and a practical method for mitigating it. The reported results suggest that suppressing co-occurrence priors and strengthening temporal order sensitivity are complementary steps toward more reliable compositional video understanding.
