Stuck on Shuffle: On Attribute–Object Binding in Vision–Language Models

Netanel Y. Tamir; Michael Green; Issar Tzachor; Or Shimshi; Dvir Samuel; Or Patashnik; Rami Ben-Ari

Stuck on Shuffle: On Attribute–Object
Binding in Vision–Language Models

Netanel Y. Tamir^♠︎, Michael Green^♠︎, Issar Tzachor^♠︎, Or Shimshi^♠︎,
Dvir Samuel^♦︎, Or Patashnik^♥︎, Rami Ben-Ari^♠︎

^♠︎ OriginAI ^♦︎ NVIDIA ^♥︎ Tel Aviv University

arXiv Code SABER Benchmark Finetuning Dataset

SABER query with a golden cannon, tie-dye submarine, and glowing giraffe. Only candidate 3 preserves the bindings. Below, text-only triplets train LoRA adapters in a shared multimodal embedding space. — **Top:** All four candidates contain a cannon, a submarine, and a giraffe, but only one gives each object the right attribute. SABER tests whether models can tell the difference. **Bottom:** Training on text examples improves this ability, even when both the query and the candidates are images.

The paper

Abstract

Retrieval is a fundamental task in multimodal systems: given an image or text query, the goal is to find the most relevant images or texts in a collection, often using a shared embedding space. Reliable retrieval requires these embeddings to capture not only which concepts are present, but also how they are composed. One key compositional challenge is attribute–object binding: encoding which attributes belong to which objects. Existing benchmarks provide a limited view of this capability, typically covering a subset of modality pairings, simple two-object swaps, or small attribute sets. Consequently, it remains unclear how robustly current models preserve attribute–object bindings across different modality pairings and more complex settings.

We therefore introduce SABER, a controlled benchmark for evaluating attribute–object binding across all four image and text modality pairings, spanning diverse attribute types and scenes with up to ten objects. Evaluating models on SABER reveals substantial binding failures in both CLIP-style dual encoders and recent MLLM-based embedders. These findings motivate us to ask how binding can be improved efficiently. While multimodal supervision may be effective, collecting suitable images is difficult and costly. Text, on the other hand, is well suited to binding supervision: it is cheap to generate, supports a broader range of attributes and enables precisely controlled hard negatives. Recent work also shows that capabilities learned through text-only supervision can transfer to visual modalities in MLLMs.

We thus finetune an MLLM-based embedder exclusively on synthetic attribute-binding text. Despite the absence of image supervision, the gains transfer to cross-modal and image–image retrieval while preserving general multimodal embedding performance. Our model achieves SOTA results on SABER while also improving on negation and instance-retrieval benchmarks, demonstrating broader benefits of our binding-focused supervision.

The benchmark

Meet SABER

Synthetic Attribute Binding Evaluation for Representations

1,800multiple choice questions

7,200candidates in the retrieval gallery

2–10attribute–object bindings per scene

232everyday objects

50visual attributes

4-way multiple choice questions

Identify the binding-preserving match among four candidates. The object and attribute sets are held fixed, but the negative candidates shuffle which attribute belongs to which object.

Large-gallery retrieval

Given a query, retrieve the candidates that preserve all of its attribute–object bindings from a shared gallery of 7,200 candidates. A candidate can contain additional objects and still be a correct match, as long as every query binding is preserved.

SABER multiple-choice examples with two, six, and ten objects. Green labels identify the candidate that preserves each query’s attribute assignments. — **SABER multiple-choice questions.** Each query has four candidates with 2–10 attribute–object pairs: one preserves the bindings, while the others form hard negatives by reassigning attributes. For two-object examples, one negative instead replaces an object or attribute. Although only images are shown, the accompanying captions enable evaluation within and across modalities.

A nested SABER chain, read left to right and top to bottom. Scenes grow from two to ten objects, adding one attribute–object pair at each step while retaining every earlier binding. — **Nested SABER chains.** Nine linked multiple-choice questions form a chain from two to ten objects, read left to right and top to bottom. Each successive query adds one binding while preserving all earlier ones. Half of SABER’s questions are organized into such chains, allowing later correct candidates to serve as additional positives for earlier queries in large-gallery retrieval.

The method

Teaching attribute binding through text

No image supervision during finetuning.

Query

damp leaf and grey stone

Positive · preserve the binding

High above the misty valley, a damp leaf lay half-buried beside a grey stone.

Hard negative · swap the binding

High above the misty valley, a damp stone lay half-buried beside a grey leaf.

01

Construct controlled triplets

200,000 synthetic text triplets pair short queries with contextualized positives and binding-swapped hard negatives.

02

Finetune shared parameters

LoRA adapters and a triplet loss adapt Qwen3-VL-Embedding. Shared parameters let textual supervision affect image representations.

03

Preserve general capabilities

LiNeS rescales layer-wise updates to mitigate forgetting.

Results

Better binding across modality pairings

Text-only finetuning improves accuracy on SABER and all nine existing attribute-binding benchmarks.

Vertical grouped bars comparing CLIP ViT-B/32, Qwen3-VL-Embedding-8B, and Ours-8B. Left: four SABER-MCQ modality pairings. Right: all 12 settings across nine existing attribute-binding benchmarks. Ours-8B outperforms both baselines in every setting; each bar is labeled with its accuracy. — **Attribute binding results.** Text-only finetuning improves accuracy across all four SABER modality pairings and all nine existing benchmarks. SABER scores average 2–10 bindings; other benchmarks use native protocols. SugarCrepe, SugarCrepe++, and BiVLC use Swap-Att. subsets.

Text-only finetuning improves both multiple-choice accuracy and large-gallery retrieval on SABER.

Model	MCQ Accuracy (%)	Independent retrieval mAP (%)	Nested retrieval mAP (%)
CLIP ViT-B/32	43.6	57.8	29.3
Qwen3-VL-Embedding-8B	52.5	71.9	38.3
Ours-8B	84.2	91.1	54.1

From attribute binding to negation

Negation

The benefits of binding supervision extend to negation understanding.

Beyond attribute binding

Instance retrieval

Instance retrieval is the task of finding other images of the same specific object, not just the same category.

Grouped bar chart of all nine Table 6 settings, with scores labeled above each bar. Ours-8B improves over Qwen3-VL-Embedding-8B in every setting. — **Instance retrieval results.** Text-only binding finetuning improves mAP across all nine settings, with gains of 2.1–9.9 points over the pretrained model. Results cover eight image→image settings and one text→image setting.

One possible explanation for these gains is that improved attribute binding helps distinguish visually similar objects within the same fine-grained category, such as black shirts with different prints. To test this, we restrict retrieval to a single fine-grained ILIAS category at a time—for example, searching for a card only among cards, or a mural only among murals. The finetuned model improves across all 38 categories, with a mean gain of 8.5 mAP points.

Card query with top-five retrievals. The pretrained model finds the same card at ranks 1 and 3; Ours-8B finds it at ranks 1, 2, and 5.

Mural query with top-five retrievals. The matching mural moves from rank 3 for Qwen3-VL-Embedding-8B to rank 1 for Ours-8B.

Black T-shirt query with top-five retrievals. The pretrained model finds the same shirt only at rank 5; Ours-8B finds it at ranks 1, 2, 3, and 5.

BibTeX

@misc{stuckonshuffle,
  title = {Stuck on Shuffle: On Attribute--Object Binding
           in Vision--Language Models},
  author = {Tamir, Netanel Y. and Green, Michael and Tzachor, Issar
            and Shimshi, Or and Samuel, Dvir and Patashnik, Or
            and Ben-Ari, Rami},
  url = {https://stuck-on-shuffle.github.io/}
}