VC-Tooler: Learning Compositional and Adaptive Visual Tool Use

Abstract

Agentic multimodal reasoning extends passive image understanding by allowing VLMs to actively acquire and refine visual evidence through visual tool interactions. Effective visual tool use requires three capabilities: grounding tool calls in visual context, composing tools across multiple steps, and adapting reasoning to tool-returned observations. However, existing approaches largely focus on grounding within fixed tool spaces and rigid invocation patterns, leaving composition and adaptation insufficiently addressed. We present VC-Tooler, which learns visual tool use as a compositional and adaptive capability. To this end, we first build a trajectory bank through a hierarchical synthesis pipeline covering three capability levels: single-tool grounding, multi-tool composition, and diverse tool contexts and interfaces. We then train the model in two stages: a supervised cold start that establishes these capabilities, followed by reinforcement learning that encourages accurate, efficient, and context-aware visual tool use. VC-Tooler achieves state-of-the-art performance among open-source models on both general-purpose and agentic benchmarks, including 95.8% on V* and 35.3% on VTC-Bench, and shows promising transfer under richer tool settings at inference time.

Highlights

🧠

Core Idea

We recast visual tool use as reasoning over an evolving visual workspace, not isolated function calls. The model learns to invoke tools compositionally (chaining multiple calls toward a goal) and adaptively under heterogeneous visual contexts.

🎯

Method

A two-stage training framework: Stage I synthesizes hierarchical cold-start supervision (ST · MT · DTC), diversifying tool contexts with 1,000+ reinstantiated tool schemas; Stage II refines the policy via GRPO with a lightweight tool reward that rewards faithful use of tool-returned observations.

🏆

Performance

Open-source SOTA across 8 benchmarks (V*, HRBench-4K/8K, CharXiv-DQ/RQ, MME-RealWorld, VTC-Bench and TIR-Bench), reaching 95.8% on V* and 35.3% on VTC-Bench, surpassing prior open-source agentic baselines.

🔌

Generalization

Thanks to DTC reinstantiation, VC-Tooler transfers to unseen tool schemas never observed during training: it matches or exceeds its native-setting accuracy when the six seen tools are augmented with 11 unseen tools at inference, where supervised-only models often stall or degrade.

Method Overview

VC-Tooler adopts a capability-oriented two-stage training framework. The key design principle is to train visual tool use as reasoning over tool-induced state transitions, rather than as isolated function calling.

VC-Tooler two-stage training framework — **Stage I** provides hierarchical supervision (ST → MT → DTC) for schema-grounded invocation, cross-step composition, and robustness to diverse tool contexts. **Stage II** refines the policy with agentic reinforcement learning that rewards faithful reasoning over tool feedback.

Stage I: Hierarchical Tool-Use Supervision

A plan-then-execute pipeline synthesizes trajectories at three capability levels:

ST · single-tool grounding: schema interpretation & argument prediction.
MT · multi-tool composition: chain calls over tool-returned evidence.
DTC · diverse tool contexts: re-instantiate schemas to break invocation memorization.

Stage II: Agentic RL with a Tool Reward

We refine the policy with GRPO using three rewards: an accuracy reward for final-answer correctness, a format reward for output validity, and a tool reward that shapes tool-grounded behavior.

The tool reward is produced by a lightweight critic that scores each rollout on whether the model incorporates returned observations into subsequent reasoning and avoids redundant or non-progressing calls, providing a shaping signal beyond sparse outcome rewards.

Trajectory Bank

We curate the SFT trajectory bank with a synthesis pipeline with plan-then-execute and diverse tool-context reinstantiating. A Planner proposes an ordered tool sequence for each query, an Executor grounds each step in the visual environment to produce a tool-grounded trajectory, and a Reinstantiation module rewrites tool schemas under heterogeneous contexts, jointly producing supervision for the three capability levels (ST · MT · DTC).

Plan-then-execute trajectory synthesis: Planner → Executor → Reinstantiation produces ST, MT, DTC trajectories — Trajectory synthesis via plan-then-execute and diverse tool-context reinstantiating. **Left**: Planner generates a tool sequence; Executor grounds each call in the visual environment; Reinstantiation rewrites schemas to inject diverse tool contexts. **Right**: the resulting three trajectory types: *Single-Tool Invocation* (M = 1, foundational invocation), *Multi-Tool Composition* (M > 1, compositional planning), and *Diverse Tool Context* (reinstantiated schemas, adaptive generalization).

Composition of curated SFT trajectory bank — (a) Composition of the curated SFT trajectory bank: ST / MT / DTC.

Tool and pattern diversity word cloud — (b) Diversity of tools patterns covered.

Over 1,000 tool interfaces · multi-domain images · plan-then-execute trajectories · post-filtered for verifiability and efficiency.

Representative Unseen Tool Schemas

VC-Tooler generalizes to novel tool schemas never seen during training. Below are examples of reinstantiated tool definitions (click to expand).

add_objects 3-D scene editing

{
  "name": "add_objects",
  "description": "Adds new 3D objects to the scene.",
  "parameters": {
    "type": "object",
    "properties": {
      "image_idx": {
        "type": "number",
        "description": "The index of the image to add objects to (starting from 0)."
      },
      "objects": {
        "type": "array",
        "items": {
          "type": "object",
          "properties": {
            "shape": {
              "type": "string",
              "description": "The geometric shape of the object (e.g., 'pyramid', 'sphere', 'cone')."
            },
            "color": {
              "type": "string",
              "description": "The color of the object (e.g., 'orange', 'red', 'blue')."
            },
            "count": {
              "type": "number",
              "description": "The number of objects of this shape and color to add."
            }
          },
          "required": ["shape", "color", "count"]
        },
        "description": "A list of objects to add, each defined by its shape, color, and count."
      }
    },
    "required": ["image_idx", "objects"]
  }
}

identify_and_highlight_objects visual grounding

{
  "name": "identify_and_highlight_objects",
  "description": "Identifies specific objects in an image based on a description and highlights them with colored bounding boxes.",
  "parameters": {
    "type": "object",
    "properties": {
      "image_idx": {
        "type": "number",
        "description": "The index of the image to process (starting from 0)."
      },
      "object_description": {
        "type": "string",
        "description": "A clear description of the objects to be identified and highlighted."
      },
      "num_objects": {
        "type": "number",
        "description": "The number of objects to identify and highlight."
      }
    },
    "required": ["image_idx", "object_description", "num_objects"]
  }
}

calculate_triangle_side math reasoning

{
  "name": "calculate_triangle_side",
  "description": "Calculates the length of a specific side in a 30-60-90 triangle given the length of another side and the type of side it is.",
  "parameters": {
    "type": "object",
    "properties": {
      "image_idx": {
        "type": "number",
        "description": "The index of the image containing the triangle (starting from 0)."
      },
      "given_side_length": {
        "type": "number",
        "description": "The length of the known side."
      },
      "given_side_type": {
        "type": "string",
        "description": "The type of the known side: 'shorter_leg', 'longer_leg', or 'hypotenuse'."
      },
      "target_side_type": {
        "type": "string",
        "description": "The type of the side to be calculated: 'shorter_leg', 'longer_leg', or 'hypotenuse'."
      }
    },
    "required": ["image_idx", "given_side_length", "given_side_type", "target_side_type"]
  }
}

image_translate_region text translation

{
  "name": "image_translate_region",
  "description": "Find regions containing texts, translate the text within the region, and overlay the translated text back onto the original position.",
  "parameters": {
    "type": "object",
    "properties": {
      "img_idx": {
        "type": "number",
        "description": "The index of the image containing foreign text (starting from 0)."
      },
      "translation_tasks": {
        "type": "array",
        "description": "A list of translation tasks, each pairing a specific region with its translated text.",
        "items": {
          "type": "object",
          "properties": {
            "bbox": {
              "type": "array",
              "description": "The bounding box [x1, y1, x2, y2] that fully covers the text region.",
              "items": {"type": "number"},
              "minItems": 4,
              "maxItems": 4
            },
            "text": {
              "type": "string",
              "description": "The translated text to be placed in this specific bounding box."
            }
          },
          "required": ["bbox", "text"]
        }
      }
    },
    "required": ["img_idx", "translation_tasks"]
  }
}

Main Results

Model	General-Purpose Benchmarks						Agentic Benchmarks
Model	V*	HRBench-4K	HRBench-8K	CharXiv-DQ	CharXiv-RQ	MME-RW	VTC-Bench	TIR-Bench
General-Purpose MLLMs
Qwen3-VL-8B	90.1	82.3	78.0	83.0	46.4	61.9	28.7	18.4
LLaVA-OV	75.4	63.0	59.8	–	–	57.4	–	–
InternVL3-8B	81.2	70.0	69.3	73.6	37.6	–	–	16.9
o4-mini	94.6	–	–	94.3	72.0	–	33.7	37.5
Agentic Reasoning Models
DeepEyes	85.6	75.1	72.6	–	–	–	29.3	17.3
Thyme	82.2	77.0	72.0	66.9	44.5	64.8	28.8	14.8
DeepEyes-V2	81.8	77.9	73.8	78.6	48.9	64.9	30.0	–
AdaReasoner	81.7	73.6	70.1	58.4	38.2	61.3	29.3	–
PyVision-RL	88.7	78.1	74.3	–	–	–	–	19.8
CodaDance	84.8	75.2	72.3	–	44.1	–	–	–
Ours
VC-Tooler-SFT	92.1	82.9	78.6	83.8	51.7	62.9	27.8	20.2
VC-Tooler-RL	95.8	83.9	83.8	84.0	54.1	69.5	35.3	21.0
Δ vs Qwen3-VL-8B	+5.7	+1.6	+5.8	+1.0	+7.7	+7.6	+6.6	+2.6

Best in bold, second-best underlined. VC-Tooler-RL achieves open-source SOTA across all 8 benchmarks.

Per-benchmark accuracy of all methods under four tool configurations: Original, No tools, Seen, and Mixed — Generalization across tool availability. Per-benchmark accuracy of all methods under four tool configurations: *Original* (each method's native tools), *No tools*, *Seen* (the six training tools), and *Mixed* (seen tools augmented with unseen ones). Prior methods rarely benefit from a larger tool pool and can even degrade, whereas VC-Tooler matches or exceeds its original-setting accuracy under the seen and mixed pools.

Tool call distribution: SFT vs RL across V*, HRBench-8K, CharXiv(RQ), and VTC-Bench — Tool call distribution of SFT vs RL across benchmarks. RL training encourages more efficient tool use (fewer redundant calls) and unlocks generalization to *unseen* tools (hatched bars) not present during training.