arxiv:2611.04418

VidUEU-Agent: A Data-Curation Agent for Multimodal Understanding, Editing, and Unified Tasks

Published on Nov 3

· Submitted by

Tengju_Ru on Nov 4

Upvote

Authors:

Tengju Ru ,

Abstract

VidUEU-Agent automates instruction data synthesis from raw videos via a multi-stage pipeline with autonomous iteration, yielding a 100K-sample dataset that improves multimodal model performance.

Generated by thinkingmachines/Inkling-Small

The advancement of multimodal learning is heavily constrained by the scarcity and high annotation cost of high-quality instruction data. While raw videos offer an abundant and dynamic source of visual knowledge, transforming them into precise instruction data remains highly challenging. To bridge this gap, we present VidUEU-Agent, an automated pipeline that synthesizes instruction data from raw videos for multimodal understanding, editing, and unified tasks. VidUEU-Agent follows a four-stage pipeline: it performs global perception to filter low-quality shots, mines keyframes and multimodal signals, constructs metadata and routes the metadata to synthesize instruction samples. More importantly, our agent framework can autonomously iterate the data distribution based on downstream training results, thereby optimizing data quality, balance, and training effectiveness. Scaling this pipeline, we construct VidUEU-100K, a large-scale dataset of 100K training samples spanning multiple domains and tasks, providing a robust foundation for developing comprehensive multimodal capabilities. Extensive experiments demonstrate that fine-tuning baseline models on VidUEU-100K yields significant and consistent performance gains across diverse representative benchmarks, validating the effectiveness of our proposed agent framework and dataset.

View arXiv page View PDF GitHub 17 Add to collection

Community

Lruca

Paper author Paper submitter 1 day ago

🚀 We are excited to share our latest work: “VidUEU-Agent: A Data-Curation Agent for Multimodal Understanding, Editing, and Unified Tasks”!

The scarcity and high annotation cost of high-quality multimodal instruction data remain major bottlenecks for multimodal model development. Raw videos contain rich visual knowledge and temporal information, but transforming them into accurate, verifiable, and task-oriented instruction data remains challenging.

To address this, we introduce VidUEU-Agent, an agentic data-curation framework that automatically constructs instruction data from raw videos for multimodal understanding, image editing, and unified understanding-and-generation tasks.

🔑 Key Highlights

🎬 Four-Stage Data Construction Pipeline:
VidUEU-Agent performs global perception to analyze video structure and filter low-quality shots, mines informative keyframes and multimodal signals, constructs structured metadata, and routes each metadata entry to suitable task types for instruction synthesis. Agent planning, tool use, and multi-stage quality control support the entire process.

🔍 Evidence-Grounded Data Synthesis:
The framework combines vision-language models with visual tools for quality assessment, semantic analysis, and information verification. During instruction synthesis, the model cites the evidence signals it uses, improving data traceability and factual consistency.

🔄 Autonomous Iteration with Training Feedback:
VidUEU-Agent uses downstream training and evaluation results to identify weaknesses and refine data distributions across capabilities, domains, and task types. Across four iterations, each using 30K samples to fine-tune BAGEL from the same initial checkpoint, performance improves overall on the three feedback benchmarks:

📦 VidUEU-100K Dataset:
Using this pipeline, we construct VidUEU-100K from 8,268 diverse videos, containing nearly 20,000 dense metadata annotations and 100,000 instruction samples: 40K understanding, 30K editing, and 30K unified samples, spanning nine task paradigms and five scenario categories.

Our work demonstrates how raw videos can be transformed into diverse, evidence-grounded instruction data through agent planning, tool augmentation, and training-feedback-driven refinement. We hope VidUEU-Agent and VidUEU-100K provide a scalable and reproducible foundation for developing unified multimodal models.

📄 Paper: https://arxiv.org/abs/2611.04418

💻 Code: https://github.com/LRuca/VidUEU-Agent

🤗 Dataset: https://huggingface.co/datasets/Lruca/VidUEU100k

We welcome feedback and discussions!