fusion-embedding-2-2b-preview

fusion-embedding-2-2b-preview is the second generation of Eximius Labs' unified multimodal embedding models: text, images, video, and audio in one vector space. Audio is added to a byte-frozen vision-language base through modality-gated deep adapters, so gaining audio never changes a single existing text, image, or video vector. For the first-generation architecture see fusion-embedding-1-2b-preview (that line is final at v0.3).

GitHub | Technical report: arXiv:2607.18666

Model Overview

fusion-embedding-2-2b-preview embeds all four modalities on a Qwen3-VL-Embedding-2B base that is byte-identical to its original release: its text, image, and video behaviour, and its benchmark scores, carry over exactly. Audio is added by training 60.6M parameters (about 2.3% of the stack): a perceiver-resampler that maps frozen Qwen2.5-Omni audio-tower features into the base's input space, plus 28 gated adapters (44.2M) that give the frozen language model in-layer capacity for audio. The adapters run only while audio is being encoded; every other forward pass returns the frozen layers' output untouched, so the invariance is bitwise, not approximate. Adding audio never invalidates an index you have already built.

This v0.3 release ships two revisions of the same model. The default is tuned for audio-text retrieval; a keyword-tuned revision is available for spoken-command use.

Versions in this release

Revision	Use it for	AudioCaps a2t R@10	SpeechCommands
v0.3-preview (default)	general audio-text retrieval, RAG, clustering	0.785	0.894
v0.3-kw-preview	spoken-keyword and command retrieval	0.749	0.929

The default (v0.3-preview) is the recommended model for almost all use. The keyword-tuned revision trades audio-text retrieval quality for higher zero-shot keyword-spotting accuracy; use it only when spoken-command matching is the priority, and use the default otherwise.

Highlights (v0.3-preview)

Feature	Value
Parameters	~2.06B frozen base + 640M frozen audio tower; 60.6M trained
Modalities	text, image, video, audio
AudioCaps 883 (audio-to-text)	R@1 0.333 · R@5 0.646 · R@10 0.785 · mAP@10 0.255
AudioCaps 883 (text-to-audio)	R@1 0.297 · R@5 0.653 · R@10 0.782 · mAP@10 0.445
MAEB(audio-only), 19 tasks	mean 0.454; leads the board on IEMOCAP speaker-gender (0.938)
Non-audio preservation	text, image, video outputs bit-for-bit the base model's
Embedding dimension	2048 (Matryoshka: 64, 128, 256, 512, 1024, 1536, 2048)
Base model	Qwen/Qwen3-VL-Embedding-2B (byte-frozen)
Audio tower	Qwen/Qwen2.5-Omni-7B audio encoder (frozen)

Evaluation

AudioCaps retrieval (883-clip test, min-rank over references)

Model	a2t R@10	t2a R@10
fusion-embedding-2 v0.2	0.743	0.775
v0.3-preview	0.785	0.782
v0.3-kw-preview	0.749	0.771

The v0.3 default improves audio-to-text retrieval by 4.2 points over v0.2.

MAEB(audio-only), full 19-task board — v0.3-preview

Mean 0.4536. Scores are the main metric per task.

Task	Score
IEMOCAPGender	0.938
BeijingOpera	0.924
NMSQAPairClassification	0.794
GTZANAudioReranking	0.717
JamAltArtistA2ARetrieval	0.687
GTZANGenre	0.644
VoxPopuliLanguageID	0.616
MInDS14	0.578
CREMADPairClassification	0.543
MridinghamTonic	0.540
VoxPopuliAccentPairClassification	0.509
CREMA_D	0.277
VoxCelebSA	0.273
BirdCLEF	0.184
CommonLanguageAgeDetection	0.171
SIBFLEURS	0.115
VoxPopuliGenderClustering	0.079
VehicleSoundClustering	0.025
CREMA_DClustering	0.006

IEMOCAP speaker-gender classification is the strongest cell relative to the field. Clustering tasks are the weakest and are a target for the next release.

Non-audio preservation and other modalities

Text, image, and video outputs are bit-for-bit identical to the frozen Qwen3-VL-Embedding-2B base (the adapter gate is closed on those inputs). Their benchmark scores are therefore the base model's own, unchanged by anything added for audio. Audio-to-image retrieval is emergent: the model is trained on audio-text pairs only, with no audio-image supervision.

Notes and limitations

• Emotion recognition regressed relative to v0.2. On RAVDESS zero-shot emotion the score is 0.21 (v0.2: 0.35). The v0.3 training mix improved spoken-content understanding at the cost of vocal-prosody sensitivity. If speech-emotion is central to your use, evaluate before adopting.
• One evaluation cell, CommonLanguageAgeDetection, derives from Common Voice, which is part of the training data. It scores low (0.171) and does not inflate the reported mean, but the potential overlap is noted for completeness.
• These are research previews under CC-BY-NC-4.0.

Usage

Requirements

• fusion_embedding package: pip install git+https://github.com/Eximius-Labs/fusion-embedding
• transformers>=4.46, torch (CUDA), torchvision, pillow, soundfile, librosa
• ~14 GB GPU memory at bf16

via inference.py (this repository)

from inference import FusionEmbedder

fe = FusionEmbedder.from_pretrained(
    "EximiusLabs/fusion-embedding-2-2b-preview",
    revision="v0.3-preview",           # default, retrieval-tuned
    # revision="v0.3-kw-preview",      # keyword-tuned alternative
)

a = fe.embed_audio("dog.wav")            # audio file or (array, sr=...)
t = fe.embed_text("a dog barks")         # uses the base's native chat template
i = fe.embed_image("dog.jpg")            # PIL image or path

print((a @ t).item(), (a @ i).item())    # cosine similarities in the shared space

# Matryoshka: pass dim= for smaller embeddings (64..2048)
t_small = fe.embed_text("a dog barks", dim=256)

The checkpoint contains the gated adapters and the loader refuses to run without them. All inputs use the base model's chat-template format; embedding quality is sensitive to this formatting, so use the templates provided by FusionEmbedder.

Cross-modal ranking tip

When ranking a gallery of one modality against queries of another, per-modality mean-centering of the gallery improves cross-modal recall:

gallery = FusionEmbedder.center(gallery_embeddings)

Changelog

• v0.3-preview (default): retrieval-tuned flagship. AudioCaps a2t R@10 0.785 (+0.042 over v0.2); MAEB(audio-only) mean 0.454; leads IEMOCAP speaker-gender. RAVDESS emotion regressed to 0.21.
• v0.3-kw-preview: keyword-tuned revision. SpeechCommands zero-shot 0.929; AudioCaps a2t R@10 0.749.
• v0.2-preview: AudioCaps 2.0 fine-tune. a2t R@10 0.743.
• v0.1-preview: first modality-gated-adapter release.

License

Weights (all revisions): CC-BY-NC-4.0 (research preview). The code is Apache-2.0. The frozen base and audio tower retain their original licenses.

Citation

Model:

@software{fusion_embedding_2_2026,
  title  = {Fusion Embedding 2: Modality-Gated Deep Adapters for a
            Unified Text, Image, Video, and Audio Embedding Space},
  author = {Tonmoy, Abdul Basit},
  year   = {2026},
  url    = {https://huggingface.co/EximiusLabs/fusion-embedding-2-2b-preview}
}

Technical report:

@article{tonmoy2026fusion,
  title   = {Fusion Embedding: A Unified Embedding Space for Text, Image,
             Video, and Audio},
  author  = {Tonmoy, Abdul Basit and Hoque, Kazi Fardinul and
             Arham, Md. Shahrier Islam and Luthra, Arman},
  journal = {arXiv preprint arXiv:2607.18666},
  year    = {2026}
}