FIGMA: Towards Fine-Grained Music Retrieval

Despite being trained on long, richly detailed captions, CLAP-based models effectively use only the first 40–50 tokens. Everything after—the musical substance—is ignored.

The cause is architectural. Standard models compress audio into a single mean-pooled vector and text into a single [CLS] token before computing similarity. This collapses both modalities, discarding temporal audio structure and token-level text distinctions. Long captions behave like bags of words.

We confirmed this empirically: truncating MusicBench captions to their first 50 tokens produces nearly identical retrieval scores as using the full caption. Further, fine-tuning LAION-CLAP on FGMCaps training data—which has detailed music-theoretic annotations—yields only marginal gains, showing the bottleneck is in the objective, not the data.

Architecture. We freeze both encoders—MuQ (audio, pretrained self-supervised on music) and Microsoft Multilingual E5 Large (text)—and train only lightweight projection heads (~22M parameters). Each projector is two Transformer encoder layers + a linear map into a shared 512-d space, applied separately to both global pooled and full-sequence features.

Frame-level loss. For each audio frame, we find the maximally similar caption token, average these per-frame scores, and apply InfoNCE. Paired clips are positives; all others in the batch are negatives. This encourages grounding of tempo in rhythm tokens, key in harmony tokens, and so on.

Multi-view loss. Global and frame-level objectives are weighted by α = 0.6:

\[\mathcal{L}_{\mathrm{Multi\text{-}View}} = \alpha\,\mathcal{L}_{\mathrm{global}} + (1-\alpha)\,\mathcal{L}_{\mathrm{frame}}\]

Training. 15 epochs, batch size 256, Adam at 1×10⁻⁴, temperature τ = 0.07, early stopping on validation Recall@1.

The first large-scale dataset combining music-theoretic annotations—chord, tempo, key, beat—with natural language captions at scale.

Audio is drawn from four public sources (MTG-Jamendo, Music4All, JamendoMaxCaps, MusicBench). Musical attributes are extracted automatically: BeatNet for tempo and beat count, Omnizart for chord progressions, Essentia's KeyExtractor for key and mode. Captions are generated by Qwen3-Next-80B-A3B-Instruct from structured attribute prompts with randomized attribute order to avoid position bias. Stratified splitting ensures no data leakage between train / validation / test.

FIGMA is evaluated against 10+ baselines including all LAION-CLAP variants, MS-CLAP, MuQ-MuLaN, M2D-CLAP, and CLAMP 3. On MusicBench, FIGMA reaches 34.52% T2A R@1—a 21.4% relative gain over the previous best (CLAMP 3). The gains are larger on the harder, out-of-domain FMACaps-Eval benchmark (1,000 pairs from Free Music Archive), where FIGMA achieves a 73.3% relative improvement. On the in-distribution FGMCaps test set, FIGMA reaches 26.15% T2A R@1 while the best baseline manages only 2.22%.

Fine-grained music retrieval requires aligning specific audio moments to specific text tokens—not just global embeddings. FIGMA demonstrates that a frame-level, token-wise contrastive objective atop frozen encoders is sufficient to unlock the musical information already present in rich captions. Paired with FGMCaps—the first dataset systematically combining chord, tempo, beat, and key annotations at scale—FIGMA sets a new state of the art across in-domain and out-of-domain benchmarks while remaining compute-efficient (~22M trainable parameters). We hope both the model and dataset serve as a foundation for future work on fine-grained audio understanding and precision music search.