What Images Cannot Say: Language-Guided Olfactory Representation Learning

Images tell us what a scene looks like, but rarely what it would feel like to be there.

While recent datasets pair visual scenes with electronic-nose measurements, aligning smell signals with images remains challenging because many olfactory cues arise from contextual environmental factors that are not directly visible in pixels.

We introduce SCENT, a multimodal framework that uses language guidance as a semantic bridge between vision and olfaction.

Our approach leverages Vision-Language Models (VLMs) to generate scene descriptors capturing objects, environmental context, and plausible ambient smell cues suggested by the visual scene.

These descriptors provide semantic guidance for learning olfactory representations.

We train a smell encoder that maps electronic-nose signals into a shared embedding space aligned with both visual and textual representations, and introduce a languageguided latent decomposition that separates object-specific odors from contextual environmental contributions.

Experiments on the New York Smells dataset demonstrate that SCENT significantly improves crossmodal retrieval compared to vision-only baselines, achieving state-of-theart performance on smell-to-image and smell-to-text retrieval tasks.

In addition, our framework produces interpretable olfactory representations that enable the disentanglement of complex smell mixtures.

Our results reveal the importance of contextual semantic information for grounding olfactory perception in multimodal learning and pave the way for future research in this area.