Cross-Modal Retrieval: Searching Across Text, Images, and Audio

You have a database of product images and a user types “red running shoes with white soles.” Traditional search requires you to have tagged every image with text metadata. Cross-modal retrieval lets you search images directly with natural language — no tagging required.

The shared embedding space

Cross-modal retrieval works because models like CLIP, SigLIP, and their successors can map different modalities (text, images, audio, video) into the same vector space. In this space, semantically related content clusters together regardless of modality.

A photo of a golden retriever and the text “golden retriever playing in a park” produce vectors that are near each other. This proximity enables cross-modal search: embed your query in one modality, search the index in another.

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("clip-ViT-L-14")

# Embed text query
query_embedding = model.encode("red running shoes with white soles")

# Embed images (done once, stored in index)
image_embeddings = model.encode([img1, img2, img3, ...])

# Find most similar images to text query
similarities = cosine_similarity(query_embedding, image_embeddings)
top_results = similarities.argsort()[-10:][::-1]

Search directions

Cross-modal retrieval is bidirectional:

Text → Image: The most common direction. Users describe what they want in words, system returns matching images. Powers visual search in e-commerce, stock photo platforms, and content management systems.

Image → Text: Upload a photo, find related documents, articles, or descriptions. Useful for reverse image search, finding documentation for a product from its photo, or identifying objects.

Image → Image: While technically same-modal, using CLIP embeddings for image-to-image search captures semantic similarity rather than pixel similarity. A cartoon dog and a photo of a real dog will match, which wouldn’t happen with traditional image similarity.

Audio → Text / Text → Audio: Emerging models project audio and text into shared spaces, enabling search for audio clips using text descriptions and vice versa.

Building a cross-modal search system

Step 1: Choose your embedding model

Model	Strengths	Considerations
CLIP (ViT-L/14)	Well-understood, widely supported	Older, surpassed on benchmarks
SigLIP	Better zero-shot performance than CLIP	Google ecosystem
OpenCLIP	Open-source, various sizes available	Community-maintained
Jina CLIP v2	Strong multilingual support	Good for international products
Cohere Embed v3	Multimodal API with good retrieval metrics	Managed service

Step 2: Index your content

Pre-compute embeddings for all your searchable content and store them in a vector database:

import chromadb

client = chromadb.Client()
collection = client.create_collection("products")

for product in products:
    embedding = model.encode(product.image)
    collection.add(
        embeddings=[embedding.tolist()],
        metadatas=[{"name": product.name, "category": product.category}],
        ids=[product.id]
    )

Step 3: Query across modalities

query = "minimalist desk lamp, warm light"
query_embedding = model.encode(query)

results = collection.query(
    query_embeddings=[query_embedding.tolist()],
    n_results=10
)

Step 4: Re-rank for quality

Initial retrieval is fast but approximate. Re-rank the top results using a more expensive model for better precision:

top_candidates = initial_retrieval(query, n=50)
reranked = rerank_model.score(query, top_candidates)
final_results = sorted(reranked, key=lambda x: x.score, reverse=True)[:10]

Challenges and solutions

The semantic gap. Embedding models capture high-level semantics but may miss fine details. “Red shoes” and “maroon shoes” may be close in embedding space but different enough to matter. Solution: combine embedding search with metadata filters.

Modality bias. Some shared embedding spaces are biased toward one modality. CLIP, for instance, was trained primarily for text-to-image matching and may underperform on image-to-text. Solution: evaluate retrieval quality in both directions and consider direction-specific models.

Scale. Vector search at scale (millions+ items) requires approximate nearest neighbor (ANN) algorithms. FAISS, ScaNN, and vector databases like Pinecone, Weaviate, and Qdrant handle this, but you trade exact results for speed. Tune the accuracy-speed tradeoff for your use case.

Freshness. New content needs embedding and indexing before it’s searchable. For real-time applications, your embedding pipeline needs to keep up with ingest rate.

Real-world applications

• E-commerce: “Show me a dress like this one but in blue” — image query with text modification
• Digital asset management: Creative teams searching photo/video libraries with natural language
• Medical imaging: Finding similar cases across text reports and diagnostic images
• Accessibility: Enabling visually impaired users to search image collections through text descriptions
• Content moderation: Finding visually similar content to known violations across large platforms

Cross-modal retrieval transforms how we interact with unstructured data. The ability to search across modalities without manual tagging is one of those capabilities that, once you have it, you wonder how you managed without it.