# How Do Embeddings Work in LLMs?

Canonical URL: https://www.adaline.ai/blog/what-is-embeddings-models-in-llms
LLM text URL: https://www.adaline.ai/blog/what-is-embeddings-models-in-llms/llms.txt
Published: 2025-04-11T00:00:00.000Z
Modified: 2026-03-26T10:52:21.522Z
Author: Nilesh Barla
Category: Research
Visibility: public
Reading time: 12 min
Topics: Research, Adaline, AI agent observability, agent evals, self-improving agents

## Summary

A Comprehensive Analysis of Vector Representations in Large Language Models

## Article

Embeddings are the foundation of how large language models understand and process text. These numerical vectors transform words and sentences into mathematical representations that machines can work with effectively. Recent advancements in embedding models have dramatically improved how LLMs interpret language nuances and relationships between concepts.

The field has evolved rapidly, with models like Gemini Embedding, Gecko, and OpenAI’s text-embedding-3-large pushing performance boundaries across multiple languages and specialized domains. These innovations enable:

- More accurate semantic searching
- Improved text clustering capabilities
- Enhanced classification performance
- Better cross-lingual understanding

This article explores how embeddings function within large language models (LLMs). We'll examine their fundamental structure and purpose, the architectural approaches used in their development, and how they're generated from larger LLMs through distillation processes.

You'll learn about the critical role of training data in embedding quality, including how synthetic data and filtering techniques enhance performance. We'll also cover evaluation frameworks like the Massive Text Embedding Benchmark (MTEB) that measure embedding effectiveness.

By understanding how embeddings work, you'll gain insights into the mathematical foundations that power modern language AI systems and their practical applications.

# Fundamentals of Embeddings in LLMs

Let’s begin with the fundamentals and how and why LLMs need an embedding model.

## Definition and Purpose of Embeddings

[Embeddings](https://www.ibm.com/think/topics/vector-embedding#:~:text=Vector%20embeddings%20are%20numerical%20representations,ML%29%20models%20can%20process) are [numerical vectors that capture the semantic meaning](/blog/understanding-embeddings-in-llms) of data in LLMs. They serve as the core mechanism for how LLMs represent and manipulate text. These mathematical representations allow machines to process and understand language in a format they can work with efficiently.

Image: https://a-us.storyblok.com/f/1023026/2124x1124/fd8dbb41aa/gecko-versatile-text-embeddings.png

**Source**: [Gecko: Versatile Text Embeddings Distilled from Large Language Models](https://arxiv.org/abs/2403.20327)

The primary purpose of embeddings is to transform discrete text elements into continuous vector spaces. This conversion enables LLMs to perform essential operations like measuring similarity, executing searches, clustering related items, and making comparisons based on meaning rather than exact matches.

## The Vector Representation Concept

When text is converted into vectors, each piece of information becomes a point in a high-dimensional space. Mathematically, if we denote a token by an index _i_, an embedding function _f_ maps it to a d-dimensional vector,

```math
f(i) \in \mathbb{R}^d
```

This process creates a structured space where:

- Similar words cluster together
- Related concepts appear near each other
- Semantic relationships are preserved through geometric distance

## How Embeddings Capture Semantic Meaning

Embeddings capture meaning through the relative positions of vectors. In well-trained models, the distance between word vectors directly correlates with their semantic similarity. For example:

- "King" - "Man" + "Woman" ≈ "Queen"
- Words like "doctor" and "physician" have highly correlated vectors

This geometric organization emerges naturally during model training. As the model learns to predict words in context, it adjusts embedding vectors to reflect meaningful relationships.

## The Dimensionality of Embedding Spaces

Modern embedding spaces typically range from 256 to 4096 dimensions. For example:

- [OpenAI’s](https://platform.openai.com/docs/guides/embeddings#embedding-models) text-embedding-3-small and text-embedding-3-large have dimensions of 1536 and 3072, respectively.
- [Gemini Embedding](https://www.arxiv.org/pdf/2503.07891): 3072 dimensions
- [Gecko](https://arxiv.org/pdf/2403.20327) with 256 dimensions outperforms some models with 768 dimensions

Higher dimensions can capture more nuanced relationships but require more computational resources. Some models now support “[elastic embeddings](https://arxiv.org/pdf/2106.02223)” that can be truncated to smaller sizes when needed.

## Evolution from Word2vec to Contextual Embeddings

The journey from static to contextual embeddings represents significant progress in NLP. Early models like [word2vec](https://arxiv.org/abs/1301.3781) assigned a single vector to each word regardless of context. Modern LLMs generate contextual embeddings where the same word receives different vectors depending on its usage.

Image: https://a-us.storyblok.com/f/1023026/1936x1218/0aec129842/cbow-and-skip-gram.png

_The illustration of how CBOW and Skip-gram work_ | **Source**: [Efficient Estimation of Word Representations in Vector Space](https://arxiv.org/pdf/1301.3781)

Contextual embedding models produce richer representations that capture nuance and polysemy. This advancement enables more sophisticated language understanding and generation capabilities.

# Architecture and Training Objectives

## Common Architectural Approaches for Embedding Models

Most modern text embedding models use a [bi-encoder transformer ](https://arxiv.org/pdf/1810.04805)architecture. This design processes each text input independently through a transformer encoder like BERT or [RoBERTa](https://arxiv.org/pdf/1907.11692). After encoding, a pooling operation (typically mean-pooling or using the [CLS] token) converts variable-length sequences into fixed-size vectors.

Image: https://a-us.storyblok.com/f/1023026/2750x1232/9cab2a9cf9/overview-of-the-working-procedure-of-bert.png

_Overview of the working procedure of BERT _**| Source**: [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://arxiv.org/abs/1810.04805)

Unlike cross-encoders that process text pairs together, bi-encoders enable efficient scaling for tasks requiring comparisons across large datasets. This architecture is particularly valuable for retrieval tasks where pre-computing embeddings significantly reduces computation time.

## Training Objectives

The primary training objective for embedding models is[ ](https://arxiv.org/pdf/2403.20327#:~:text=diverse%20tasks%20like%20question%20answering%2C,a%20single%20unified%20training%20process)[contrastive learning](https://arxiv.org/pdf/2403.20327#:~:text=diverse%20tasks%20like%20question%20answering%2C,a%20single%20unified%20training%20process). This approach works by:

1. Bringing similar text pairs closer in vector space
2. Pushing dissimilar texts farther apart
3. Using a temperature parameter to control the separation strength

Mathematically, the loss function (InfoNCE) looks like:

```math
L = -\log \frac{e^{sim(q, d^+)/\tau}}{\sum_{j=1}^n e^{sim(q, d^-_j)/\tau}}
```

Where q is a query, d+ is a positive match, and d- are negative examples.

## Pre-training Methodologies

Pre-training typically involves exposing models to vast amounts of text data. Effective approaches include:

- Training on billions of text pairs (e.g., English E5 used 1 billion pairs)
- Using translation pairs for multilingual models
- Employing continued language model training with special formatting
- Leveraging naturally occurring text pairs (title-body, anchor-clicked page)

This stage establishes broad semantic understanding before task-specific fine-tuning begins.

## Post-training Optimization Strategies

After primary training, several optimization techniques improve embedding quality:

- **[Matryoshka Representation Learning](https://arxiv.org/abs/2205.13147)**** (MRL)**: Trains models to produce embeddings that remain useful when truncated to smaller dimensions
- **Multimodal extensions**: Additional training to embed images alongside text
- **Calibration**: Normalizing embedding magnitudes for consistent similarity scores
- **Fine-tuning on proprietary data**: Adding domain-specific knowledge

## The Relationship Between Model Size and Embedding Quality

[Performance strongly correlates with model size](https://arxiv.org/pdf/2210.07316v3), with larger models generally producing better embeddings. Key insights include:

- Multi-billion parameter models dominate most embedding tasks
- Smaller, well-tuned models can sometimes match larger ones on specific tasks
- Increasing model size offers diminishing returns against computational costs
- Size-efficient architectures can deliver competitive performance at lower resource requirements

Image: https://a-us.storyblok.com/f/1023026/2672x790/84f7825323/relationship-between-model-size-and-embedding-quality.png

_Table showing models with more dimensions perform well_** | Source**: [Gecko: Versatile Text Embeddings Distilled from Large Language Models](https://arxiv.org/abs/2403.20327)

# From LLMs to Embedding Models

## How LLMs Serve as Embedding Generators

Large language models can generate embeddings in two main ways. First, by[ ](https://arxiv.org/abs/2403.20327)[extracting hidden-layer representations](https://arxiv.org/abs/2403.20327) directly from the LLM. Second, by prompting the LLM to output some form of embedding. Though LLMs contain rich semantic knowledge, running a multi-billion parameter model for every piece of text is impractical and costly.

Instead, the dominant approach uses **derivative models** of LLMs for embedding. OpenAI's text-embedding-ada-002 exemplifies this strategy. Rather than calling GPT-4 directly, this specialized model is trained on similar data but fine-tuned to output vectors instead of text tokens.

## The Process of Distillation from Larger LLMs

[Knowledge distillation](/blog/llm-distillation-explained) transfers capabilities from a powerful teacher model (an LLM) to a smaller student model (the embedding model). This typically involves:

1. Using the LLM to generate diverse query-passage pairs
2. Having the LLM relabel and filter the data
3. Training the smaller model on this curated dataset

The **Gecko** model demonstrates this effectively with a two-step distillation pipeline:

- [Step 1] Generate synthetic training data with the LLM
- [Step 2] Use the LLM to identify positive and negative examples for each query

Image: https://a-us.storyblok.com/f/1023026/2684x1090/889b63d732/two-step-distillation-pipeline-of-gecko.png

_The illustration represents the two-step distillation pipeline of Gecko_ | **Source**: [Gecko: Versatile Text Embeddings Distilled from Large Language Models](https://arxiv.org/abs/2403.20327)

## Fine-tuning Strategies for Embedding-specific Tasks

Fine-tuning embedding models requires specialized approaches:

- **Task-specific prompting**: Including task descriptions like "Represent the meaning of this sentence for retrieval:”
- **Contrastive objectives**: Training the model to differentiate between relevant and irrelevant matches
- **Multi-task learning**: Combining diverse datasets (question answering, paraphrasing, classification)
- **In-batch negatives**: Using other positive examples in a batch as negatives for efficient training

## Architectural Modifications for Embedding Efficiency

Several modifications improve the embedding model's efficiency:

- **Pooling strategies**: Using mean-pooling or specialized pooling layers
- **Sparse embeddings**: Incorporating keyword features alongside dense vectors
- **Context length extensions**: Enhancing models to handle longer texts (up to 8K tokens)
- **Dimensional optimization**: Implementing variable-dimension embeddings through MRL

## Tradeoffs Between Model Size and Embedding Quality

The relationship between model size and quality presents important tradeoffs. Larger models generally produce better embeddings but require more resources. The[ ](https://arxiv.org/pdf/2210.07316v3)[latency-performance graph](https://arxiv.org/pdf/2210.07316v3) reveals three viable approaches:

1. [Maximum speed] Word embedding models like Glove offer the fastest processing
2. [Maximum performance] Large models like GTR-XXL, ST5-XXL, or SGPT-5.8B deliver the highest quality
3. [Balanced approach] Mid-size models like MPNet and MiniLM provide good quality with reasonable speed

# Data Considerations and Synthetic Training

## The Role of Training Data in Embedding Quality

The quality and diversity of training data directly impacts embedding performance. High-quality[ ](https://arxiv.org/pdf/2403.20327#:~:text=The%20recent%20development%20of%20general,has%20primarily%20been%20on%20augmenting)[embedding models](https://arxiv.org/pdf/2403.20327#:~:text=The%20recent%20development%20of%20general,has%20primarily%20been%20on%20augmenting) require extensive exposure to varied language patterns. This includes different domains, tasks, and linguistic structures.

Traditional approaches relied on large, manually labeled datasets. However, creating these datasets is time-consuming, expensive, and often introduces unintended biases. Modern embedding models now utilize a mix of human-labeled and synthetically generated data to achieve optimal results.

## Synthetic Data Generation Using Other LLMs

[LLM-generated synthetic data](https://arxiv.org/pdf/2503.14023) has become a game-changer for training embedding models. This process typically involves:

1. Prompting an LLM to generate diverse queries for a given passage
2. Creating tasks that span different applications (question answering, fact verification)
3. Producing pairs of related text that maintain semantic connections

For example, the [Gecko model uses a two-step approach](https://arxiv.org/abs/2403.20327#:~:text=Gecko%20uses%20a%20two,different%20domains%20and%20linguistic%20patterns):

- First generating queries from passages
- Then using these to create a dataset called FRet containing 6.6M examples

Image: https://a-us.storyblok.com/f/1023026/2388x1130/f11abe9057/fret-methodology.png

_Gecko utilizes FRet methodology to generate task and query pair_** | Source**: [Gecko: Versatile Text Embeddings Distilled from Large Language Models](https://arxiv.org/abs/2403.20327)

## Data Filtering Techniques and Importance

Not all generated data is equally valuable. Several[ ](https://arxiv.org/pdf/2503.07891v1)[filtering techniques](https://arxiv.org/pdf/2503.07891v1) improve dataset quality:

- Removing duplicates to ensure variety
- Ensuring minimum diversity across queries
- Discarding trivial or nonsensical examples
- Using LLMs themselves as quality judges

The Gemini Embedding model demonstrates this by using "Gemini auto-rater" to filter lower-quality examples like unrealistic search queries. This filtering process showed consistent improvements across different languages.

## Hard-Negative Mining Strategies

[Hard negatives](https://arxiv.org/abs/2403.20327#:~:text=In%20the%20second%20step%2C%20a,using%20the%20original%20seed%20passages) are examples that appear relevant but aren't actual matches. They're crucial for training robust models. Effective mining strategies include:

- Using an initial embedding model to find nearest neighbors for queries
- Having an LLM score these candidates based on relevance
- Selecting passages that appear similar but don't answer the query

This challenging training signal forces the model to make finer-grained distinctions between truly relevant and superficially similar content.

## Balancing Diversity and Quality in Training Data

Creating effective training datasets requires careful balance. The most successful approaches combine:

1. [Task diversity] Including classification, retrieval, similarity tasks
2. [Domain variety] Covering academic, conversational, news, and technical content
3. [Language representation] Balancing high and low-resource languages
4. [Data quality] Prioritizing well-formed, accurate examples

Experiments show uniformly sampling across diverse tasks outperforms dataset-proportional sampling, highlighting the importance of exposure to varied linguistic patterns.

# Evaluation and Benchmarking

## The MTEB (Massive Text Embedding Benchmark) Explained

The[ ](https://arxiv.org/pdf/2210.07316v3)[Massive Text Embedding Benchmark](https://arxiv.org/pdf/2210.07316v3) (MTEB) provides a comprehensive framework for evaluating embedding model performance. It spans 8 embedding tasks covering 58 datasets and 112 languages. Before MTEB, models were often evaluated on limited task sets, making it unclear whether models excelling at similarity tasks would perform well in clustering or retrieval.

Image: https://a-us.storyblok.com/f/1023026/2612x1352/5232ac6643/mteb-massive-text-embedding-benchmark.png

**Source**: [MTEB: Massive Text Embedding Benchmark](https://arxiv.org/abs/2210.07316)

MTEB tasks include:

1. Bitext mining
2. Classification
3. Clustering
4. Pair classification
5. Reranking
6. Retrieval
7. Semantic Textual Similarity (STS)
8. Summarization

This diverse evaluation framework helps identify truly versatile embedding models versus those specialized for narrow applications.

## Key Metrics for Embedding Quality Assessment

Different tasks require[ ](https://arxiv.org/pdf/2210.07316v3)[specific evaluation metrics](https://arxiv.org/pdf/2210.07316v3) to properly assess performance:

```csv
Task Type	Primary Metrics	What They Measure
Classification	Accuracy, F1	Label prediction quality
Clustering	V-measure	Cluster quality without label dependence
STS	Spearman correlation	Alignment with human similarity judgments
Retrieval	nDCG@10, MRR	Ranking quality of relevant results
Reranking	MAP	Ability to reorder candidates properly
```

MTEB reports a single mean score summarizing performance across tasks while also providing task-specific metrics for detailed analysis.

## Task-Specific Evaluation Considerations

Each task presents unique evaluation challenges:

1. [STS tasks] Measure how well embedding similarity correlates with human judgments using Spearman correlation.
2. [Retrieval tasks] Evaluate if a query can find relevant documents using metrics like Precision@k and nDCG@10.
3. [Clustering tasks] Assess how well embeddings group similar items using adjusted Rand Index.
4. [Multilingual tasks] Test if embeddings maintain quality across languages and can connect concepts across language barriers.

## Challenges in Fair Comparison Across Models

[Fair comparison](https://arxiv.org/pdf/2210.07316v3) faces several obstacles:

- **Data contamination**: Some models may have seen evaluation data during training
- **Implementation details**: Pre-processing or hyperparameters can significantly impact results
- **Task weighting**: Different benchmarks may emphasize tasks differently
- **Resource requirements**: Comparing models with vastly different computational needs

## Interpreting Benchmark Results Meaningfully

When examining[ ](https://arxiv.org/pdf/2210.07316v3)[benchmark results](https://arxiv.org/pdf/2210.07316v3), consider:

1. No single model dominates all tasks, suggesting the field hasn't converged on a universal solution
2. Task performance varies significantly – a model excellent at STS may be mediocre at retrieval
3. Balance raw performance against practical considerations like speed and memory usage
4. Consider language coverage when selecting models for multilingual applications
5. Look beyond average scores to task-specific performance matching your use case

A thorough evaluation looks beyond headline numbers to understand whether a model aligns with specific application requirements.

# Comparative Analysis of Embedding Models

## Detailed Comparison of Major Embedding Models

The[ ](https://arxiv.org/pdf/2503.07891v1)[embedding model landscape](https://arxiv.org/pdf/2503.07891v1) features several leading contenders with distinct characteristics. Recent benchmarks reveal significant performance variations:

```csv
Model	MTEB Multilingual	MTEB English	MTEB Code	Dimensions	Languages
Gemini Embedding	68.32	73.3	74.66	3,072	100+
Gecko Embedding	62.13	69.53	65.4	768	Many
gte-Qwen2-7B	62.51	70.72	56.41	3,584	Many
multilingual-e5-large	63.23	65.53	57.94	1,024	Multiple
Cohere-embed-v3	61.1	66.01	51.94	1,024	100+
OpenAI embed-3-large	58.92	66.43	58.95	3,072	Multiple
```

Gemini Embedding establishes a new benchmark with substantial leads across all categories, particularly in cross-lingual capabilities.

## Strengths and Weaknesses of Each Approach

Each model demonstrates distinct advantages:

**Gemini Embedding**:

- **Strengths**: Superior performance across tasks, excellent multilingual capability
- **Weaknesses**: Larger dimensional size requires more storage

**Gecko**:

- **Strengths**: Compact yet powerful, efficient at 256 dimensions
- **Weaknesses**: Performs less consistently across languages

**GTE-Qwen2**:

- **Strengths**: Strong bilingual performance (Chinese-English), open source
- **Weaknesses**: Less effective for code tasks

**E5 Models**:

- **Strengths**: Well-documented, established performance
- **Weaknesses**: Lower performance on code, limited by backbone architecture

## Performance Across Languages and Domains

[Multilingual performance](https://arxiv.org/pdf/2210.07316v3) varies significantly between models. LaBSE excels in bitext mining across many languages, while models like SGPT-BLOOM perform well only on languages they've extensively seen during pre-training.

For domain-specific performance:

- **Scientific domains**: SPECTER showed strong performance
- **Code-related tasks**: Gemini Embedding and Gecko lead
- **General English tasks**: Most models perform comparably well
- **Low-resource languages**: Performance drops significantly for most models

## Efficiency Considerations

[Speed-performance tradeoffs](https://arxiv.org/pdf/2210.07316v3) reveal three distinct categories:

1. **Speed optimized**: Word embedding models (Glove, Komninos) process thousands of examples per second
2. **Performance optimized**: Large models (Gemini, GTR-XXL) provide highest quality at higher latency
3. **Balanced**: Mid-sized models (MPNet, MiniLM) offer good performance with reasonable speed

Storage requirements scale with embedding dimensions – Gemini's 3,072 dimensions require significantly more storage than Gecko's 256 dimensions.

## Selection Criteria for Specific Use Cases

When selecting an embedding model, consider:

1. **Task alignment**: Choose models that excel at your primary task (retrieval, clustering, etc.)
2. **Language requirements**: For multilingual applications, prioritize models with strong cross-lingual capabilities
3. **Computational constraints**: Balance quality against available resources
4. **Openness**: Consider whether you need a fully accessible model or can use API-based services
5. **Specialization needs**: Domain-specific applications may benefit from specialized models

The best choice depends heavily on your specific application constraints and performance requirements.

# Conclusion

Embeddings serve as the mathematical foundation that enables large language models to understand and process human language. Through our exploration, we've uncovered several critical insights:

- Embeddings transform text into vector representations, creating a mathematical space where semantic relationships can be measured and manipulated
- Modern embedding models use bi-encoder transformer architectures and contrastive learning objectives to optimize performance
- The distillation process transfers knowledge from larger LLMs to create more efficient, specialized embedding models
- Training data quality significantly impacts embedding performance, with synthetic data generation and filtering techniques providing substantial improvements
- Evaluation frameworks like MTEB provide comprehensive assessment across diverse tasks and languages

The embedding model landscape continues to evolve rapidly, with Gemini Embedding currently setting new performance standards across multilingual, English, and code domains. When selecting an embedding model, consider your specific task requirements, language needs, computational constraints, and whether you need an open-source solution or can use API-based services.

As embedding technology advances, we can expect even more effective representations that capture nuanced semantic relationships while requiring fewer computational resources. Understanding these foundational technologies helps inform better implementation decisions in your AI systems.