# Embedding Utilities

Embedding utility components provide helper functions and tools for working with embeddings, including text processing and similarity calculations.

## Text Embedder

The Text Embedder component provides a unified interface for generating embeddings from text using various embedding models.

### Usage

Text Embedder features:

* Unified embedding interface
* Multiple model support
* Batch processing
* Text preprocessing
* Caching capabilities

### Inputs

| Name              | Display Name    | Info                                     |
| ----------------- | --------------- | ---------------------------------------- |
| text              | Text            | Input text to embed                      |
| embedding\_model  | Embedding Model | Connected embedding model component      |
| batch\_size       | Batch Size      | Number of texts to process in each batch |
| cache\_embeddings | Cache Results   | Whether to cache embedding results       |

### Outputs

| Name       | Display Name | Info                                    |
| ---------- | ------------ | --------------------------------------- |
| embeddings | Embeddings   | Generated embeddings for the input text |
| vectors    | Vectors      | Raw numerical vectors                   |

## Embedding Similarity

The Embedding Similarity component calculates similarity scores between embeddings using various distance metrics.

### Usage

Similarity calculation features:

* Multiple similarity metrics
* Batch similarity calculation
* Threshold filtering
* Ranking and sorting
* Performance optimization

### Inputs

| Name                  | Display Name         | Info                                     |
| --------------------- | -------------------- | ---------------------------------------- |
| query\_embedding      | Query Embedding      | Reference embedding for comparison       |
| candidate\_embeddings | Candidate Embeddings | Set of embeddings to compare against     |
| similarity\_metric    | Similarity Metric    | Distance metric (cosine, euclidean, dot) |
| threshold             | Threshold            | Minimum similarity score                 |
| top\_k                | Top K                | Number of top results to return          |

### Outputs

| Name               | Display Name      | Info                               |
| ------------------ | ----------------- | ---------------------------------- |
| similarity\_scores | Similarity Scores | Calculated similarity scores       |
| ranked\_results    | Ranked Results    | Results sorted by similarity score |
| filtered\_results  | Filtered Results  | Results above the threshold        |

## Similarity Metrics

### Cosine Similarity

* **Range**: -1 to 1 (1 = identical, 0 = orthogonal, -1 = opposite)
* **Use Case**: Most common for text embeddings
* **Benefits**: Normalized, angle-based comparison

### Euclidean Distance

* **Range**: 0 to ∞ (0 = identical, larger = more different)
* **Use Case**: Spatial relationships
* **Benefits**: Intuitive geometric interpretation

### Dot Product

* **Range**: -∞ to ∞ (higher = more similar)
* **Use Case**: When magnitude matters
* **Benefits**: Fast computation, considers vector magnitude

### Manhattan Distance

* **Range**: 0 to ∞ (0 = identical)
* **Use Case**: Robust to outliers
* **Benefits**: Less sensitive to extreme values

## Advanced Features

### Batch Processing

* **Parallel Computation**: Efficient batch similarity calculation
* **Memory Management**: Optimized memory usage for large datasets
* **Chunking**: Automatic chunking for very large datasets
* **Progress Tracking**: Real-time processing progress

### Caching and Optimization

* **Embedding Cache**: Cache frequently used embeddings
* **Result Cache**: Cache similarity calculations
* **Index Optimization**: Optimized similarity search
* **Memory Pooling**: Efficient memory management

### Text Preprocessing

* **Normalization**: Text cleaning and normalization
* **Tokenization**: Smart text tokenization
* **Language Detection**: Automatic language detection
* **Encoding Handling**: Proper text encoding management

## Use Cases

### Semantic Search

* Find similar documents
* Content recommendation
* Duplicate detection
* Content clustering

### Question Answering

* Find relevant passages
* Context retrieval
* Answer ranking
* Knowledge base search

### Content Analysis

* Document similarity
* Topic modeling
* Content classification
* Sentiment analysis

## Usage Notes

* **Performance**: Optimized for large-scale similarity calculations
* **Flexibility**: Support for various embedding models and metrics
* **Scalability**: Efficient batch processing capabilities
* **Accuracy**: High-precision similarity calculations
* **Integration**: Easy integration with vector databases
* **Monitoring**: Built-in performance monitoring and logging
