Embeddings

Embeddings are numerical vector representations of data—commonly text, but also images, code, etc.—that encode semantic meaning in a way machines can process and compare. They transform unstructured data into a format suitable for computation and similarity measurement.

Key Concepts

  • Purpose: Map input (e.g., a sentence) into high-dimensional space where semantically similar items are nearby.
  • Use Cases:
    • Semantic search
    • Document clustering
    • Recommendation systems
    • Few-shot learning
    • Context injection for LLMs (e.g., via RAG)

How It Works

  • Input is tokenized and fed through a neural network (often a transformer).
  • The model outputs a dense vector representing the semantic features of the input.

Example

  • “Dog” → [0.12, 0.55, 0.34, …]
  • “Puppy” → [0.10, 0.53, 0.36, …]
  • These vectors will be close together using similarity measures like cosine similarity or dot product.

Tools / APIs

  • OpenAI Embeddings API (text-embedding-ada-002)
  • Hugging Face models
  • Sentence Transformers (all-MiniLM-L6-v2)
Links to this note

Approximate Nearest Neighbors (ANN)

is a class of algorithms designed to efficiently retrieve vectors that are close to a query vector in

Top-K Retrieval

Top-K retrieval refers to the process of returning the K most relevant or closest results from a dataset in response to a query. In vector search

Vector Databases

Vector databases are specialized data stores designed to handle high-dimensional vector representations—typically generated from embeddings—and

AI Engineer

AI vs AGI LLMs Inference Training Embeddings Vector Databases AI Agents Roadmap RAG Prompt Engineering Benefits of Pre-trained Models Limitations