dllmforge.rag_embedding_open_source

This module provides embedding functionality for RAG (Retrieval-Augmented Generation) pipelines. It can be used to 1) vectorize document chunks, and 2) vectorize user queries. The module uses Azure OpenAI embeddings model as an example of using hosted embedding APIs. Note you need Azure OpenAI service and a deployed embedding model on Azure to use this module.

Classes

LangchainHFEmbeddingModel([model_name])

Class for embedding queries and document chunks using LangChain's HuggingFaceEmbeddings.

class dllmforge.rag_embedding_open_source.LangchainHFEmbeddingModel(model_name: str = 'sentence-transformers/all-MiniLM-L6-v2')

Class for embedding queries and document chunks using LangChain’s HuggingFaceEmbeddings.

Initialize the HuggingFaceEmbeddings from LangChain.

Parameters:

model_name – Name or path of the Hugging Face model (default: “sentence-transformers/all-MiniLM-L6-v2”).

__init__(model_name: str = 'sentence-transformers/all-MiniLM-L6-v2')

Initialize the HuggingFaceEmbeddings from LangChain.

Parameters:

model_name – Name or path of the Hugging Face model (default: “sentence-transformers/all-MiniLM-L6-v2”).

static validate_embedding(embedding: List[float]) → bool

Validate that the embedding vector is non-empty and numeric.

embed(query_or_chunks: str | List[Dict[str, Any]]) → List[float] | List[Dict[str, Any]]

Embed a single query string or a list of document chunks.

Parameters:

query_or_chunks – A string (query) or list of dicts with keys: “text”, “file_name”, “page_number”.

Returns:

list of floats (vector embedding). - For document chunks: list of dicts with keys:

”chunk_id”, “chunk”, “page_number”, “file_name”, “text_vector”.

Return type:

• For a string query