dllmforge

DLLMForge - Deltares LLM Forge Toolkit

A comprehensive toolkit for building and deploying LLM-based applications with RAG capabilities, agentic workflows, and enterprise-grade features.

class dllmforge.SimpleAgent(system_message: str = None, temperature: float = 0.1, model_provider: str = 'azure-openai', llm=None, enable_text_tool_routing: bool = False, max_tool_iterations: int = 3)

Simple agent class for LangGraph workflows.

Initialize a simple LangGraph agent.

Parameters:

• system_message – System message for the agent

• temperature – LLM temperature setting

• model_provider – LLM provider (“azure-openai”, “openai”, “mistral”)

__init__(system_message: str = None, temperature: float = 0.1, model_provider: str = 'azure-openai', llm=None, enable_text_tool_routing: bool = False, max_tool_iterations: int = 3)

Initialize a simple LangGraph agent.

Parameters:

• system_message – System message for the agent

• temperature – LLM temperature setting

• model_provider – LLM provider (“azure-openai”, “openai”, “mistral”)

add_tool(tool_func: Callable) → None

Add a tool to the agent.

Parameters:

tool_func – Function decorated with @tool

add_node(name: str, func: Callable) → None

Add a node to the workflow.

Parameters:

• name – Node name

• func – Node function

add_edge(from_node: str, to_node: str) → None

Add a simple edge between nodes.

Parameters:

• from_node – Source node

• to_node – Target node

add_conditional_edge(from_node: str, condition_func: Callable) → None

Add a conditional edge.

Parameters:

• from_node – Source node

• condition_func – Function that determines routing

create_simple_workflow() → None

Create a simple agent -> tools workflow with optional text-based tool routing.

compile(checkpointer=None) → None

Compile the workflow.

process_query(query: str, stream: bool = True) → None

Process a query with the agent.

Parameters:

• query – User query

• stream – Whether to stream the response

run_interactive() → None

Run the agent in interactive mode.

dllmforge.create_basic_agent(system_message: str = None, temperature: float = 0.1, model_provider: str = 'azure-openai') → SimpleAgent

Create a basic agent with standard setup.

Parameters:

• system_message – System message for the agent

• temperature – LLM temperature

• model_provider – LLM provider (“azure-openai”, “openai”, “mistral”)

Returns:

Configured agent instance

Return type:

SimpleAgent

dllmforge.create_basic_tools() → List[Callable]

Create basic utility tools for testing.

Returns:

List of tool functions

class dllmforge.AzureOpenAIEmbeddingModel(model: str = 'text-embedding-3-large', api_base: str = None, deployment_name_embeddings: str = None, api_key: str = None, api_version: str = None)

Class for embedding queries and document chunks using Azure OpenAI Embeddings.

Initialize the embedding model using provided arguments or environment variables for Azure OpenAI. :param model: Name of the embedding model to use :param api_base: Azure OpenAI API base URL :param deployment_name_embeddings: Azure OpenAI deployment name for embeddings :param api_key: Azure OpenAI API key :param api_version: Azure OpenAI API version

__init__(model: str = 'text-embedding-3-large', api_base: str = None, deployment_name_embeddings: str = None, api_key: str = None, api_version: str = None)

Initialize the embedding model using provided arguments or environment variables for Azure OpenAI. :param model: Name of the embedding model to use :param api_base: Azure OpenAI API base URL :param deployment_name_embeddings: Azure OpenAI deployment name for embeddings :param api_key: Azure OpenAI API key :param api_version: Azure OpenAI API version

static validate_embedding(embedding: List[float]) → bool

Validate that the embedding is not empty.

static encode_filename(filename: str) → str

Encode filename to be safe for Azure Cognitive Search document keys.

embed(query_or_chunks: str | List[Dict[str, Any]]) → List[float] | List[Dict[str, Any]]

Embed a single query string or a list of document chunks. :param query_or_chunks: A string (query) or a list of dictionaries (document chunks)

Each dictionary should have keys: “text”, “file_name”, “page_number”

Returns:

list of floats (embedding vector) For document chunks: list of dictionaries with keys: “chunk_id”, “chunk”, “page_number”,

”file_name”, “text_vector”

Return type:

For a string query

class dllmforge.PDFLoader

Loader for PDF documents using PyPDF2.

load(file_path: Path) → Tuple[List[Tuple[int, str]], str]

Load a PDF document and extract text from its pages. :param file_path: Path to the PDF file

Returns:

Tuple containing (pages_with_text, file_name) where pages_with_text is a list of (page_number, text) pairs

class dllmforge.TextChunker(chunk_size: int = 1000, overlap_size: int = 200)

Class for chunking text into smaller segments with overlap. For detailed information about chunking strategies in RAG applications, including: - Why chunking is important - How to choose chunk size and overlap - Different splitting techniques - Evaluation methods See: https://www.mongodb.com/developer/products/atlas/choosing-chunking-strategy-rag/

Initialize the TextChunker. :param chunk_size: Maximum size of each chunk in characters :param overlap_size: Number of characters to overlap between chunks (recommended: 5-20% of chunk_size)

__init__(chunk_size: int = 1000, overlap_size: int = 200)

Initialize the TextChunker. :param chunk_size: Maximum size of each chunk in characters :param overlap_size: Number of characters to overlap between chunks (recommended: 5-20% of chunk_size)

chunk_text(pages_with_text: List[Tuple[int, str]], file_name: str = None, metadata: dict = None) → List[Dict[str, Any]]

Split text into chunks while preserving sentence boundaries. :param pages_with_text: List of tuples containing (page_number, text) pairs :param file_name: Name of the source file (optional) :param metadata: Metadata information extracted from the document (optional)

Returns:

{

‘text’: str, # The chunk text ‘page_number’: int, # Source page number ‘chunk_index’: int, # Index of the chunk ‘total_chunks’: int, # Total number of chunks from this document ‘file_name’: str # Name of the source file

}

Return type:

List of dictionaries containing chunks with metadata

class dllmforge.IndexManager(search_client_endpoint=None, search_api_key=None, index_name=None, embedding_dim=None)

__init__(search_client_endpoint=None, search_api_key=None, index_name=None, embedding_dim=None)

create_index(api_base=None, deployment_name_embeddings=None, api_key=None)

upload_documents(vectorized_chunks)

class dllmforge.Retriever(embedding_model, index_name=None, search_client_endpoint=None, search_api_key=None)

__init__(embedding_model, index_name=None, search_client_endpoint=None, search_api_key=None)

get_embeddings(text)

invoke(query_text, top_k=5)

class dllmforge.LLMResponder(llm)

__init__(llm)

augment_prompt_with_context(query_text, chunks)

generate(query_text, retrieved_chunks)

class dllmforge.RAGEvaluator(llm_provider: str = 'auto', deltares_llm: DeltaresOllamaLLM | None = None, temperature: float = 0.1, api_key: str | None = None, api_base: str | None = None, api_version: str | None = None, deployment_name: str | None = None, model_name: str | None = None)

RAGAS-inspired evaluator for RAG pipelines.

This evaluator provides four key metrics: - Context Precision: - Context Recall: Measures the ability to retrieve all necessary information - Faithfulness: Measures factual accuracy and absence of hallucinations - Answer Relevancy: Measures how relevant and to-the-point answers are

Initialize the RAG evaluator.

Parameters:

llm_provider – LLM provider to use (“openai”, “anthropic”, “deltares” or “auto”)

__init__(llm_provider: str = 'auto', deltares_llm: DeltaresOllamaLLM | None = None, temperature: float = 0.1, api_key: str | None = None, api_base: str | None = None, api_version: str | None = None, deployment_name: str | None = None, model_name: str | None = None)

Initialize the RAG evaluator.

Parameters:

llm_provider – LLM provider to use (“openai”, “anthropic”, “deltares” or “auto”)

evaluate_context_relevancy(question: str, retrieved_contexts: List[str]) → EvaluationResult

Evaluate the relevancy of retrieved contexts to the question.

This metric measures the signal-to-noise ratio in the retrieved contexts. It identifies which sentences from the context are actually needed to answer the question.

Parameters:

• question – The user’s question

• retrieved_contexts – List of retrieved context chunks

Returns:

EvaluationResult with score and explanation

evaluate_context_precision(question: str, retrieved_contexts: List[str], ground_truth_answer: str | None, top_k: int = 5) → EvaluationResult

Evaluate Context Precision@k following the Ragas implementation.

For each of the top-k retrieved chunks, the LLM judges whether the chunk supports the reference answer. Average Precision (AP) is then computed as:

AP = sum(Precision@i * rel_i) / (# relevant chunks)

Parameters:

• question – The question to evaluate.

• reference_answer – The correct or gold answer.

• retrieved_contexts – Ranked list of retrieved chunks.

• top_k – Number of top chunks to evaluate.

Returns:

EvaluationResult with precision@k score and explanation.

evaluate_context_recall(question: str, retrieved_contexts: List[str], ground_truth_answer: str) → EvaluationResult

Evaluate the recall of retrieved contexts against a ground truth answer. This metric measures the ability of the retriever to retrieve all necessary information needed to answer the question by checking if each statement from the ground truth can be found in the retrieved context.

Parameters:

• question – The user’s question

• retrieved_contexts – List of retrieved context chunks

• ground_truth_answer – The reference answer to compare against

Returns:

EvaluationResult with score and explanation

evaluate_faithfulness(question: str, generated_answer: str, retrieved_contexts: List[str]) → EvaluationResult

Evaluate the faithfulness of the generated answer to the retrieved contexts.

This metric measures the factual accuracy of the generated answer by checking if all statements in the answer are supported by the retrieved contexts.

Parameters:

• question – The user’s question

• generated_answer – The answer generated by the RAG system

• retrieved_contexts – List of retrieved context chunks

Returns:

EvaluationResult with score and explanation

evaluate_answer_relevancy(question: str, generated_answer: str) → EvaluationResult

Evaluate the relevancy of the generated answer to the question. This metric measures how relevant and to-the-point the answer is by generating probable questions that the answer could answer and computing similarity to the actual question. :param question: The user’s question :param generated_answer: The answer generated by the RAG system

Returns:

EvaluationResult with score and explanation

calculate_ragas_score(context_precision: float, context_recall: float, faithfulness: float, answer_relevancy: float) → float

Calculate the RAGAS score as the harmonic mean of all four metrics. :param context_precision: Context precision score :param context_recall: Context recall score :param faithfulness: Faithfulness score :param answer_relevancy: Answer relevancy score

Returns:

RAGAS score (harmonic mean)

evaluate_rag_pipeline(question: str, generated_answer: str, retrieved_contexts: List[str], ground_truth_answer: str | None = None) → RAGEvaluationResult

Evaluate a complete RAG pipeline using all four metrics.

Parameters:

• question – The user’s question

• generated_answer – The answer generated by the RAG system

• retrieved_contexts – List of retrieved context chunks

• ground_truth_answer – Optional ground truth answer for context recall evaluation

Returns:

Complete evaluation results

print_evaluation_summary(result: RAGEvaluationResult)

Print a formatted summary of the evaluation results. :param result: The evaluation results to summarize

save_evaluation_results(result: RAGEvaluationResult, output_file: str)

Save evaluation results to a JSON file. :param result: The evaluation results to save :param output_file: Path to the output JSON file

class dllmforge.AnthropicAPI(api_key=None, model='claude-3-7-sonnet-20250219', deployment_claude37=None, deployment_claude35=None)

Class to interact with Anthropic’s Claude API.

Initialize the Anthropic API client with configuration.

__init__(api_key=None, model='claude-3-7-sonnet-20250219', deployment_claude37=None, deployment_claude35=None)

Initialize the Anthropic API client with configuration.

check_server_status()

Check if the Anthropic API service is accessible.

list_available_models()

List available models from Anthropic.

send_test_message(prompt='Hello, how are you?')

Send a test message to the model and get a response.

chat_completion(messages, temperature=0.7, max_tokens=1000)

Get a chat completion from the model.

class dllmforge.LlamaIndexAPI(model_provider: str = 'azure-openai', temperature: float = 0.0, api_key=None, api_base=None, api_version=None, deployment_name=None, model_name=None)

Class to interact with various LLM providers using LlamaIndex.

Initialize the LlamaIndex API client with specified configuration. :param model_provider: Provider of model to use. Options are:

• “azure-openai”: Use Azure OpenAI

• “openai”: Use OpenAI

• “mistral”: Use Mistral

Parameters:

• temperature (float) – Temperature setting for the model (0.0 to 1.0)

• api_key (str) – API key for the provider

• api_base (str) – API base URL (for Azure)

• api_version (str) – API version (for Azure)

• deployment_name (str) – Deployment name (for Azure)

• model_name (str) – Model name (for OpenAI/Mistral)

__init__(model_provider: str = 'azure-openai', temperature: float = 0.0, api_key=None, api_base=None, api_version=None, deployment_name=None, model_name=None)

Initialize the LlamaIndex API client with specified configuration. :param model_provider: Provider of model to use. Options are:

• “azure-openai”: Use Azure OpenAI

• “openai”: Use OpenAI

• “mistral”: Use Mistral

Parameters:

• temperature (float) – Temperature setting for the model (0.0 to 1.0)

• api_key (str) – API key for the provider

• api_base (str) – API base URL (for Azure)

• api_version (str) – API version (for Azure)

• deployment_name (str) – Deployment name (for Azure)

• model_name (str) – Model name (for OpenAI/Mistral)

check_server_status()

Check if the LLM service is accessible.

send_test_message(prompt='Hello, how are you?')

Send a test message to the model and get a response. :param prompt: The prompt string to send. :type prompt: str

Returns:

Dictionary containing the response and metadata.

Return type:

dict

chat_completion(messages, temperature=None, max_tokens=None)

Get a chat completion from the model. :param messages: List of message dicts or tuples (role, content) :type messages: list :param temperature: Optional temperature override :type temperature: float :param max_tokens: Optional max tokens override :type max_tokens: int

Returns:

Dictionary containing the response and metadata.

Return type:

dict

Modules

IE_agent_config	Configuration module for Information Extraction Agent.
IE_agent_document_processor	Document Processor module for preprocessing documents into text or images for LLM processing.
IE_agent_extractor	Synchronous Information Extractor module for extracting structured information from documents using LLM.
IE_agent_extractor_docling	Synchronous Information Extractor module for extracting structured information from documents using LLM with Docling.
IE_agent_schema_generator	Schema Generator module for automatically generating Pydantic models based on user descriptions and example documents using LLM.
agent_core	Simple agent core for DLLMForge - Clean LangGraph utilities.
anthropic_api
langchain_api	Create LLM object and api calls from langchain, including Azure and non-Azure models.
llamaindex_api	Create LLM object and API calls using llama_index, including Azure and non-Azure models.
openai_api
rag_embedding	This module provides embedding functionality for RAG (Retrieval-Augmented Generation) pipelines.
rag_embedding_open_source	This module provides embedding functionality for RAG (Retrieval-Augmented Generation) pipelines.
rag_evaluation	RAGAS Evaluation Module for DLLMForge
rag_preprocess_documents	This module provides document preprocessing functionality for RAG (Retrieval-Augmented Generation) pipelines.
rag_search_and_response	This module provides "create index/vector-database", "search" and "response" functionality for RAG (Retrieval-Augmented Generation) pipelines.