This document outlines the key stages of the data indexing and retrieval pipeline within the Generative-Question-Answering (GenQ&A) framework. The pipeline is designed to streamline information processing and retrieval, ensuring efficient and accurate responses to user queries.
1. Data Indexing (Training Phase)
The data indexing phase focuses on preparing raw information for efficient retrieval during the inference stage. This involves multiple steps, from gathering and structuring the data to converting it into a format that is optimized for fast and accurate retrieval.
Data Loading
The first step in the pipeline is data loading, which involves collecting information from a variety of sources. These sources can include websites, documents, and FAQs.
- Websites (URLs): Crawled for textual content like articles, blogs, and service FAQs.
- Documents: Including .pdf, .txt, .docx, .csv, .md, and .xlsx file formats.
- FAQs: Predefined Q&A pairs.
This stage ensures that all relevant raw data is collected and made available for processing, serving as the foundation of the knowledge base.
Crawling & Splitting
The crawling and splitting phase focuses on extracting and preparing data for embeddings by ensuring that the content is clean, organized, and optimized for efficient retrieval. The process begins with a crawler, which extracts textual data from web sources, including JavaScript-rendered content that is often difficult to capture with standard tools. This ensures that even dynamic web pages are fully indexed. After crawling, the extracted text, along with any additional content from uploaded documents, is sent to a splitter to break it into smaller, manageable chunks.
The splitting process plays a crucial role in structuring the data. It divides large blocks of text into smaller chunks, making it easier for the system to process and retrieve relevant information. Different splitting strategies are employed based on the type of data to ensure the best use of the knowledge base.
-
Recursive Character Text Splitting:
This method is applied to general textual data and is designed to break down large text blocks into smaller, manageable chunks. The splitting relies on predefined separators such as paragraphs, new lines, spaces, or punctuation marks to divide the text. The size of each chunk is determined based on a specified maximum character limit, ensuring that the data remains concise while retaining its semantic coherence. This strategy is effective for most textual data, enabling efficient storage and retrieval without losing the core meaning of the content. -
Document-Specific Splitting for FAQ Excel Sheets:
For FAQ documents, such as Excel sheets containing question-answer pairs, a specialized splitting strategy is employed to maintain the connection between questions and their corresponding answers. Unlike general text splitting, which may inadvertently fragment question-answer pairs into multiple chunks, this method ensures that each question-answer pair remains intact within a single chunk and preserved as a cohesive unit. -
Document-Specific Splitting for Complex Financial Reports:
Complex financial reports require an entirely different handling approach due to their unique structure, which often consists of a significant amount of numerical data interspersed with descriptive text. When the system detects that numerical data exceeds textual content (e.g., financial tables or balance sheets), it classifies the document as a financial report. In such cases, tables are first summarized and their summaries vectorized individually, while the table itself is stored in full as metadata. This complete metadata acts as a reference point, capturing the key context of the numerical data. During the retrieval phase, the metadata is integrated into the context window of LLM, ensuring that both the numerical data and its surrounding context are easily accessible and properly aligned for accurate response generation.
Data Embedding
After the data is split into smaller chunks, the next step is to transform these chunks into numerical representations known as embeddings. This transformation is carried out using an embedding model, which converts the textual data into high-dimensional vectors. These vectors capture the semantic meaning of the text, enabling the system to understand the content in a way that goes beyond simple keyword matching. By representing text as embeddings, the system can perform advanced similarity searches, ensuring that user queries retrieve the most contextually relevant information.
Data Storing
The generated embeddings are stored in a vector database. This specialized database allows for efficient storage and retrieval of high-dimensional vectors, enabling fast and accurate search capabilities.
2. Data Retrieval & Generation (Inference Phase)
The second phase of the pipeline is focused on responding to user queries by retrieving relevant information from the indexed data and generating a meaningful answer. This process is designed to be both efficient and accurate, providing users with precise and context-aware responses.
User Query
This query can be a standalone question or part of an ongoing conversation. If chat history exists, the system uses a large language model (LLM) to generate a clear and concise standalone query, ensuring that the context of the conversation is preserved. Once the query is prepared, it is converted into an embedding using the same embedding model employed during the data indexing phase. This embedding represents the semantic meaning of the query, making it suitable for comparison with the stored embeddings in the vector database.
Semantic Caching
Before performing a full retrieval from the database, the system checks the semantic cache. The cache is a temporary storage area that keeps track of previously asked questions and their corresponding responses. If the user’s query matches a question already stored in the cache, the system immediately returns the cached response. This bypasses the need for additional processing, significantly improving response times, reducing latency and LLM costs. If the query is not found in the cache, the system moves to the retrieval phase.
Retrieval
The retrieval phase is a critical step in the GenQ&A pipeline, where the system identifies the most relevant pieces of information in response to a user's query. This process begins by comparing the query embedding with the embeddings stored in the vector database. The comparison relies on a similarity search, a mathematical method that measures how closely the query embedding aligns with each stored embedding.
The similarity calculation is often performed using cosine similarity, a metric that evaluates the cosine of the angle between two vectors in a high-dimensional space. This metric effectively captures the semantic closeness between the user’s query and the text chunks in the database. A higher cosine similarity score indicates a closer match, ensuring that only the most relevant information is retrieved.
To manage the volume of retrieved data, the system employs a top-K selection strategy, where K represents the number of most similar chunks to return. This parameter is configurable based on the application's requirements, balancing precision and performance.
By focusing only on these top-ranked chunks, the system minimizes unnecessary processing and ensures that the downstream generation phase receives high-quality, contextually relevant information.
Output Generation
The gathered chunks, along with the user's original query and chat history, are used to form a prompt. This prompt is fed into an LLM, which generates a response based on the provided information. The response is generated in real-time and delivered to the user. For a smoother user experience, the system supports streaming, allowing users to see the response as it is being generated. This makes the interaction feel more immediate and dynamic.
Key Takeaways
Efficient Data Processing: The pipeline ensures that raw data is transformed into a structured format optimized for retrieval, enabling fast and accurate responses.
Semantic Search: The use of embeddings allows the system to go beyond keyword matching, delivering results based on the meaning of the text.
Caching for Speed and Efficiency: Frequently asked questions are handled instantly through semantic caching, improving response times.
Scalable Storage: The vector database can handle growing datasets, ensuring the system remains robust and responsive.
Streaming Support: Real-time response streaming enhances the user experience by providing immediate feedback.
By following these steps, the GenQ&A pipeline combines state-of-the-art technologies to deliver a seamless and effective question-answering experience.