Multiple Documents QnA
Learn how to query multiple documents correctly
Last updated
Learn how to query multiple documents correctly
Last updated
From the last Web Scrape QnA example, we are only upserting and querying 1 website. What if we have multiple websites, or multiple documents? Let's take a look and see how we can achieve that.
In this example, we are going to perform QnA on 2 PDFs, which are FORM-10K of APPLE and TESLA.
Find the example flow called - Conversational Retrieval QA Chain from the marketplace templates.
We are going to use PDF File Loader, and upload the respective files:
Click the Additional Parameters of PDF File Loader, and specify metadata object. For instance, PDF File with Apple FORM-10K uploaded can have a metadata object {source: apple}, whereas PDF File with Tesla FORM-10K uploaded can have {source: tesla} . This is done to seggregate the documents during retrieval time.
After filling in the credentials for Pinecone, click Upsert:
On the Pinecone console you will be able to see the new vectors that were added.
After verifying data has been upserted to Pinecone, we can now start asking question in the chat!
However, the context retrieved used to return the answer is a mix of both APPLE and TESLA documents. As you can see from the Source Documents:
We can fix this by specifying a metadata filter from the Pinecone node. For example, if we only want to retrieve context from APPLE FORM-10K, we can look back at the metadata we have specified earlier in the Upsert step, then use the same in the Metadata Filter below:
Let's ask the same question again, we should now see all context retrieved are indeed from APPLE FORM-10K:
Each vector databse provider has different format of filtering syntax, recommend to read through the respective vector database documentation
However, the problem with this is that metadata filtering is sort of "hard-coded". Ideally, we should let the LLM to decide which document to retrieve based on the question.
We can solve the "hard-coded" metadata filter problem by using Tool Agent.
By providing tools to agent, we can let the agent to decide which tool is suitable to be used depending on the question.
Create a Retriever Tool with following name and description:
search_apple
Use this function to answer user questions about Apple Inc (APPL). It contains a SEC Form 10K filing describing the financials of Apple Inc (APPL) for the 2022 time period.
Connect to Pinecone node with metadata filter {source: apple}
Repeat the same for Tesla:
search_tsla
Use this function to answer user questions about Tesla Inc (TSLA). It contains a SEC Form 10K filing describing the financials of Tesla Inc (TSLA) for the 2022 time period.
{source: tesla}
It is important to specify a clear and concise description. This allows LLM to better decide when to use which tool
Your flow should looks like below:
Now, we need to create a general instruction to Tool Agent. Click Additional Parameters of the node, and specify the System Message. For example:
Save the Chatflow, and start asking question!
Follow up with Tesla:
We are now able to ask questions about any documents that we've previously upserted to vector database without "hard-coding" the metadata filtering by using tools + agent.
With the Tool Agent approach, user has to create multiple retriever tools to retrieve documents from different sources. This could be a problem if there is a large number of document sources with different metadata. Using the example above with only Apple and Tesla, we could potentially expand to other companies such as Disney, Amazon, etc. It would be a tedious task to create one retrever tool for each company.
Metadata Retriever comes into play. The idea is to have LLM extract the metadata from user question, then use it as filter when searching through vector databases.
For example, if a user is asking questions related to Apple, a metadata filter {source: apple} will be automatically applied on vector database search.
In this scenario, we can have a single retriever tool, and place the Metadata Retriever between vector database and retriever tool.
For some LLMs, function callings capabilities are not supported. In this case, we can use XML Agent to prompt the LLM in a more structured format/syntax, with the goal of using the provided tools.
It has the underlying prompt:
We've covered using Conversational Retrieval QA Chain and its limitation when querying multiple documents. And we were able to overcome the issue by using OpenAI Function Agent/XML Agent + Tools. You can find the templates below:
