Multiple Documents QnA

Learn how to query multiple documents correctly

---

From the last Web Scrape QnA example, we are only upserting and querying 1 website. What if we have multiple websites, or multiple documents? Let's take a look and see how we can achieve that.

In this example, we are going to perform QnA on 2 PDFs, which are FORM-10K of APPLE and TESLA.

Upsert

1. Find the example flow called - Conversational Retrieval QA Chain from the marketplace templates.

2. We are going to use PDF File Loader, and upload the respective files:

1. Click the Additional Parameters of PDF File Loader, and specify metadata object. For instance, PDF File with Apple FORM-10K uploaded can have a metadata object {source: apple}, whereas PDF File with Tesla FORM-10K uploaded can have {source: tesla} . This is done to seggregate the documents during retrieval time.

1. After filling in the credentials for Pinecone, click Upsert:

1. On the Pinecone console you will be able to see the new vectors that were added.

Query

1. After verifying data has been upserted to Pinecone, we can now start asking question in the chat!

1. However, the context retrieved used to return the answer is a mix of both APPLE and TESLA documents. As you can see from the Source Documents:

1. We can fix this by specifying a metadata filter from the Pinecone node. For example, if we only want to retrieve context from APPLE FORM-10K, we can look back at the metadata we have specified earlier in the Upsert step, then use the same in the Metadata Filter below:

1. Let's ask the same question again, we should now see all context retrieved are indeed from APPLE FORM-10K:

Each vector databse provider has different format of filtering syntax, recommend to read through the respective vector database documentation

1. However, the problem with this is that metadata filtering is sort of "hard-coded". Ideally, we should let the LLM to decide which document to retrieve based on the question.

Tool Agent

We can solve the "hard-coded" metadata filter problem by using Tool Agent.

By providing tools to agent, we can let the agent to decide which tool is suitable to be used depending on the question.

1. Create a Retriever Tool with following name and description:

Name	Description
search_apple	Use this function to answer user questions about Apple Inc (APPL). It contains a SEC Form 10K filing describing the financials of Apple Inc (APPL) for the 2022 time period.

1. Connect to Pinecone node with metadata filter {source: apple}

1. Repeat the same for Tesla:

Name	Description	Pinecone Metadata Filter
search_tsla	Use this function to answer user questions about Tesla Inc (TSLA). It contains a SEC Form 10K filing describing the financials of Tesla Inc (TSLA) for the 2022 time period.	{source: tesla}

It is important to specify a clear and concise description. This allows LLM to better decide when to use which tool

Your flow should looks like below:

1. Now, we need to create a general instruction to Tool Agent. Click Additional Parameters of the node, and specify the System Message. For example:

You are an expert financial analyst that always answers questions with the most relevant information using the tools at your disposal.
These tools have information regarding companies that the user has expressed interest in.
Here are some guidelines that you must follow:
* For financial questions, you must use the tools to find the answer and then write a response.
* Even if it seems like your tools won't be able to answer the question, you must still use them to find the most relevant information and insights. Not using them will appear as if you are not doing your job.
* You may assume that the users financial questions are related to the documents they've selected.
* For any user message that isn't related to financial analysis, respectfully decline to respond and suggest that the user ask a relevant question.
* If your tools are unable to find an answer, you should say that you haven't found an answer but still relay any useful information the tools found.
* Dont ask clarifying questions, just return answer.

The tools at your disposal have access to the following SEC documents that the user has selected to discuss with you:
- Apple Inc (APPL) FORM 10K 2022
- Tesla Inc (TSLA) FORM 10K 2022

The current date is: 2024-01-28

1. Save the Chatflow, and start asking question!

1. Follow up with Tesla:

1. We are now able to ask questions about any documents that we've previously upserted to vector database without "hard-coding" the metadata filtering by using tools + agent.

Metadata Retriever

With the Tool Agent approach, user has to create multiple retriever tools to retrieve documents from different sources. This could be a problem if there is a large number of document sources with different metadata. Using the example above with only Apple and Tesla, we could potentially expand to other companies such as Disney, Amazon, etc. It would be a tedious task to create one retrever tool for each company.

Metadata Retriever comes into play. The idea is to have LLM extract the metadata from user question, then use it as filter when searching through vector databases.

For example, if a user is asking questions related to Apple, a metadata filter {source: apple} will be automatically applied on vector database search.

In this scenario, we can have a single retriever tool, and place the Metadata Retriever between vector database and retriever tool.

XML Agent

For some LLMs, function callings capabilities are not supported. In this case, we can use XML Agent to prompt the LLM in a more structured format/syntax, with the goal of using the provided tools.

It has the underlying prompt:

You are a helpful assistant. Help the user answer any questions.

You have access to the following tools:

{tools}

In order to use a tool, you can use <tool></tool> and <tool_input></tool_input> tags. You will then get back a response in the form <observation></observation>
For example, if you have a tool called 'search' that could run a google search, in order to search for the weather in SF you would respond:

<tool>search</tool><tool_input>weather in SF</tool_input>
<observation>64 degrees</observation>

When you are done, respond with a final answer between <final_answer></final_answer>. For example:

<final_answer>The weather in SF is 64 degrees</final_answer>

Begin!

Previous Conversation:
{chat_history}

Question: {input}
{agent_scratchpad}

Conclusion

We've covered using Conversational Retrieval QA Chain and its limitation when querying multiple documents. And we were able to overcome the issue by using OpenAI Function Agent/XML Agent + Tools. You can find the templates below:

37KB

ToolAgent Chatflow.json

38KB

XMLAgent Chatflow.json