Web Scrape QnA
Learn how to scrape, upsert, and query a website
Last updated
Learn how to scrape, upsert, and query a website
Last updated
Let's say you have a website (could be a store, an ecommerce site, a blog), and you want to scrap all the relative links of that website and have LLM answer any question on your website. In this tutorial, we are going to go through how to achieve that.
You can find the example flow called - WebPage QnA from the marketplace templates.
We are going to use Cheerio Web Scraper node to scrape links from a given URL and the HtmlToMarkdown Text Splitter to split the scraped content into smaller pieces.
If you do not specify anything, by default only the given URL page will be scraped. If you want to crawl the rest of relative links, click Additional Parameters of Cheerio Web Scraper.
Select Web Crawl or Scrape XML Sitemap in Get Relative Links Method.
Input 0 in Get Relative Links Limit to retrieve all links available from the provided URL.
Input desired URL to be crawled.
Click Fetch Links to retrieve links based on the inputs of the Get Relative Links Method and Get Relative Links Limit in Additional Parameters.
In Crawled Links section, remove unwanted links by clicking Red Trash Bin Icon.
Lastly, click Save.
On the top right corner, you will notice a green button:
A dialog will be shown that allow users to upsert data to Pinecone:
Note: Under the hood, following actions will be executed:
Scraped all HTML data using Cheerio Web Scraper
Convert all scraped data from HTML to Markdown, then split it
Splitted data will be looped over, and converted to vector embeddings using OpenAI Embeddings
Vector embeddings will be upserted to Pinecone
On the Pinecone console you will be able to see the new vectors that were added.
Querying is relatively straight-forward. After you have verified that data is upserted to vector database, you can start asking question in the chat:
In the Additional Parameters of Conversational Retrieval QA Chain, you can specify 2 prompts:
Rephrase Prompt: Used to rephrase the question given the past conversation history
Response Prompt: Using the rephrased question, retrieve the context from vector database, and return a final response
It is recommended to specify a detailed response prompt message. For example, you can specify the name of AI, the language to answer, the response when answer its not found (to prevent hallucination).
You can also turn on the Return Source Documents option to return a list of document chunks where the AI's response is coming from.
Apart from Cheerio Web Scraper, there are other nodes that can perform web scraping as well:
Puppeteer: Puppeteer is a Node.js library that provides a high-level API for controlling headless Chrome or Chromium. You can use Puppeteer to automate web page interactions, including extracting data from dynamic web pages that require JavaScript to render.
Playwright: Playwright is a Node.js library that provides a high-level API for controlling multiple browser engines, including Chromium, Firefox, and WebKit. You can use Playwright to automate web page interactions, including extracting data from dynamic web pages that require JavaScript to render.
The same logic can be applied to any document use cases, not just limited to web scraping!
If you have any suggestion on how to improve the performance, we'd love your contribution!
