Ask or search…

Web Crawl

Web crawl is the automated process of systematically browsing and indexing web pages for information retrieval.

Crawl Multiple Pages

  1. 1.
    Click Additional Parameters.
  2. 2.
    Select Web Crawl or Scrape XML Sitemap in Get Relative Links Method.
  3. 3.
    Input 0 in Get Relative Links Limit to retrieve all links available from the provided URL.
  1. 1.
    Click Manage Links.
  2. 2.
    (Optional) Input desired URL to be crawled.
  3. 3.
    Click Fetch Links to retrieve links based on the inputs of the Get Relative Links Method and Get Relative Links Limit in Additional Parameters.
  4. 4.
    In Crawled Links section, remove unwanted links by clicking Red Trash Bin Icon.
  5. 5.
    Lastly, click Save.


Loads a list of available URLs