Sitemap URL Crawler

No credit card required!
1
Create a free account on Botster
2
Choose a bot and provide seed data
3
Start the bot and wait for completion
4
Download or share your results

About this bot

In short: This online tool will extract sitemap URLs, their response codes and response statuses into a spreadsheet.

Click to view the example results file for a better idea of the output.

Sitemap URL extractor video tutorial

Watch me show the bot in action in this video tutorial (the video demonstrates an older bot version but the launching algorithm is the same):

Why use the website XML sitemap checker

This fast and simple bot can be used for multiple business-related purposes, such as:

  • Checking your website’s sitemap for validity
  • Checking the sitemap for broken links
  • Seamless migration of a website to a new server or platform
  • Detecting unwanted redirects and errors
  • Improving your site’s indexation on search engines
  • Improving your site’s SEO performance
  • And many more!

The website sitemap crawler tool can easily grab the following details:

  • Input URL
  • Sitemap URL
  • URL
  • Response code
  • Response status

How to check the sitemap of a website

  • Sign up on Botster
  • Go to the bot's start page
  • Enter the sitemap URL
  • Set the crawl rate
  • Specify the user agent
  • Set the number of URLs to scrape
  • Click the "Start this bot" button

That's it – the XML sitemap check is now in progress!

Software walkthrough

1. Open the Launch bot form.

Click on the "Start bot" button on the right-hand side of this page to open the spider's form:

2. Enter the details.

Give your "Job" a meaningful title, and optionally specify (or create) a project folder:

3. Enter the sitemap URL.

4. Set the crawl rate.

The crawl rate is the maximum number of requests per second (1-10) that the XML sitemap extractor will make.

5. Specify the user agent.

Leave the default user agent (BotsterCrawler/1.0) or enter yours.

6. Specify the number of URLs to export.

7. Set up notifications.

Specify if you would like to receive a notification when the sitemap XML checker completes the crawl:

8. Start the bot!

Click the "Start this bot" button on the right-hand side:

That's it! You will be taken to your "Jobs" section. The SEO sitemap checker online is now working and will notify you once it's done. 

Data output

After the HTML sitemap checker tool completes the job you can download your data as an Excel (XLSX), CSV or JSON file.

To use this bot you need to register or log in first.

Frequently asked questions

Create a free account on Botster and launch the XML sitemap URL extractor. This tool operates online. No desktop installation is needed.

To check sitemap for errors, enter your website’s sitemap URL into a special field and launch the sitemap link checker. Analyze the response statuses of each URL in the resulting file. If the response statuses of all the links are ‘OK’, your sitemap is working correctly. If not, maintenance is needed.

Once the sitemap URL checker is done scraping, open your job and click on the Excel logo. The file will be automatically saved to your computer. Also, you can download sitemap as a CSV or JSON file.


Endpoint: POST https://botster.io/api/v2/bots/sitemap-url-parser

Request body:

input string or array of strings (required)
Link to sitemap

crawl_rate string (required)
Crawl rate
Max requests per second the bot will make. Min = 1, max = 10

user_agent string (optional)
User agent
You can specify the user agent if necessary. We're using BotsterCrawler/1.0 by default.

limit (required)
Limit
Valid values:

Min100
Max100000
Step100

cron string (optional)
Set periodicity with which the process should execute. Use cron format.
Example: 0 8 * * * - run every day at 8AM

new_items_only boolean (optional)
The file will contain only the new items that appeared since the latest data crawl.


Version history

Version 1

Version 2 - current page


Your file is being prepared...