TL;DR: This bot scrapes and returns organic search results from Google SERPs for a provided list of queries.
Have a look at this example output file to understand what the results will look like.
Video tutorial
Watch me show how the bot works in detail:
How to scrape search results from Google?
- Sign up for Botster
- Open the bot's start page
- Select the region you'd like to scrape the results for (city, country or state)
- Select your target language
- Pick the number of search results to parse (from 10 to 100)
- Select the target device type: desktop, mobile or tablet
- Finally, input search keywords (each keyword must go on a new line)
- Click the "Start this bot" button
You're done!
How does Google scraper work?
The scraper will return the following data:
This bot analyzes the Google results page returned in response to a given keyword query and visualizes the SEO-relevant elements in a convenient text feed containing:
- Website’s position on SERP;
- URL;
- domain;
- title;
- description.
An SEO expert can use this data to run search analytics and identify their platform’s immediate competitors in the organic results.
Input data
Starting the bot will open a page containing several mandatory and optional fields.
The first step is to assign a unique name to the current query and to choose or create a folder that will store the output files. Afterward, you can proceed to the following settings:
- Region – narrow down the results to a certain geographical location, in which case the bot will filter out all irrelevant domains. Leaving the field empty results in a global search.
- Language – only show the results in a certain language. Leaving the field empty results in a global search.
- Top positions – the number of search positions you would like to be displayed (10, 20, 50, Device – filter the websites according to the types of devices they are optimized to run on (PCs, tablets, smartphones).
Input the keywords you would like to analyze in the next field. If there are several, a new line should be used for each.
The last set of options allows to opt-in for notifications via:
- e-mail;
- Slack;
- Telegram;
- JSON POST request.
How to interpret the results
You can use the results in:
- SEO
- Analytics and data analysis
- Competitive analysis
After identifying the resources that tend to receive preferential treatment from Google and land higher on the results page, an SEO expert can analyze the features they share and optimize their platform accordingly. This applies to design, tags, keywords, titles and so on.
