Recently whilst working on a keyword research project I was trying to find a way to get access to more questions and queries thrown up by Google within the ‘People Also Ask’ boxes.
Fortunately, after a little research and testing, I was able to find a way to do exactly that using a well known SEO tool, Screaming Frog, a free Chrome extension and a pretty straightforward XPATH string to create my own Google SERP scraper.
This allowed me to list of a load of additional keywords that I could potentially use to optimise for featured snippets. So, this is how I scrape ‘People Also Asked’ boxes in organic search results.
Creating a Google SERP Scraper to find “People Also Ask” Questions
- Setup Screaming Frog’s configuration correctly
- Set the correct User-Agent
- Create a template to refer to when scraping ‘People Also Asked’ boxes
- Add in your preferred keyword search queries
- Download the ‘Scraper’ Chrome extension
- Find a SERP where ‘People Also Asked’ boxes appear
- Identify the correct <div> class
- Create an XPATH string to use when scraping
- Add the XPATH string into Screaming Frog
- Import the SERP URLs and export your ‘People Also Asked’ queries
Setup Screaming Frog’s configuration correctly
If you haven’t done it already, you’ll need to download the Screaming Frog SEO Spider. Unfortunately, you will need to have a licence key in order to use the configuration features required for this task.
After you’ve downloaded and purchased a licence key you’ll need to update the crawlers configuration settings. To do this, go to Configuration > Speed, change the Max Threads to 1, ensure that the Limit URI/s box is checked and that the Max URI/s is set to 0.5.
Setting the correct User-Agent
Now that the speed configuration settings have been updated, we need to set the User-Agent to Chrome. To change the User-Agent go to:
Configuration > User-Agent using the top toolbar.
Using the dropdown menu, change the User-Agent from the default to Chrome.
Create an Excel or Google Sheet template to refer to
Before you can start using your Google SERP scraper to get a list of questions and phrases from the ‘People Also Ask’ boxes you’ll need to get a list of search result URLs to paste into Screaming Frog.
I find that the best way to do this is to create a basic template using Excel or Google Sheets that contains the following information:
- Google Address
- Search Queries
- URL Parameters
- The concatenated URL
You’ll need to use the relevant Google search engine based on your location. For example, for me, the most relevant engine would be Google.co.uk or Google.com. You’ll also need to add the search parameter used to the end of the URL as you can see in the screenshot above.
Add in your preferred search queries
When you add the search queries, you can type them in using spaces, however, you will need to complete a Find & Replace to replace any spaces with an ‘+’.
Use the following formula in the Concatenated String column to combine all of the elements together to create a new URL to paste into Screaming Frog:
=CONCATENATE(A2,B2,C2) and hit Enter/Return.
Download the ‘Scraper’ Chrome extension
To get the information we need from the SERPs we need to scrape some information for the ‘People Also Asked’ boxes.
To do that, we need to download a free Chrome extension called ‘Scraper’
Find a SERP where the ‘People Also Asked’ boxes appear
Before we can add the SERP URLs into Screaming Frog, we need to get some more information. Complete a search that produces the People Also Asked boxes. Most popular questions will do this.
Identify the <div> class by using the ‘Inspect’ feature in Chrome
After you’ve completed the search and a suggested question box has appeared, use the ‘Inspect Element’ feature in Chrome to view the code and identify the <div> class.
This is what we will use as part of the XPATH string to gather the data that we need.
Hover over one of the questions in the People Also Asked box and identify the <div> class:
Use the <div> class to create an XPATH string when scraping
Now that you know the class, we can create a basic XPATH string to use within the Scraper extension to pull the additional questions and queries within the SERP box feature.
To enter the XPATH string, right click one of the questions and select the ‘Scraper’ tool and put in the following XPATH string – //div[@class=”INSERT-DIV-CLASS-HERE”]:
You’ll need to check the <div> class before using the same one over and over again as they change frequently. Once you have added the class in, hit Scrape and you will find a list of queries that appear within the People Also Asked box.
Add the XPATH string into Screaming Frog
Now we know that the XPATH string works, we can add it to Screaming Frog and import our SERP URLs.
To add the XPATH string into Screaming Frog go to Configuration > Custom > Extraction:
The default extraction name is ‘Extractor 1’. I’d recommend adding a personalised title to make it easier to identify when you export the report. You can then select the extraction type (XPATH) and paste the string into the input area.
Once you have added the extraction details, click OK.
Add your search result URLs from your template and crawl
Before you start a new crawl using the extraction details we have added, you need to copy and paste the search result URLs from your Excel or Google Sheets template.
You want to copy the concatenated URL. It should look something like this:
https://www.google.com/search?q=what+are+redirects&ion=0&num=10&source=Int&cr=countryGB
To paste multiple URLs into Screaming Frog you can go to:
Mode > List > Upload > Enter Manually OR Paste URLs
