how-to-scrape-google-trends

How to Scrape Google Trends Data With Python

• Why scrape Google Trends data?
• 1. Install libraries
• 2. Send a request
• 3. Save results to CSV
• 4. Create a result comparison

This article provides step-by-step instructions on how to get Google Trends data with Python and SERP Scraper API (a part of Web Scraper API), which requires a paid subscription or a free trial.

Why scrape Google Trends data?

Here are some of the uses for scraped Google Trends data:

• Keyword research: Google Trends is widely used among SEO specialists and content marketers. Since it provides insights into the past and present popularity of search terms, these professionals can tailor their marketing strategies to gain more website traffic.

• Market research: Google Trends data can be used for market research, helping businesses understand consumer interests and preferences over time. For example, e-commerce businesses can use Google Trends search insights for product development.

• Societal research: Google Trends website is a valuable resource for journalists and researchers, offering a glimpse into societal trends and public interest in specific topics.

These are just a few examples. Google Trends data can also help with investment decisions, brand reputation monitoring, and other cases.

1. Install libraries

For this guide, you'll need the following:

• Credentials for SERP Scraper API – you can claim a 7-day free trial by registering on the dashboard;
• Python;
• Requests library to make requests;
• Pandas library to manipulate received data.

Open your terminal and run the following pip command:

pip install requests pandas

Then, import these libraries in a new Python file:

import requests
import pandas as pd

2. Send a request

Let’s begin with building an initial request to the API:

…

For more information about possible parameters, check our documentation.

If everything’s in order, when you run the code, you should see the raw results of the query in the terminal window like this:

3. Save results to CSV

Now that you have the results, adjust the formatting and save in the CSV format – this way, it’ll be easier to analyze the data. All this can be done with the help of the pandas Python library.

The response you get from the API provides you with four categories of information: interest_over_time, breakdown_by_region, related_topics, and related_queries. Let’s split each category into its own separate CSV file.

Begin by converting each into a pandas dataframe:

…

As the data for related_topics is multi-leveled, you'll have to flatten the structure into a single-leveled one. Thus, the function flatten_topic_data was added to do so.

The only thing left is to save the data to a file:

…

You’ve now created a folder structure to hold all of your separate CSV files grouped by keyword:

4. Create a result comparison

Let’s begin with multiple keyword handling. To make the code iterable, split it into reusable functions.

First, extract the code for the request to the API into a function that takes a query as an argument and returns you the response:

…

Next, you need a function that would transform a raw response into pandas dataframes, save said dataframes as CSV files, and return them:

…

Now that the request and dataframe creation is covered, you can create comparisons:

…

This function will accept the dataframes for all the queries you have created, go over them, and merge them for comparison on key metrics.

The last thing to do is to create the core logic of your application. Adding it all together, the final version of the code should look like this:

…

Running the code will create comparison CSV files that have the combined information of the supplied keywords on each of the categories:

• interest_over_time

• breakdown_by_region

• related_topics

• related_queries

Looking to scrape data from other Google sources? Google Sheets for Basic Web Scraping, Google Play Scraper, How To Scrape Google Jobs, Google News Scrpaer, How to Scrape Google Scholar, How to Scrape Google Flights with Python, Scrape Google Search Results