Pushshift API
The pushshift.io Reddit API was designed and created by the /r/datasets mod team to help provide enhanced functionality and search capabilities for searching Reddit comments and submissions. The project lead, /u/stuck_in_the_matrix, is the maintainer of the Reddit comment and submissions archives located at https://files.pushshift.io.
This RESTful API gives full functionality for searching Reddit data and also includes the capability of creating powerful data aggregations. With this API, you can quickly find the data that you are interested in and find fascinating correlations.
There are two main ways of accessing the Reddit comment and submission database. One is by using the API directly via https://api.pushshift.io/ and the other is through accessing the back-end Elasticsearch search engine via https://elastic.pushshift.io/ This document will explain both approaches and give examples on how to effectively use the API. This document will also explore the use of the API parameters to utilize more focused searches.
There are two main endpoints used to search all publicly available comments and submissions on Reddit:
In the next section, we will explore how to perform more effective searches using the comment search endpoint.
To search for comments, use the https://api.pushshift.io/reddit/search/comment/ endpoint. Let's start with a few examples and then go over the various parameters available when using this endpoint. One of the simplest searches is using just the q parameter. The q parameter is used to search for a specific word or phrase. Here is an example:
Search for the most recent comments mentioning the word "science"
https://api.pushshift.io/reddit/search/comment/?q=science
This will search the most recent comments with the term "science" in the body of the comment. This search is not case-sensitive, so it will find any occurence of the term "science" regardless of capitalization. The API defaults to sorting by recently made comments first. After performing this search, 25 results are returned. This is the default size for searches and can be adjusted using the size parameter. This will be discussed in further detail in the parameters section. Data is returned in JSON format and actual search results are included in the "data" key. There is also a "metadata" key that gives additional information about the search including total number of results found, how long the search took to process, etc. If aggregations are requested, all aggregation data is returned under the aggs key.
There are numerous additional parameters that can be used when performing a comment search. Let's go over them and provide examples for each.
| Parameter | Description | Default | Accepted Values |
|---|---|---|---|
| q | Search term. | N/A | String / Quoted String for phrases |
| ids | Get specific comments via their ids | N/A | Comma-delimited base36 ids |
| size | Number of results to return | 25 | Integer <= 500 |
| fields | One return specific fields (comma delimited) | All Fields Returned | string or comma-delimited string |
| sort | Sort results in a specific order | "desc" | "asc", "desc" |
| sort_type | Sort by a specific attribute | "created_utc" | "score", "num_comments", "created_utc" |
| aggs | Return aggregation summary | N/A | ["author", "link_id", "created_utc", "subreddit"] |
| author | Restrict to a specific author | N/A | String |
| subreddit | Restrict to a specific subreddit | N/A | String |
| after | Return results after this date | N/A | Epoch value or Integer + "s,m,h,d" (i.e. 30d for 30 days) |
| before | Return results before this date | N/A | Epoch value or Integer + "s,m,h,d" (i.e. 30d for 30 days) |
| frequency | Used with the aggs parameter when set to created_utc | N/A | "second", "minute", "hour", "day" |
| metadata | display metadata about the query | false | "true", "false" |
You can retrieve comments directly by using the ids parameter. To get a batch of comments by their id, use the following example:
Retrieve three comments using their base 36 id values
https://api.pushshift.io/reddit/comment/search?ids=dlrezc8,dlrawgw,dlrhbkq
There are quite a few parameters to review, so let's start by providing some more complex examples and how to use the parameters above. Let's continue with the previous example above and expand on our "science" keyword search. What if we wanted to search for the term "science" but restrict it to a specific subreddit? By using the subreddit parameter, we can do that:
Search for the most recent comments mentioning the word "science" within the subreddit /r/askscience
https://api.pushshift.io/reddit/search/comment/?q=science&subreddit=askscience
This will return 25 comments containing the term "science" but only from the /r/askscience subreddit. Since we didn't ask for a specific sort method, the most recent comments are returned (the sort parameter defaults to "desc"). What if we wanted the first comment ever to /r/askscience that mentioned the word "science"? We could use the sort and size parameters to handle that.
Search for the most recent comments mentioning the word "science" within the subreddit /r/askscience
https://api.pushshift.io/reddit/search/comment/?q=science&subreddit=askscience&sort=asc&size=1
This is the result:
…From the result returned, we can see that the first comment ever made to /r/askscience mentioning "science" happened on epoch date 1270637661, which translates to Wednesday, April 7, 2010 10:54:21 AM (GMT). Let's quickly go over the metadata pieces. We can see that the execution time for this search was around 30 milliseconds. There were a total of 36 shards searched and all were successful. The search did not time out (timed_out parameter) which is good. This is an attribute you may want to check if you use the API programmatically as some searches that are more complicated may sometimes time out. The total_results value is 134,785. This tells us the total number of comments in /r/askscience that mention the word science. Since we did not use the before or after parameters, this number represents the entirety of the comments made to /r/askscience.
Let's continue by using additional parameters to highlight the power of the search API. The before and after parameters allow you to restrict the time-frame for the search by giving an epoch timestamp for both. However, the API also understands more human-like values for the before and after parameters. You can use a number followed by the characters s,m,h,d (which stand for second, minute, hour and day) to limit the time-frame as well. Let's run through some examples.
If you wanted to do a search for "Rome" in the subreddit /r/askhistorians but limit it only to the past 30 days, you could use the after parameter with the value 30d (30 days).
Search the subreddit /r/askhistorians for comments mentioning Rome within the past 30 days
https://api.pushshift.io/reddit/search/comment/?q=rome&subreddit=askhistorians&after=30d
What if there was a recent news story three days ago, but we wanted to limit the search window between 4 days ago and 2 days ago? We could use both the before and after parameter to do so. In the next example, we will search for comments mentioning Trump that were made between 4 and 2 days ago and sort by ascending.
Search all subreddits for the term "Trump" and return comments made between 2 and 4 days ago
https://api.pushshift.io/reddit/search/comment/?q=trump&after=4d&before=2d&sort=asc
Let's say you wanted to do a search for the last 150 comments, but you only need the author and body fields returned for each comment. Using the fields parameter, you can tell the API which pieces of information you want to filter. This is primarily to help reduce bandwidth if you are making a lot of requests and only need specific fields returned.
Here is an example using the fields parameter to search for the past 150 comments that mention "government" and only returning the author and body fields:
Search all subreddits for the term "government" and return comments with only the body and author keys
https://api.pushshift.io/reddit/search/comment/?q=government&size=150&fields=body,author
Using one of the examples above that searched for the first occurrence of the word "science" in the subreddit /r/askscience, we saw that the author of the comment was "MockDeath." What if we wanted to get the first 100 comments that "MockDeath" made to Reddit? We can use the author parameter, along with the sort and size parameters.
Search all subreddits and get the first 100 comments ever made by the user /u/MockDeath
https://api.pushshift.io/reddit/search/comment/?author=MockDeath&sort=asc&size=100
Aggregations is a powerful method to give summary data for a search. Using the aggs parameter, we can quickly create facets around specific parameters and see how data changes over time. The aggs parameter for comment searches accepts the following values: author, subreddit, reated_utc and link_id. We can do a lot of very cool things using this parameter, so let's dive into some examples.
Let's say we wanted to see the frequency of usage for the term "Trump" over time. We'd like to be able to see how many comments were posted per hour over the past 7 days for this term. Using aggregations and the aggs parameter, we can get that data quickly. Here's an example using this criteria:
Create a time aggregation using the term trump to show the number of comments mentioning trump each hour over the past 7 days
We used the frequency parameter along with the aggs parameter to create hourly buckets to show the total number of comments mentioning Trump over the past 7 days. The size parameter was set to 0 because we are only interested in getting aggregation data and not comment data. The aggregation data is returned in the response under the key aggs -> created_utc. Here is a snippet of the first part of the return:
{
"aggs": {
"created_utc": [
{
"doc_count": 685,
"key": 1502406000
},
{
"doc_count": 1238,
"key": 1502409600
},
{
"doc_count": 1100,
"key": 1502413200
},The doc_count value is the total number of comments containing the term "trump." The key value is the epoch time for that particular bucket. In this example, the first bucket has an epoch time of 1502406000 which corresponds to Thursday, August 10, 2017 11:00:00 PM. This key value is the beginning time of the bucket, so in this example, 685 comments contain the term "trump" between the time Thursday, August 10, 2017 11:00:00 PM and Thursday, August 10, 2017 12:00:00 PM. The frequency parameter allows you to create buckets per second, minute, hour, day, week, month, year. Using this aggregation, you could use the data to create a chart (i.e. Highcharts) and graph the activity of comments for specific terms, authors, subreddits, etc. This is an extremely powerful data analysis tool.
What if you wanted to not only get the frequency of specific comment terms over time, but also wanted to see which subreddits were the most po
No open issues yet, or sync has not completed.