Baike.dev
All toolsOnline toolsAI codingTrendingOpen sourceNewsSubmit
Log in
< Back to tools
T

trendingtopics

> 编程语言
Open source

Rails app for tracking trends in server logs - powered by the Cloudera Hadoop Distribution on EC2

359 stars0 likes0 views
WebsiteGitHub

About

Rails app for tracking trends in server logs - powered by the Cloudera Hadoop Distribution on EC2

h1. Tracking Trends with Hadoop & Hive on EC2

This repository contains the full source code for "Trendingtopics.org":http://www.trendingtopics.org, built by "Data Wrangling":http://www,datawrangling.com to demonstrate how Hadoop & EC2 can power a data driven website. The trend statistics and time series data that run the site are updated periodically by launching a temporary EC2 cluster running the "Cloudera Hadoop Distribution":http://www.cloudera.com/hadoop-ec2. Our initial seed data consists of the raw Wikipedia database content dump along with hourly "traffic logs":http://stats.grok.se/ for all articles collected from the Wikipedia squid proxy (curated by Domas Mituzas). We made the first 7 months of this hourly data for all articles available as an Amazon Public Dataset.

The current trend calculations are run with Hadoop Streaming and Hive. The output produced by these Hadoop jobs is loaded into MySQL and indexed to power the live site. The demo data included with the Rails app on Github was generated from a sample of 100 trending articles on June 6th 2009. A much larger snapshot is available on Amazon Public Datasets (snap-753dfc1c). The Rails app and MySQL database are deployed on Amazon EC2 using Paul Dowman’s "EC2onRails":http://ec2onrails.rubyforge.org/.

!http://trendingtopics.s3.amazonaws.com/images/trendingtopics_dashboard.png!

h3. Application Features

  • Ranked list of the most significant trends over the last 30 days along with total pageviews
  • Ranked list of "Rising" articles trending in the last 24 hours
  • Daily time series charts and "sparklines":http://en.wikipedia.org/wiki/Sparkline for over 2.5 Million Wikipedia articles
  • Autocomplete functionality and search results ranked by article trend score

h3. How Hadoop is Used in the Application

  • Cleaning raw log data and joining title strings with Wikipedia page ids
  • Aggregating hourly time series data for daily pageview charts and sparklines
  • Generating Statistics that power search autocomplete and the ranking of search results
  • Running periodic trend estimation jobs / regressions

The rest of this document will walk through the code which powers the site and describe some basic approaches for extracting trends from log data with Hadoop and Hive. The individual steps are run manually for the code walkthrough, but would but likely be automated with some combination of Rake, Cron, Puppet, or Chef in a real production environment.

h3. Prerequisites

  • "Get an Amazon EC2 Account":http://aws.amazon.com/ec2/
  • Complete the "Amazon EC2 Getting Started Guide":http://docs.amazonwebservices.com/AWSEC2/latest/GettingStartedGuide/
  • Download "Cloudera EC2 Hadoop scripts":http://www.cloudera.com/hadoop-ec2
  • Clone the "trendingtopics code on github":http://github.com/datawrangling/trendingtopics/tree/master

h2. Raw Data: Hourly Wikipedia Article Traffic Logs

The "Wikipedia Traffic Statistics":http://developer.amazonwebservices.com/connect/entry.jspa?externalID=2596 Amazon Public Dataset we will process covers a 7 month period from October, 01 2008 to April, 30 2009. This data is regularly logged from the Wikipedia squid proxy by Domas Mituzas and was compiled by Data Wrangling for this demo.

h3. Log file format

Each log file is named with the date and time of collection: pagecounts-20090430-230000.gz. The individual hourly files are around 55 MB when compressed, so 8 months of compressed data takes up about 300 GB of space.

Each line has 4 fields: projectcode, pagename, pageviews, bytes

  $ grep '^en Barack' pagecounts-20090521-100001 
  en Barack 8 1240112
  en Barack%20Obama 1 1167
  en Barack_H._Obama 1 142802
  en Barack_H_Obama 3 428946
  en Barack_H_Obama_Jr. 2 285780
  en Barack_Hussein_Obama,_Junior 2 285606
  en Barack_O%27Bama 1 142796
  en Barack_Obama 701 139248439
  en Barack_Obama%27s_first_100_days 2 143181
  en Barack_Obama,_Jr 2 285755

Many of the raw wiki log page titles are percent-encoded. To match these with the page titles in the Wikipedia database, you can transform them as follows in Python:

  $ python
 >>> import urllib
 >>> escaped_title = '%22Klondike_Kate%22_Rockwell'
 >>> print urllib.unquote_plus(escaped_title)
 "Klondike_Kate"_Rockwell

h3. Handling Wikipedia Redirects

To further complicate the raw data processing, many page titles are actually Wikipedia redirects which do not match to the primary key of the Wikipedia "Pages" table. The non-redirect articles are referred to as belonging to "namespace-0". As part of our Hadoop pre-processing, we can perform a join against a redirect lookup table to find the true Wikipedia pageid for a given article title.

mysql> select * from page_lookups where page_id = 534366; +---------+------------------------------------------------+--------------+---------+-------------+ | id | redirect_title | true_title | page_id | page_latest | +---------+------------------------------------------------+--------------+---------+-------------+ | 219291 | Barack_Obama | Barack Obama | 534366 | 276223690 | | 3151538 | Barak_Obama | Barack Obama | 534366 | 276223690 | | 3151543 | 44th_President_of_the_United_States | Barack Obama | 534366 | 276223690 | | 3151544 | Barach_Obama | Barack Obama | 534366 | 276223690 | | 3151545 | Senator_Barack_Obama | Barack Obama | 534366 | 276223690 | .... ....

| 3151644 | Rocco_Bama | Barack Obama | 534366 | 276223690 | | 3151645 | Barack_Obama's | Barack Obama | 534366 | 276223690 | | 3151646 | B._Obama | Barack Obama | 534366 | 276223690 | +---------+------------------------------------------------+--------------+---------+-------------+ 110 rows in set (11.15 sec)

h2. Using Hadoop & Hive on EC2 to Spot Trends in Time Series Data

Processing large log datasets like this is a typical use case for Hadoop. By using MapReduce and EC2 we can run millions of regressions on a distributed cluster to quickly answer analytical questions or generate real time analytics on metrics of interest.

h3. Task 1: Generate Daily Timelines from 1TB of Historical Hourly Logs

Our first task is to calculate daily page views for all 2.5 million english Wikipedia articles, so we can display dynamic Google finance style timelines on the site.

!http://trendingtopics.s3.amazonaws.com/images/american_idol_daily_pageviews.png!

h4. Mounting the Public Dataset Volume

First, we need to access the Amazon public dataset. From your local machine, launch a small EC2 Ubuntu instance:

skom:~ pskomoroch$ ec2-run-instances ami-5394733a -k gsg-keypair -z us-east-1a

Once it is running and you have the instance id, create and attach an EBS Volume using the Wikipedia Traffic Statistics "public snapshot":http://developer.amazonwebservices.com/connect/entry.jspa?externalID=2596 (make sure the volume is created in the same availability zone as the ec2 instance)

skom:~ pskomoroch$ ec2-create-volume --snapshot snap-753dfc1c -z us-east-1a
skom:~ pskomoroch$ ec2-attach-volume vol-ec06ea85 -i i-df396cb6 -d /dev/sdf

Next, ssh into the instance and mount the volume

skom:~ pskomoroch$ ssh [email protected]
root@domU-12-xx-xx-xx-75-81:/mnt# mkdir /mnt/wikidata
root@domU-12-xx-xx-xx-75-81:/mnt# mount /dev/sdf /mnt/wikidata

h4. Uploading the Data to S3

Hadoop can use Amazon S3 as a distributed file system out of the box, so it will be easier for later EC2 jobs if we store a full copy of the EBS data on S3. You could eliminate the need for this step by using the Beta of "Cloudera Hadoop with EBS Integration":http://www.cloudera.com/hadoop-ec2-ebs-beta (persistent temporary clusters on AWS). We will copy the raw pagecount data up to S3 from the EBS volume using S3cmd. This install and configuration can be automated later on, but for now you will need your AWS access keys handy for the configuration.

root@domU-12-xx-xx-xx-75-81:/mnt# apt-get update
root@domU-12-xx-xx-xx-75-81:/mnt# apt-get install -y s3cmd
root@domU-12-xx-xx-xx-75-81:/mnt# s3cmd --configure

Set an environment variable MYBUCKET to a bucket you own on Amazon S3

root@domU-12-xx-xx-xx-75-81:/mnt# MYBUCKET=mytrendingtopics	

You can upload all the files with a single command, or select a subset based on the file name in case an upload fails:

/mnt# time s3cmd put --force wikidata/wikistats/pagecounts/pagecounts-200904* s3://$MYBUCKET/wikistats/
/mnt# time s3cmd put --force wikidata/wikistats/pagecounts/pagecounts-200903* s3://$MYBUCKET/wikistats/
/mnt# time s3cmd put --force wikidata/wikistats/pagecounts/pagecounts-200902* s3://$MYBUCKET/wikistats/
/mnt# time s3cmd put --force wikidata/wikistats/pagecounts/pagecounts-200901* s3://$MYBUCKET/wikistats/
/mnt# time s3cmd put --force wikidata/wikistats/pagecounts/pagecounts-200812* s3://$MYBUCKET/wikistats/
/mnt# time s3cmd put --force wikidata/wikistats/pagecounts/pagecounts-200811* s3://$MYBUCKET/wikistats/
/mnt# time s3cmd put --force wikidata/wikistats/pagecounts/pagecounts-200810* s3://$MYBUCKET/wikistats/	
....

Each month has approximately 40GB of compressed data and takes around 30 minutes to upload from a small EC2 instance. Data transfer between EC2 and S3 is free, so it will cost 50 cents for the entire one time upload.

h4. Customize the Cloudera Hadoop Ubuntu launch scripts

One of the advantages of the Cloudera Hadoop distribution is that it allows you to customize your Hadoop cluster and operating system environment on EC2 to meet the specific needs of your MapReduce jobs. We will customize our cluster by making several additional edits to the files bin/hadoop-ec2-env.sh and bin/hadoop-ec2-init-remote-cloudera-ubuntu.sh.

By default, the scripts launch a Fedora instance. We can see which images are available by typing "ec2-describe-images -a | grep cloudera" at the command line. For this tutorial we will be using Ubuntu instances, so comment out the fedora version

HADOOP_VERSION=fedora-20090602

HADOOP_VERSION=ubuntu-20090602

Also change USER_DATA_FILE to point to the Ubuntu script (hadoop-ec2-init-remote-cloudera-ubuntu.sh).

The script to run on instance boot.

#USER_DATA_FILE=hadoop-ec2-init-remote-cloudera-fedora.sh USER_DATA_FILE=hadoop-ec2-init-remote-cloudera-ubuntu.sh

We can optionally select a larger EC2 instance type to use in the cluster. Using one of the high CPU instances (c1.medium or c1.xlarge) is recommended:

The EC2 instance type: m1.small, m1.large, m1.xlarge

#INSTANCE_TYPE="m1.small" #INSTANCE_TYPE="m1.large" #INSTANCE_TYPE="m1.xlarge" INSTANCE_TYPE="c1.medium" #INSTANCE_TYPE="c1.xlarge"

We will install several extra packages on the Ubuntu instances by editing the USER_PACKAGES setting at the top of the bin/hadoop-ec2-init-remote-cloudera-ubuntu.sh file:

Specify user packages to install here, separated by whitespace (e.g. 'r-base' for R)

USER_PACKAGES='r-base r-base-core r-base-dev r-base-html r-base-latex r-cran-date python-rpy python-setuptools python-docutils python-support python-distutils-extra python-simplejson git-core s3cmd'

We also add our AWS keys so that our Hadoop cluster will be configured to use S3 as a file system. To configure Hadoop with our keys we need to add properties for fs.s3n.awsAccessKeyId and fs.s3n.awsSecretAccessKey to the hadoop-site.xml section of bin/hadoop-ec2-init-remote-cloudera-ubuntu.sh

cat > /etc/hadoop/conf.dist/hadoop-site.xml

fs.s3n.awsAccessKeyId YOUR_ACCESS_KEY_HERE

fs.s3n.awsSecretAccessKey YOUR_SECRET_ACCESS_KEY_HERE

dfs.block.size 134217728 true

We can also change th

Issues· 0 open

View all issuesOpen on GitHub

No open issues yet, or sync has not completed.

> Tags

Ruby

No comments yet. Be the first to share.

> Details

PublishedAug 1, 2026
UpdatedSep 17, 2026
Category编程语言
PricingOpen source

> Related tools

T
TypeScript
JavaScript 的超集,为前端与全栈提供静态类型
P
Python
通用编程语言,广泛用于 Web、数据与 AI
G
Go
Google 推出的简洁高效系统语言