用于跟踪服务器日志趋势的 Rails 应用程序 - 由 EC2 上的 Cloudera Hadoop 分发版提供支持
h1. Tracking Trends with Hadoop & Hive on EC2
This repository contains the full source code for "Trendingtopics.org":http://www.trendingtopics.org, built by "Data Wrangling":http://www,datawrangling.com to demonstrate how Hadoop & EC2 can power a data driven website. The trend statistics and time series data that run the site are updated periodically by launching a temporary EC2 cluster running the "Cloudera Hadoop Distribution":http://www.cloudera.com/hadoop-ec2. Our initial seed data consists of the raw Wikipedia database content dump along with hourly "traffic logs":http://stats.grok.se/ for all articles collected from the Wikipedia squid proxy (curated by Domas Mituzas). We made the first 7 months of this hourly data for all articles available as an Amazon Public Dataset.
The current trend calculations are run with Hadoop Streaming and Hive. The output produced by these Hadoop jobs is loaded into MySQL and indexed to power the live site. The demo data included with the Rails app on Github was generated from a sample of 100 trending articles on June 6th 2009. A much larger snapshot is available on Amazon Public Datasets (snap-753dfc1c). The Rails app and MySQL database are deployed on Amazon EC2 using Paul Dowman’s "EC2onRails":http://ec2onrails.rubyforge.org/.
!http://trendingtopics.s3.amazonaws.com/images/trendingtopics_dashboard.png!
h3. Application Features
h3. How Hadoop is Used in the Application
The rest of this document will walk through the code which powers the site and describe some basic approaches for extracting trends from log data with Hadoop and Hive. The individual steps are run manually for the code walkthrough, but would but likely be automated with some combination of Rake, Cron, Puppet, or Chef in a real production environment.
h3. Prerequisites
h2. Raw Data: Hourly Wikipedia Article Traffic Logs
The "Wikipedia Traffic Statistics":http://developer.amazonwebservices.com/connect/entry.jspa?externalID=2596 Amazon Public Dataset we will process covers a 7 month period from October, 01 2008 to April, 30 2009. This data is regularly logged from the Wikipedia squid proxy by Domas Mituzas and was compiled by Data Wrangling for this demo.
h3. Log file format
Each log file is named with the date and time of collection: pagecounts-20090430-230000.gz. The individual hourly files are around 55 MB when compressed, so 8 months of compressed data takes up about 300 GB of space.
Each line has 4 fields: projectcode, pagename, pageviews, bytes
$ grep '^en Barack' pagecounts-20090521-100001
en Barack 8 1240112
en Barack%20Obama 1 1167
en Barack_H._Obama 1 142802
en Barack_H_Obama 3 428946
en Barack_H_Obama_Jr. 2 285780
en Barack_Hussein_Obama,_Junior 2 285606
en Barack_O%27Bama 1 142796
en Barack_Obama 701 139248439
en Barack_Obama%27s_first_100_days 2 143181
en Barack_Obama,_Jr 2 285755Many of the raw wiki log page titles are percent-encoded. To match these with the page titles in the Wikipedia database, you can transform them as follows in Python:
$ python
>>> import urllib
>>> escaped_title = '%22Klondike_Kate%22_Rockwell'
>>> print urllib.unquote_plus(escaped_title)
"Klondike_Kate"_Rockwellh3. Handling Wikipedia Redirects
To further complicate the raw data processing, many page titles are actually Wikipedia redirects which do not match to the primary key of the Wikipedia "Pages" table. The non-redirect articles are referred to as belonging to "namespace-0". As part of our Hadoop pre-processing, we can perform a join against a redirect lookup table to find the true Wikipedia pageid for a given article title.
mysql> select * from page_lookups where page_id = 534366; +---------+------------------------------------------------+--------------+---------+-------------+ | id | redirect_title | true_title | page_id | page_latest | +---------+------------------------------------------------+--------------+---------+-------------+ | 219291 | Barack_Obama | Barack Obama | 534366 | 276223690 | | 3151538 | Barak_Obama | Barack Obama | 534366 | 276223690 | | 3151543 | 44th_President_of_the_United_States | Barack Obama | 534366 | 276223690 | | 3151544 | Barach_Obama | Barack Obama | 534366 | 276223690 | | 3151545 | Senator_Barack_Obama | Barack Obama | 534366 | 276223690 | .... ....
| 3151644 | Rocco_Bama | Barack Obama | 534366 | 276223690 | | 3151645 | Barack_Obama's | Barack Obama | 534366 | 276223690 | | 3151646 | B._Obama | Barack Obama | 534366 | 276223690 | +---------+------------------------------------------------+--------------+---------+-------------+ 110 rows in set (11.15 sec)
h2. Using Hadoop & Hive on EC2 to Spot Trends in Time Series Data
Processing large log datasets like this is a typical use case for Hadoop. By using MapReduce and EC2 we can run millions of regressions on a distributed cluster to quickly answer analytical questions or generate real time analytics on metrics of interest.
h3. Task 1: Generate Daily Timelines from 1TB of Historical Hourly Logs
Our first task is to calculate daily page views for all 2.5 million english Wikipedia articles, so we can display dynamic Google finance style timelines on the site.
!http://trendingtopics.s3.amazonaws.com/images/american_idol_daily_pageviews.png!
h4. Mounting the Public Dataset Volume
First, we need to access the Amazon public dataset. From your local machine, launch a small EC2 Ubuntu instance:
skom:~ pskomoroch$ ec2-run-instances ami-5394733a -k gsg-keypair -z us-east-1aOnce it is running and you have the instance id, create and attach an EBS Volume using the Wikipedia Traffic Statistics "public snapshot":http://developer.amazonwebservices.com/connect/entry.jspa?externalID=2596 (make sure the volume is created in the same availability zone as the ec2 instance)
skom:~ pskomoroch$ ec2-create-volume --snapshot snap-753dfc1c -z us-east-1a
skom:~ pskomoroch$ ec2-attach-volume vol-ec06ea85 -i i-df396cb6 -d /dev/sdfNext, ssh into the instance and mount the volume
skom:~ pskomoroch$ ssh [email protected]
root@domU-12-xx-xx-xx-75-81:/mnt# mkdir /mnt/wikidata
root@domU-12-xx-xx-xx-75-81:/mnt# mount /dev/sdf /mnt/wikidatah4. Uploading the Data to S3
Hadoop can use Amazon S3 as a distributed file system out of the box, so it will be easier for later EC2 jobs if we store a full copy of the EBS data on S3. You could eliminate the need for this step by using the Beta of "Cloudera Hadoop with EBS Integration":http://www.cloudera.com/hadoop-ec2-ebs-beta (persistent temporary clusters on AWS). We will copy the raw pagecount data up to S3 from the EBS volume using S3cmd. This install and configuration can be automated later on, but for now you will need your AWS access keys handy for the configuration.
root@domU-12-xx-xx-xx-75-81:/mnt# apt-get update
root@domU-12-xx-xx-xx-75-81:/mnt# apt-get install -y s3cmd
root@domU-12-xx-xx-xx-75-81:/mnt# s3cmd --configureSet an environment variable MYBUCKET to a bucket you own on Amazon S3
root@domU-12-xx-xx-xx-75-81:/mnt# MYBUCKET=mytrendingtopics You can upload all the files with a single command, or select a subset based on the file name in case an upload fails:
/mnt# time s3cmd put --force wikidata/wikistats/pagecounts/pagecounts-200904* s3://$MYBUCKET/wikistats/
/mnt# time s3cmd put --force wikidata/wikistats/pagecounts/pagecounts-200903* s3://$MYBUCKET/wikistats/
/mnt# time s3cmd put --force wikidata/wikistats/pagecounts/pagecounts-200902* s3://$MYBUCKET/wikistats/
/mnt# time s3cmd put --force wikidata/wikistats/pagecounts/pagecounts-200901* s3://$MYBUCKET/wikistats/
/mnt# time s3cmd put --force wikidata/wikistats/pagecounts/pagecounts-200812* s3://$MYBUCKET/wikistats/
/mnt# time s3cmd put --force wikidata/wikistats/pagecounts/pagecounts-200811* s3://$MYBUCKET/wikistats/
/mnt# time s3cmd put --force wikidata/wikistats/pagecounts/pagecounts-200810* s3://$MYBUCKET/wikistats/
....Each month has approximately 40GB of compressed data and takes around 30 minutes to upload from a small EC2 instance. Data transfer between EC2 and S3 is free, so it will cost 50 cents for the entire one time upload.
h4. Customize the Cloudera Hadoop Ubuntu launch scripts
One of the advantages of the Cloudera Hadoop distribution is that it allows you to customize your Hadoop cluster and operating system environment on EC2 to meet the specific needs of your MapReduce jobs. We will customize our cluster by making several additional edits to the files bin/hadoop-ec2-env.sh and bin/hadoop-ec2-init-remote-cloudera-ubuntu.sh.
By default, the scripts launch a Fedora instance. We can see which images are available by typing "ec2-describe-images -a | grep cloudera" at the command line. For this tutorial we will be using Ubuntu instances, so comment out the fedora version
HADOOP_VERSION=ubuntu-20090602
Also change USER_DATA_FILE to point to the Ubuntu script (hadoop-ec2-init-remote-cloudera-ubuntu.sh).
#USER_DATA_FILE=hadoop-ec2-init-remote-cloudera-fedora.sh USER_DATA_FILE=hadoop-ec2-init-remote-cloudera-ubuntu.sh
We can optionally select a larger EC2 instance type to use in the cluster. Using one of the high CPU instances (c1.medium or c1.xlarge) is recommended:
#INSTANCE_TYPE="m1.small" #INSTANCE_TYPE="m1.large" #INSTANCE_TYPE="m1.xlarge" INSTANCE_TYPE="c1.medium" #INSTANCE_TYPE="c1.xlarge"
We will install several extra packages on the Ubuntu instances by editing the USER_PACKAGES setting at the top of the bin/hadoop-ec2-init-remote-cloudera-ubuntu.sh file:
USER_PACKAGES='r-base r-base-core r-base-dev r-base-html r-base-latex r-cran-date python-rpy python-setuptools python-docutils python-support python-distutils-extra python-simplejson git-core s3cmd'
We also add our AWS keys so that our Hadoop cluster will be configured to use S3 as a file system. To configure Hadoop with our keys we need to add properties for fs.s3n.awsAccessKeyId and fs.s3n.awsSecretAccessKey to the hadoop-site.xml section of bin/hadoop-ec2-init-remote-cloudera-ubuntu.sh
cat > /etc/hadoop/conf.dist/hadoop-site.xml
fs.s3n.awsAccessKeyId YOUR_ACCESS_KEY_HERE
fs.s3n.awsSecretAccessKey YOUR_SECRET_ACCESS_KEY_HERE
dfs.block.size 134217728 true
We can also change th
暂无开放 Issues,或尚未同步最近议题。