百科.dev
全部条目在线工具AI 编程趋势榜开源项目技术资讯提交条目
登录
< 返回工具列表
D

data-engineering-for-aws-immersion-day

> 编程语言
开源

数据工程深度学习日实验指导

198 stars0 点赞0 次浏览
访问官网GitHub

工具介绍

数据工程深度学习日实验指导

Amazon Web Services Data Engineering Immersion Day

Welcome to the lab Instruction!

Requirements:

Instructor Led :

  • AWS account - if you don't have one, please ask your instructor for the login detail.
  • Source RDS (Postgres) details - Your instructor should provide the database information.

Self-paced : If you want to run pre-requisite steps by yourself:

  • AWS account - if you don't have one, Sign-up for free AWS account . All sevices in this lab are not covered under free tier, so it may incur some minor cost.

  • Follow the instruction in PreLab-instructor to set up a RDS source database yourself. It deploys:

    • A source RDS database environment along with required networking and IAM roles.
    • Populate the source database with event ticket sales data

Click the Deploy to AWS icons below to stand up the RDS database infrastructure.

Region Launch Template
N.Virginia (us-east-1)

What you'll do:

These labs are designed to be completed in sequence, and the full set of instructions are documented below. Read and follow along to complete the labs. Our lab instructor will give you a high-level overview of the labs and help answer any questions. Don't worry if you get stuck, we provide hints along the way.

Ensure your region is US East (N. Virginia)

  • AutoComplete DMS lab: Skip Workshop Setup & Lab 1
  • Workshop Setup: Create working environment on AWS
  • Lab 1: Hydrateing the data lake via DMS
  • Lab 2: Transforming in the data lake with Glue
  • Lab 3: Consuming the data lake with Athena & QuickSight
  • Lab 4: Machine learning in the data lake
  • Lab 5: AWS Lake Formation Lab
  • Lab 6: Modernize Data Warehouse with Amazon Redshift Spectrum
  • Cleanup Put everything away nicely

Let's Begin!

AutoComplete DMS lab:

Download the lab instruction file

This lab is designed to automate the Data Lake hydration with AWS Database Migration Service (AWS DMS), so we can fast forward to Lab2-Transforming in the data lake with Glue.

If you prefer to get hands-on with AWS DMS service, please skip this lab and proceed to Workshop Setup and Lab1-Hydrateing the data lake via DMS

^ back to the top

Workshop Setup:

  1. Open the CloudFormation launch template link in a new tab. It will load a CloudFormation Dashboard and start the creation process for your lab environment, which deploys:

    • A required Virtual Private Cloud for the DMS
    • An Amazon S3 bucket for the Data Lake
    • A required S3 bucket policy to restrict the data access by AWS DMS service
    • A Glue Service Role to use in later hands-on lab

Click the Deploy to AWS icons below to stand up the core workshop infrastructure.

Region Launch Template
N.Virginia (us-east-1)
  1. The template will automatically take you to the CloudFormation Console and start the stack creation process in N.Virginia region.

Proceed through the wizard to launch the stack. Leave all options at their default values, but make sure to check the box to allow CloudFormation to create IAM roles on your behalf:

See the Events tab for progress on the stack launch that may take up to 5 minutes. You can also see details of any problems here if the launch fails. Proceed to the next step once the stack status advances to "CREATE_COMPLETE".

  1. Click in the Output tab and take note of value for BucketName, GlueLabRole and DMSLabRoleS3, which you are going to use in future labs.

Checkpoint:

At this point, the data lake workshop environment has been setup, looks like this:

^ back to top

Lab 1 - Hydrating the data lake via DMS

Download the lab1 instruction file

Here's what you're going to work on in lab 1:

Checkpoint:

At this point, your DMS task should be finished with 'load Complete' status, and 16 tables are loaded in S3 from RDS by DMS

^ back to the top

Lab 2 - Transforming data with Glue

Download the lab2 instruction file

Here's what you're going to work on in lab 2:

Checkpoint:

Nice work! You've successfully converted CSV raw data to Parquet, and added parquet tables to the Glue Data Catalog

^ back to top

Lab 3 - Consuming data with Athena and Quicksight

Download the lab3 instruction file

Here's what you're going to work on in lab 3:

Checkpoint:

Sweet! Now you have queried the Data Lake and visualized it in QuickSight. Next, We'll consume the Data Lake via machine learning.

^ back to top

Lab 4 - Machine learning in the data lake

Download the lab4 instruction file

Here's what you will be implementing:

Checkpoint:

Congratulations, you've successfully built the Data Lake from end to end. If you have time, try the optional steps in the 4 labs above. Otherwise, please remember to follow the steps below in the Workshop Cleanup to make sure all assets created during the workshop are removed so you do not see unexpected charges after today.

^ back to the top

Lab 5 - AWS Lake Formation Lab

Make sure you have completed

  • Lab 1. Hydrating the Data Lake with DMS
  • Lab 2. ETL with AWS Glue

Download the lab5 instruction file

Lab 6 - Modernize Data Warehouse with Amazon Redshift Spectrum

In this lab, we show you how to query petabytes of data with Amazon Redshift and exabytes of data in your Amazon S3 data lake, without loading or moving objects. We will also demonstrate how you can leverage views which union data in direct attached storage as well as in your S3 Datalake to create a single source of truth. Finally, we will demonstrate strategies for aging off old data into S3 and maintaining only the most recent data in Amazon Redshift direct attached storage.

Contents

  • Before You Begin
  • What Happened in 2016
  • Go Back In Time
  • Create a Single Version of Truth
  • Plan for the Future

Before You Begin

This lab requires a new Redshift cluster in US-WEST-2 (Oregon), use the following link to And gather the following information from the stack output above:

  • [Your-Redshift_Hostname]
  • [Your-Redshift_Port]
  • [Your-Redshift_Username]
  • [Your-Redshift_Password]
  • [Your-Redshift_Role_ARN]

What Happened in 2016

In the first part of this lab, we will perform the following activities:

  • Load the Green company data for January 2016 into Redshift direct-attached storage (DAS) with COPY.
  • Collect supporting/refuting evidence for the impact of the January, 2016 blizzard on taxi usage.
  • The CSV data is by month on Amazon S3. Here's a quick screenshot from the S3 console:
https://s3.console.aws.amazon.com/s3/buckets/us-west-2.serverless-analytics/NYC-Pub/green/?region=us-west-2&tab=overview&prefixSearch=green_tripdata_2016
  • Here's Sample data from one file which can be previewed directly in the S3 console:
https://s3.console.aws.amazon.com/s3/object/us-west-2.serverless-analytics/NYC-Pub/green/green_tripdata_2013-08.csv?region=us-west-2&tab=select

Build your DDL

Login to your query editor, create a schema workshop_das and table workshop_das.green_201601_csv for tables that will reside on the Redshift compute nodes, AKA the Redshift direct-attached storage (DAS) tables.

Hint

…

Build your Copy Command

  • Build your copy command via Redshift query editor, to copy the data from Amazon S3. This dataset has the number of taxi rides in the month of January 2016.

Hint

python
COPY workshop_das.green_201601_csv
FROM 's3://us-west-2.serverless-analytics/NYC-Pub/green/green_tripdata_2016-01.csv'
IAM_ROLE '[Your-Redshift_Role_ARN]'
DATEFORMAT 'auto'
IGNOREHEADER 1
DELIMITER ','
IGNOREBLANKLINES
;
  • Determine how many rows you just loaded.

Hint

select count(1) from workshop_das.green_201601_csv;
--1445285

HINT: The [Your-Redshift_Role_ARN] in the above command should be replaced by the CloudFormation output value at the beginning of the lab.

Pin-point the Blizzard

In this month, there is a date which had the lowest number of taxi rides due to a blizzard. Can you find that date?

SQL-Based Hint

python
SELECT TO_CHAR(pickup_datetime, 'YYYY-MM-DD'),
COUNT(*)
FROM workshop_das.green_201601_csv
GROUP BY 1
ORDER BY 1;

Go Back in Time

In the next part of this lab, we will perform the following activities:

  • Query historical data residing on S3 by create an external DB for Redshift Spectrum.
  • Introspect the historical data, perhaps rolling-up the data in novel ways to see trends over time, or other dimensions.
  • Enforce reasonable use of the cluster with Redshift Spectrum-specific Query Monitoring Rules (QMR).
    • Test the QMR setup by writing an excessive-use query.

Note the partitioning scheme is Year, Month, Type (where Type is a taxi company). Here's a quick Screenshot:

https://s3.console.aws.amazon.com/s3/buckets/serverless-analytics/canonical/NY-Pub/?region=us-west-2&tab=overview
https://s3.console.aws.amazon.com/s3/buckets/serverless-analytics/canonical/NY-Pub/year%253D2016/month%253D1/?region=us-east-1&tab=overview
https://s3.console.aws.amazon.com/s3/buckets/serverless-analytics/canonical/NY-Pub/year%253D2016/month%253D1/type%253Dgreen/?region=us-east-1&tab=overview

Create external schema (and DB) for Redshift Spectrum

Because external tables are stored in a shared Glue Catalog for use within the AWS ecosystem, they can be built and maintained using a few different tools, e.g. Athena, Redshift, and Glue.

  • Use the AWS Glue Crawler to create your external table adb305.ny_pub stored in parquet format under location s3://us-west-2.serverless-analytics/canonical/NY-Pub/.

    1. Navigate to the Glue Crawler Page. https://console.aws.amazon.com/glue/home?#catalog:tab=crawlers

    2. Click on Add Crawler, and enter the crawler name NYTaxiCrawler and click Next.

    3. Select Data stores as the source type and click Next.

    4. Choose S3 as the data store and the include path of s3://us-west-2.serverless-analytics/canonical/NY-Pub

    5. Create an IAM Role and enter the name AWSGlueServiceRole-RedshiftImmersion.

    6. Select Run on demand for the frequency.

    7. Click on Add database and enter the Database of spectrumdb

    8. Select all remaining defaults. Once the Crawler has been created, click on Run Crawler.

    9. Once the Crawler has completed its run, you will see a new table in the Glue Catalog. https://console.aws.amazon.com/glue/home?#catalog:t

Issues· 0 开放

查看全部 Issues在 GitHub 打开

暂无开放 Issues,或尚未同步最近议题。

> 标签

Java

暂无评论,来聊聊你的看法吧

> 工具信息

发布日期2026年8月1日
最后更新2026年9月18日
分类编程语言
定价开源

> 相关工具

T
TypeScript
JavaScript 的超集,为前端与全栈提供静态类型
P
Python
通用编程语言,广泛用于 Web、数据与 AI
G
Go
Google 推出的简洁高效系统语言