Building a Serverless Weather Pipeline on AWS: A Step-by-Step Walkthrough This is a build log for someone who's used AWS a bit — deployed a Lambda from the console, poked around S3 — but hasn't touched CDK, Step Functions, EventBridge Scheduler, or GitHub's OIDC setup before.
I'll explain each concept the first time it comes up, and show the actual code behind every piece, roughly in the order I built it.
Here's what it ends up doing: every 10 minutes, EventBridge Scheduler kicks off a Step Functions workflow that pulls current weather for five cities in parallel from a free public API, reshapes the results into JSON Lines, drops them into S3 in a partitioned layout, and makes them queryable in Athena with plain SQL.
No crawler, and no AWS credentials sitting anywhere in the GitHub repo that deploys it. kasukur / serverless-weather-pipeline AWS Serverless Weather Pipeline Serverless Weather Data Pipeline A small but complete serverless data pipeline on AWS walkthrough: EventBridge Scheduler → Step Functions → Lambda → S3 → Glue/Athena, deployed by GitHub Actions with no AWS access keys stored anywhere (authentication is via GitHub's OIDC provider). … View on GitHub Table of Contents What we're building, and why each piece is there Step 1: Project setup Step 2: Write the Lambdas first, before touching any infrastructure code Step 3: Unit test the logic before it ever touches AWS Step 4: Wire it together as a Step Functions state machine, in CDK Step 5: Trigger it on a schedule — EventBridge Scheduler Step 6: Land the data in S3, partitioned for querying Step 7: Make it queryable without a crawler — Glue partition projection Step 8: Wire up failure notifications Step 9: Deploy without a single AWS access key — GitHub OIDC Step 10: The GitHub Actions workflows themselves Step 11: Bootstrap and the first deploy Step 12: A bug that got past every green checkmark Step 13: Query the data Step 14: Cost and cleanup What this was actually a demo of What we're building, and why each piece is there Six services, each doing one job.
EventBridge Scheduler is the trigger: a managed cron that kicks off a Step Functions execution on a schedule, so nothing has to sit around just calling on a timer.
Step Functions is the orchestrator.
It describes a sequence of steps — branching, retries, parallelism — as a state machine (AWS calls the format Amazon States Language), instead of that logic living inside application code.
I reached for it because the "fetch five cities" step genuinely needs to run in parallel, retry the ones that fail transiently, and let the rest keep going if one city's API call fails outright.
In Step Functions that's a few lines of declarative config.
Hand-rolled inside one Lambda, it's a surprising amount of bookkeeping for something that sounds simple.
Lambda does the actual work at each step — three small, single-purpose functions, each one calling the next.
S3 is where the data lands, laid out with / prefixes so it reads like a partitioned table without needing an actual database.
Glue Data Catalog and Athena turn that S3 layout into something you can run SQL against.
Glue holds the schema, Athena runs the queries.
SNS catches anything that goes wrong anywhere in the workflow and emails an alert, so a broken run doesn't just fail quietly and go unnoticed.
Step 1: Project setup The whole thing is one CDK app (Python), split into two independent stacks.
One sets up secretless GitHub deploys; the other is the actual pipeline.
The split matters — the deploy-credentials stack gets deployed once, manually, with your own AWS credentials, and the pipeline stack deploys itself from GitHub Actions after that, automatically, forever, without a human needing credentials again.
Here's how it's laid out in the end: Step 2: Write the Lambdas first, before touching any infrastructure code Infrastructure-as-code has a slow feedback loop: write code, synthesize, deploy, wait, check.
Application logic doesn't have to.
Every Lambda here is plain Python with a entry point, written and unit-tested with pytest before any CDK code even references them.
Fetch calls the weather API for one city.
It has zero third-party dependencies — just from the standard library — so there's no dependency layer to build or keep in sync, and the deployment package stays tiny: Notice it raises on failure instead of catching and returning some error object.
That's on purpose.
This Lambda gets invoked once per city inside a Step Functions Map state, and Step Functions' own retry/catch handles a raised exception natively.
Swallow the error here instead, and you're just writing that same retry logic by hand, badly.
Transform takes the batch of per-city results (some of which may be failures, if a city never came back even after retries) and turns the successful ones into a JSON Lines body plus a partitioned S3 key.
I wrote it as a pure function with the AWS-facing as a thin wrapper, specifically so it's trivial to test without mocking anything: comes from Step Functions' own execution context, not inside the function.
Small detail, but it keeps the partition deterministic and testable instead of depending on whatever the wall clock happens to say the moment the Lambda runs.
Step 3: Unit test the logic before it ever touches AWS The fetch test mocks directly instead of pulling in a heavier HTTP-mocking library.
Stdlib in, stdlib mocked out: One thing that tripped me up briefly: every Lambda's entry file is named — that's just the Lambda convention, as the configured entry point — which means plain Python import machinery can't tell apart from .
Import both as a bare module called in the same test run and whichever loads first wins for every test file after it.
The fix is loading each one by explicit file path under its own module name: A little bit of ceremony, but it lets every Lambda file keep the same conventional name without the test suite getting confused about which one it's actually running.
Step 4: Wire it together as a Step Functions state machine, in CDK This is the part where CDK actually saves real effort — instead of hand-writing Amazon States Language JSON, the workflow gets built out of Python objects that generate it for you.
Chain a (a no-op that just reshapes input) into a , into two more Lambda invocations: That suffix is Amazon States Language for "resolve this as a JSONPath expression against the input, not a literal string." is one of a handful of fields Step Functions injects automatically about the execution itself — that's where actually comes from in the transform Lambda above. unwraps the Lambda's raw return value directly into the state's output.
Skip it, and you get the full invocation envelope — , , , and so on — and every downstream step has to reach into just to get at what you actually care about.
The retry policy gives each city two tries with a 2-second backoff before it gives up; the catch means a city that still fails after retries doesn't take the whole Map down with it — it gets routed to a state that records the error and moves on, while everyone else keeps going.
And chain it all together: , the last step, deserves its own section further down — the first version of it was wrong in a way that took a while to track down.
Step 5: Trigger it on a schedule — EventBridge Scheduler Not the older EventBridge "rules" cron feature — this is a newer, dedicated scheduling service with its own resource type, built for "start this one thing on this cadence" rather than routing arbitrary events around. is CDK's IAM shorthand — it writes an IAM policy statement for exactly on exactly this state machine's ARN, attached to exactly this role, so you never have to spell either ARN out by hand. means "run at exactly this cadence" rather than letting AWS jitter the start time to spread load. is a demo-friendly cadence for generating data fast; (or a expression, if you want something less regular) makes more sense for anything left running unattended.
Step 6: Land the dat