middleman-s3_sync

Middleman::S3Sync

This gem determines which files need to be added, updated and optionally deleted and only transfer these files up. This reduces the impact of an update on a web site hosted on S3.

Why not Middleman Sync?

Middleman Sync does a great job to push Middleman generated websites to S3. The only issue I have with it is that it pushes every files under build to S3 and doesn't seem to properly delete files that are no longer needed.

Version Support

• Use middleman-s3_sync version 4.x for Middleman 4.x
• Use middleman-s3_sync version 3.x for Middleman 3.x

What's New in 4.7.0

New Features

• after_s3_sync callback for post-sync hooks (notifications, custom actions)
• scan_build_dir option to sync files outside the Middleman sitemap
• routing_rules option for S3 website redirect configuration
• Improved content type detection with mime-types gem fallback

Performance & Efficiency

• Batch deletes using S3 delete_objects (up to 1,000 keys per request)
• Streaming uploads to reduce memory usage on large files
• Single-pass MD5 computation avoids redundant file reads
• Single-pass resource categorization (create/update/delete)
• Faster redundant-path pruning for CloudFront invalidations

Reliability

• Thread-safe CloudFront invalidation path tracking (mutex-protected Set)
• Cached CloudFront client to reduce re-instantiation overhead
• Proper sitemap population before sync (ensure_resource_list_updated!)
• Fixed redirect detection to return boolean values

Developer Experience

• Extension now properly delegates option writers (verbose=, dry_run=, etc.)
• GitHub Actions CI and release workflows
• Tightened gemspec with bounded dependency versions
• Ruby >= 3.0 requirement

Installation

Add this line to your application's Gemfile:

gem 'middleman-s3_sync'

And then execute:

$ bundle

Or install it yourself as:

$ gem install middleman-s3_sync

Usage

You need to add the following code to your config.rb file:

…

You can then start synchronizing files with S3 through middleman s3_sync.

Configuration Defaults

The following defaults apply to the configuration items:

Setting	Default
aws_access_key_id	-
aws_secret_access_key	-
bucket	-
delete	true
after_build	false
prefer_gzip	true
reduced_redundancy_storage	false
path_style	true
encryption	false
acl	'public-read'
version_bucket	false
cloudfront_distribution_id	-
cloudfront_invalidate	false
cloudfront_invalidate_all	false
cloudfront_wait	false

Setting AWS Credentials

There are several secure ways to provide AWS credentials for s3_sync. Using temporary, least-privilege credentials is strongly recommended.

Best Practices for AWS Credentials (Recommended)

1. AWS IAM Roles (Most Secure)

For CI/CD and Cloud Environments

• EC2 Instance Profiles: If running on EC2, use IAM roles attached to your instance. Credentials are automatically rotated and managed by AWS.
• ECS Task Roles: For container workloads, use task roles to provide permissions to specific containers.
• CI/CD Service Roles: Most CI/CD services (GitHub Actions, CircleCI, etc.) offer native AWS integrations that support assuming IAM roles.

For Local Development

• AWS IAM Identity Center (SSO) and configured profiles in your AWS config file
• AWS CLI credential process to integrate with external identity providers
• Role assumption with short-lived credentials through aws sts assume-role

To use these methods, you don't need to specify credentials in your Middleman configuration. The AWS SDK will automatically detect and use them.

2. Environment Variables with Temporary Credentials

Using environment variables with short-lived credentials from role assumption:

# Obtain temporary credentials via assume-role or similar
# Then set these environment variables
export AWS_ACCESS_KEY_ID="temporary-access-key"
export AWS_SECRET_ACCESS_KEY="temporary-secret-key"
export AWS_SESSION_TOKEN="temporary-session-token"
export AWS_BUCKET="your-bucket-name"

These environment variables are used when credentials are not otherwise specified:

Setting	Environment Variable
aws_access_key_id	ENV['AWS_ACCESS_KEY_ID']
aws_secret_access_key	ENV['AWS_SECRET_ACCESS_KEY']
aws_session_token	ENV['AWS_SESSION_TOKEN']
bucket	ENV['AWS_BUCKET']

Alternative Methods (Not Recommended for Production)

The following methods are less secure and should be avoided in production environments:

Through .s3_sync File

You can create a .s3_sync at the root of your middleman project. The credentials are passed in the YAML format. The keys match the options keys.

A sample .s3_sync file is included at the root of this repo.

SECURITY WARNING: If using this approach, ensure you add .s3_sync to your .gitignore to prevent accidentally committing credentials to your repository. Consider using this only for local development and only with temporary credentials.

Through config.rb

You can set the AWS credentials in the activation block, but this is strongly discouraged:

SECURITY WARNING: This method could lead to credentials being committed to version control, potentially exposing sensitive information. Never use long-lived credentials with this method.

Through Command Line

Credentials can be passed via command line options, but this may expose them in shell history:

SECURITY WARNING: Command line parameters may be visible in process listings or shell history. Consider using environment variables or IAM roles instead.

CloudFront Invalidation

The gem can automatically invalidate CloudFront cache after a successful sync. This ensures that your CloudFront distribution serves the latest content immediately after deployment.

Configuration

activate :s3_sync do |s3_sync|
  # ... other configuration ...
  s3_sync.cloudfront_distribution_id = 'E1234567890123'  # Your CloudFront distribution ID
  s3_sync.cloudfront_invalidate      = true             # Enable invalidation
  s3_sync.cloudfront_invalidate_all  = false            # Invalidate only changed files
end

Configuration Options

Setting	Default	Description
cloudfront_distribution_id	-	CloudFront distribution ID to invalidate
cloudfront_invalidate	false	Enable CloudFront invalidation after sync
cloudfront_invalidate_all	false	Invalidate all paths (/*) instead of only changed files
cloudfront_invalidation_batch_size	1000	Maximum paths per invalidation request
cloudfront_invalidation_max_retries	5	Maximum retries for rate-limited requests
cloudfront_invalidation_batch_delay	2	Delay in seconds between invalidation batches
cloudfront_wait	false	Wait for CloudFront invalidation to complete

Command Line Options

You can also control CloudFront invalidation via command line:

…

Available CloudFront Command Line Options

Option	Short	Description
--cloudfront-distribution-id	-d	CloudFront distribution ID
--cloudfront-invalidate	-c	Enable CloudFront invalidation
--cloudfront-invalidate-all	-a	Invalidate all paths (/*)
--cloudfront-invalidation-batch-size	-	Max paths per request (default: 1000)
--cloudfront-invalidation-max-retries	-	Max retries for rate limits (default: 5)
--cloudfront-invalidation-batch-delay	-	Delay between batches in seconds (default: 2)
--cloudfront-wait	-w	Wait for invalidation to complete

How It Works

1. Smart Invalidation: By default, only files that were created, updated, or deleted during the sync are invalidated
2. Path Optimization: Duplicate paths are removed and redundant paths (covered by wildcards) are eliminated
3. Batch Processing: Large numbers of paths are split into multiple invalidation requests to respect CloudFront limits
4. Rate Limit Handling: Automatic retry with exponential backoff when CloudFront rate limits are hit
5. Error Handling: Invalidation failures are logged but don't stop the sync process
6. Dry Run Support: Use --dry-run to see what would be invalidated without making actual API calls

IAM Permissions

Your AWS credentials need CloudFront permissions in addition to S3:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "cloudfront:CreateInvalidation",
        "cloudfront:GetInvalidation",
        "cloudfront:ListInvalidations"
      ],
      "Resource": "arn:aws:cloudfront::*:distribution/E1234567890123"
    }
  ]
}

Cost Considerations

• CloudFront allows 1,000 free invalidation paths per month
• Additional invalidations cost $0.005 per path
• Use cloudfront_invalidate_all: true for major updates to minimize costs (counts as 1 path)
• Consider the trade-off between immediate cache invalidation and cost

Callbacks

after_s3_sync

You can configure a callback that runs after the sync completes. This is useful for triggering notifications, updating external services, or running post-deployment tasks.

activate :s3_sync do |s3_sync|
  # ... other configuration ...
  
  # Using a lambda/proc
  s3_sync.after_s3_sync = ->(results) {
    puts "Created: #{results[:created]} files"
    puts "Updated: #{results[:updated]} files"
    puts "Deleted: #{results[:deleted]} files"
    puts "Invalidation paths: #{results[:invalidation_paths].join(', ')}"
  }
end

The callback receives a hash with sync results:

Key	Type	Description
:created	Integer	Number of files created
:updated	Integer	Number of files updated
:deleted	Integer	Number of files deleted
:invalidation_paths	Array	CloudFront paths that were invalidated

You can also use a symbol to call a method on the Middleman app:

# In config.rb
def notify_slack(results)
  # Send deployment notification to Slack
end

activate :s3_sync do |s3_sync|
  # ... other configuration ...
  s3_sync.after_s3_sync = :notify_slack
end

Callbacks that take no arguments are also supported:

activate :s3_sync do |s3_sync|
  s3_sync.after_s3_sync = -> { puts "Sync complete!" }
end

IAM Policy

Here's a sample IAM policy with least-privilege permissions that will allow syncing to a bucket named "mysite.com":

{
  "Version": "2012-10-17",