git-remote-s3

git-remote-s3

This library enables to use Amazon S3 as a git remote and LFS server.

It provides an implementation of a git remote helper to use S3 as a serverless Git server.

It also provide an implementation of the git-lfs custom transfer to enable pushing LFS managed files to the same S3 bucket used as remote.

Table of Contents

• Installation
• Prerequisites
• Security
  • Data encryption
  • Access control
• Use S3 remotes
  • Create a new repo
  • Clone a repo
  • Branches, etc.
  • Using S3 remotes for submodules
• Repo as S3 Source for AWS CodePipeline
  • Archive file location
  • Example AWS CodePipeline source action config
• LFS
  • Creating the repo and pushing
  • Clone the repo
• Notes about specific behaviors of Amazon S3 remotes
  • Arbitrary Amazon S3 URIs
  • Concurrent writes
• Manage the Amazon S3 remote
  • Delete branches
  • Protected branches
• Under the hood
  • How S3 remote work
  • How LFS work
  • Debugging
• Credits

Installation

git-remote-s3 is a Python script and works with any Python version >= 3.9.

Run:

pip install git-remote-s3

Prerequisites

Before you can use git-remote-s3, you must:

• Complete initial configuration:

  • Creating an AWS account
  • Configuring an IAM user or role

• Create an AWS S3 bucket (or have one already) in your AWS account.

• Attach a minimal policy to that user/role that allows the to the S3 bucket:

  json复制

  {
    "Version": "2012-10-17",
    "Statement": [
      {
        "Sid": "S3ObjectAccess",
        "Effect": "Allow",
        "Action": ["s3:PutObject", "s3:GetObject", "s3:DeleteObject"],
        "Resource": ["arn:aws:s3:::/*"]
      },
      {
        "Sid": "S3ListAccess",
        "Effect": "Allow",
        "Action": ["s3:ListBucket"],
        "Resource": ["arn:aws:s3:::"]
      }
    ]
  }

• Optional (but recommended) - use SSE-KMS Bucket keys to encrypt the content of the bucket, ensure the user/role create previously has the permission to access and use the key.

{
  "Sid": "KMSAccess",
  "Effect": "Allow",
  "Action": ["kms:Decrypt", "kms:GenerateDataKey"],
  "Resource": ["arn:aws:kms:::key/"]
}

• Install Python and its package manager, pip, if they are not already installed. To download and install the latest version of Python, visit the Python website.
• Install Git on your Linux, macOS, Windows, or Unix computer.
• Install the latest version of the AWS CLI on your Linux, macOS, Windows, or Unix computer. You can find instructions here.

Security

Data encryption

All data is encrypted at rest and in transit by default. To add an additional layer of security you can use customer managed KMS keys to encrypt the data at rest on the S3 bucket. We recommend to use Bucket keys to minimize the KMS costs.

Access control

Access control to the remote is ensured via IAM permissions, and can be controlled at:

• bucket level
• prefix level (you can use prefixes to store multiple repos in the same S3 bucket thus minimizing the setup effort)
• KMS key level

If you store multiple repos in a single bucket but would like to separate permissions to access each repo, you can do so by modifying the resource definitions for the object related action to specify the repo prefix and by adding a condition to the ListBucket action to restrict the operation to matching prefixes (and by consequence the corresponding repo) :

      {
        "Sid": "S3ObjectAccess",
        "Effect": "Allow",
        "Action": [
          "s3:PutObject",
          "s3:GetObject",
          "s3:DeleteObject"
        ],
        "Resource": ["arn:aws:s3::://*"]
      },
      {
        "Sid": "S3ListObjects",
        "Effect": "Allow",
        "Action": [
          "s3:ListBucket",
        ],
        "Condition": {
          "StringEquals": {
            "s3:prefix": ""
          }
        },
        "Resource": ["arn:aws:s3:::"]
      },

Using the condition key restricts the access operation to the content of the specific repo in the bucket.

Use S3 remotes

Create a new repo

S3 remotes are identified by the prefix s3:// and at the bare minimum specify the name of the bucket. You can also provide a key prefix as in s3://my-git-bucket/my-repo and a profile s3://my-profile@my-git-bucket/myrepo.

mkdir my-repo
cd my-repo
git init
git remote add origin s3://my-git-bucket/my-repo

You can then add a file, commit and push the changes to the remote:

echo "Hello" > hello.txt
git add -A
git commit -a -m "hello"
git push --set-upstream origin main

The remote HEAD is set to track the branch that has been pushed first to the remote repo. To change the remote HEAD branch, delete the HEAD object s3:////HEAD and then run git-remote-s3 doctor s3:///.

When you use s3+zip:// instead of s3://, an additional zip archive named repo.zip is uploaded next to the sha.bundle file. This is for example useful if you want to use the Repo as a S3 Source for AWS CodePipeline, which expects a .zip file. The path on S3 when you push to the main branch is for example refs/heads/main/repo.zip. See How S3 remote work for more details about the bundle file.

Clone a repo

To clone the repo to another folder just use the normal git syntax using the s3 URI as remote:

git clone s3://my-git-bucket/my-repo my-repo-clone

Branches, etc.

Creating branches and pushing them works as normal:

cd my-repo
git checkout -b new_branch
touch new_file.txt
git add -A
git commit -a -m "new file"
git push origin new_branch

All git operations that do not rely on communication with the server should work as usual (eg git merge)

Using S3 remotes for submodules

If you have a repo that uses submodules also hosted on S3, you need to run the following command:

git config protocol.s3.allow always

Or, to enable globally:

git config --global protocol.s3.allow always

Repo as S3 Source for AWS CodePipeline

AWS CodePipeline offers an Amazon S3 source action as location for your source code and application files. But this requires to upload the source files as a single ZIP file. As briefly mentioned in Create a new repo, git-remote-s3 can create and upload zip archives. When you use s3+zip as URI Scheme when you add the remote, git-remote-s3 will automatically upload an archive that can be used by AWS CodePipeline.

Archive file location

Let's assume your bucket name is my-git-bucket and the repo is called my-repo. Run git remote add origin s3+zip://my-git-bucket/my-repo to use it as remote. When you now commit your changes and push to the remote, an additional repo.zip file will be uploaded to the bucket. For example, if you push to the main branch (git push origin main), the file is available under s3://my-git-bucket/my-repo/refs/heads/main/repo.zip. When you push to a branch called fix_a_bug it's available under s3://my-git-bucket/my-repo/refs/heads/fix_a_bug/repo.zip. And if you create and push a tag called v1.0 it will be s3://my-git-bucket/my-repo/refs/tags/v1.0/repo.zip.

Example AWS CodePipeline source action config

Your AWS CodePipeline Action configuration to trigger when you update your main branch:

• Action Provider: Amazon S3
• Bucket: my-git-bucket
• S3 object key: my-repo/refs/heads/main/repo.zip
• Change detection options: AWS CodePipeline

Visit Tutorial: Create a simple pipeline (S3 bucket) to learn more about a S3 bucket as source action.

LFS

To use LFS you need to first install git-lfs. You can refer to the official documentation on how to do this on your system.

Next, enable the S3 integration in the repo. There are two install modes:

# Per-remote (recommended; required when other LFS remotes coexist)
git-lfs-s3 install --remote 

# Unscoped (back-compat; applies the agent to ALL remotes in the repo)
git-lfs-s3 install

--remote writes a per-remote scoped configuration so git-lfs-s3 only fires for that one remote — letting an S3 remote coexist with non-S3 LFS remotes (e.g. GitHub, GitLab) without breaking their LFS push/pull. Use it whenever the repo has more than one remote.

The bare git-lfs-s3 install form sets lfs.standalonetransferagent globally and is short for:

git config --add lfs.customtransfer.git-lfs-s3.path git-lfs-s3
git config --add lfs.standalonetransferagent git-lfs-s3

git-lfs-s3 install --remote instead writes:

git config remote..lfsurl https://lfs-alias.git-remote-s3.test//
git config lfs..standalonetransferagent git-lfs-s3
git config lfs.customtransfer.git-lfs-s3.path git-lfs-s3

The lfs-alias.git-remote-s3.test host is a synthetic, never-contacted match key (the .test TLD is reserved by RFC 6761 for non-resolvable use). It exists only because git-lfs's URL parser does not natively understand s3:// URLs and would otherwise fall back to SSH-style endpoint discovery; setting remote..lfsurl short-circuits that path and gives the scoped agent lookup a stable URL to match against.

lfs.customtransfer.git-lfs-s3.path is necessarily repo-wide (git-lfs registers transfer adapters globally, not per URL), so git-lfs-s3 is still listed in the transfers array of batch requests sent to other LFS servers. If a server rejects requests naming unknown adapters, set:

git config lfs.basictransfersonly true

which makes git-lfs omit the transfers array entirely. This setting is repo-wide and limits other remotes to basic HTTPS transfers; GitHub-style hosts already use these (even over SSH remotes), so only the rare server that speaks exclusively the pure-SSH LFS protocol is affected.

Creating the repo and pushing

Let's assume we want to store TIFF file in LFS.

mkdir lfs-repo
cd lfs-repo
git init
git lfs install
git remote add origin s3://my-git-bucket/lfs-repo
git-lfs-s3 install --remote origin
git lfs track "*.tiff"
git add .gitattributes

git add file.tiff
git commit -a -m "my first tiff file"
git push --set-upstream origin main

Clone the repo

When cloning a repo using the S3 remote for LFS, git-lfs can't know how to fetch the files since we have yet to add the configuration.

It involves 2 extra steps.

% git clone s3://my-git-bucket/lfs-repo lfs-repo-clone
Error downloading object: file.tiff (54238cf): Smudge error: Error downloading file.tiff (54238cfaaaa42dda05da0e12bf8ee3156763fa35296085ccdef63b13a87837c5): batch request: ssh: Could not resolve hostname s3: Name or service not known: exit status 255
...

To fix:

cd lfs-repo-clone
git-lfs-s3 install --remote origin
git reset --hard main

Notes about sp