setup and manage continuous delivery pipelines for code libraries in multiple languages
setup and manage continuous delivery pipelines for code libraries in multiple languages
aws-delivlib is a fabulous library for defining continuous pipelines for building, testing and publishing code libraries through AWS CodeBuild and AWS CodePipeline.
aws-delivlib is used by the AWS Cloud Development Kit and was designed to support simultaneous delivery of the AWS CDK in multiple programming languages packaged via jsii.
A delivlib pipeline consists of the following sequential stages. Each stage will execute all tasks concurrently:
…The following sections describe each stage and the configuration options available:
To install, use npm / yarn:
$ npm i aws-delivlibor:
$ yarn add aws-delivliband import the library to your project:
import delivlib = require('aws-delivlib');The next step is to add a pipeline to your app. When you define a pipeline, the minimum requirement is to specify the source repository. All other settings are optional.
const pipeline = new delivlib.Pipeline(this, 'MyPipeline', {
// options
});The following sections will describe the various options available in your pipeline.
You can also take a look at the pipeline definition releasing the delivlib library itself for a real-world, working example.
The only required option when defining a pipeline is to specify a source repository for your project.
repo: Source Repository (required)The repo option specifies your source code repository for your project. You
could use either CodeCommit or GitHub.
To use an existing repository:
import codecommit = require('@aws-cdk/aws-codecommit');
// import an existing repository
const myRepo = codecommit.Repository.fromRepositoryName(this, 'TestRepo',
'delivlib-test-repo');
// ...or define a new repository (probably not what you want)
const myRepo = new codecommit.Repository(this, 'TestRepo');
// create a delivlib pipeline associated with this codebuild repo
new delivlib.Pipeline(this, 'MyPipeline', {
repo: new delivlib.CodeCommitRepo(myRepo),
// ...
});To connect to GitHub, you will need to store a Personal GitHub Access Token as an SSM Parameter and provide the name of the SSM parameter.
import cdk = require('@aws-cdk/core');
new delivlib.Pipeline(this, 'MyPipeline', {
repo: new delivlib.GitHubRepo({
repository: 'cdklabs/aws-delivlib',
token: cdk.SecretValue.secretsManager('my-github-token'),
}),
// ...
})branch: Source Control Branch (optional)The branch option can be used to specify the git branch to build from. The
default is master.
new delivlib.Pipeline(this, 'MyPipeline', {
repo: // ...
branch: 'dev',
})Pull Request Builds can be used to validate if changes submitted via a pull request successfully build and pass tests. They are triggered automatically by GitHub or CodeCommit when pull requests are submitted or updated.
Known in delivlib as AutoBuild, they can be enabled on the Pipeline and further configured -
new delivlib.Pipeline(this, 'MyPipeline', {
// ...
autoBuild: true,
autoBuildOptions: {
publicLogs: true,
},
});Delivlib also separately exports the AutoBuild construct that can be used to configure
AutoBuild on a project that doesn't have a pipeline associated, or for jobs that can be
run outside of a pipeline.
new delivlib.AutoBuild(this, 'MyAutoBuild', {
repo: // ...
});The second stage of a pipeline is to build your code. The following options allow you to do customize your build environment and scripts:
buildSpec: Build Script (optional)The default behavior will use the buildspec.yaml file from the root of your
source repository to determine the build steps.
See the the buildspec reference documentation in the CodeBuild User Guide.
Note that if you don't have an "artifacts" section in your buildspec, you won't be able to run any tests against the build outputs or publish them to package managers.
If you wish, you can use the buildSpec option, in which case CodeBuild will not
use the checked-in buildspec.yaml:
import codebuild = require('@aws-cdk/aws-codebuild');
new delivlib.Pipeline(this, 'MyPipeline', {
// ...
buildSpec: codebuild.BuildSpec.fromObject({
version: '0.2',
phases: {
build: {
commands: [
'echo "Hello, world!"'
]
}
},
artifacts: {
files: [ '**/*' ],
'base-directory': 'dist'
}
}),
});buildImage: Build container image (optional)The Docker image to use for the build container.
Default: the default image (if none is specified) is a custom Docker image which is provided as part of the jsii distribution called jsii/superchain. It is an environment that supports building libraries that target all programming languages supported by jsii. Find more information on the contents of the jsii/superchain image on the jsii homepage.
You can use the AWS CodeBuild API to specify any Linux/Windows Docker image for your build. Here are some examples:
codebuild.LinuxBuildImage.fromDockerRegistry('golang:1.11') - use an image from Docker Hubcodebuild.LinuxBuildImage.UBUNTU_14_04_OPEN_JDK_9 - OpenJDK 9 available from AWS CodeBuildcodebuild.WindowsBuildImage.WIN_SERVER_CORE_2016_BASE - Windows Server Core 2016 available from AWS CodeBuildcodebuild.LinuxBuildImage.fromEcrRepository(myRepo) - use an image from an ECR repositoryenv: Build environment variables (optional)Allows adding environment variables to the build environment:
new delivlib.Pipeline(this, 'MyPipeline', {
// ...
environment: {
FOO: 'bar'
}
});computeType: size of the AWS CodeBuild compute capacity (default: SMALL)privileged: run in privileged mode (default: false)The third stage of a delivlib pipeline is to execute tests. Tests are executed
in parallel only after a successful build and can access build artifacts as
defined in your buildspec.yaml.
The pipeline.addTest method can be used to add tests to your pipeline. Test
scripts are packaged as part of your delivlib CDK app.
delivlib.addTest('MyTest', {
platform: delivlib.ShellPlatform.LinuxUbuntu(), // or `ShellPlatform.Windows()`
scriptDirectory: 'path/to/local/directory/with/tests',
entrypoint: 'run.sh',
});scriptDirectory refers to a directory on the local file system which must
contain the entrypoint file.
Preferably make this path relative to the current file using path.join(__dirname, ...).
The test container will be populated the build output artifacts as well as all the files from the test directory.
Then, the entry-point will be executed. If it fails, the test failed.
The last step of the pipeline is to publish your artifacts to one or more package managers. Delivlib is shipped with a bunch of built-in publishing tasks, but you could add your own if you like.
To add a publishing target to your pipeline, you can either use the
pipeline.addPublish(publisher) method or one of the built-in
pipeline.publishToXxx methods. The first option is useful if you wish to
define your own publisher, which is class the implements the
delivlib.IPublisher interface.
Built-in publishers are designed to be idempotent: if the artifacts version is
already published to the package manager, the publisher will succeed. This
means that in order to publish a new version, all you need to do is bump the
version of your package artifact (e.g. change package.json) and the publisher
will kick in.
You can use the dryRun: true option when creating a publisher to tell the
publisher to do as much as it can without actually making the package publicly
available. This is useful for testing.
The following sections describe how to use each one of the built-in publishers.
The method pipeline.publishToNpm will add a publisher to your pipeline which
can publish JavaScript modules to npmjs.
The publisher will search for js/*.tgz in your build artifacts and will npm publish each of them.
To create npm tarballs, you can use npm pack as part of your build and emit
them to the js/ directory in your build artifacts. The version of the module
is deduced from the name of the tarball.
To use this publisher, you will first need to store an npm.js publishing token in AWS Secrets Manager and supply the secret ARN when you add the publisher.
pipeline.publishToNpm({
npmTokenSecret: { secretArn: 'my-npm-token-secret-arn' }
});This publisher can publish .NET NuGet packages to nuget.org.
The publisher will search dotnet/**/*.nuget in your build artifacts and will
publish each package to NuGet. To create .nupkg files, see Creating NuGet
Packages.
Make sure you output the artifacts under the dotnet/ directory.
To use this publisher, you will first need to store a NuGet API Key with "Push" permissions in AWS Secrets Manager and supply the secret ARN when you add the publisher.
Use pipeline.publishToNuGet will add a publisher to your pipeline:
pipeline.publishToNuGet({
nugetApiKeySecret: { secretArn: 'my-nuget-token-secret-arn' }
});Important: Limitations in the mono tools restrict the hash algorithms that
can be used in the signature to SHA-1. This limitation will be removed in the
future.
You can enable digital signatures for the .dll files enclosed in your NuGet
packages. In order to do so, you need to procure a Code-Signing Certificate
(also known as a Software Publisher Certificate, or SPC). If you don't have one
yet, you can refer to
Obtaining a new Code Signing Certificate
for a way to create a new certificate entirely in the Cloud.
In order to enable code signature, change the way the NuGet publisher is added
by adding an ICodeSigningCertificate for the codeSign key (it could be a
CodeSigningCertificate construct, or you may bring your own implementation if
you wish to use a pre-exis
No open issues yet, or sync has not completed.