CI/CD for Infrastructure as Code pt 1: Defining a Workflow for GitHub Actions (DevOps the Hard Way series)
In this section of the series, we will use GitHub Actions to automatically deploy a Kubernetes cluster to AWS, whenever changes to our Terraform configuration files are pushed to GitHub.
This section builds on previous posts in the series, and assumes that all three of the following are already up and running:
- An AWS VPC, as covered in step 2 here
- A Terraform S3 backend, as covered here
- An AWS Kubernetes (EKS) cluster, as covered here
CI/CD is a DevOps concept focused on streamlining and accelerating the software development lifecycle. The goal is to establish a “pipeline” through which software can be frequently built, tested, merged, released, and sometimes even automatically deployed to a testing or production environment.
The GitHub Actions workflow (pipeline)
A GitHub Actions CI/CD pipeline is defined via a workflow, which is a list of declarations made using the GitHub Actions workflow syntax and the YAML file format.
In this post we will examine the workflow file provided by our scenario, section-by-section. A brief definition will follow each time a new keyword from the GitHub Actions workflow syntax is introduced.
main.yml
1
2
3
4
jobs:
build:
runs-on: ubuntu-latest
steps:
The file begins with the jobs, <job_id>, runs-on, and steps workflow syntax keys.
A workflow run is made up of one or more jobs, which run in parallel by default.
jobs.<job_id>
[G]ive(s) your job a unique identifier.
By using the jobs key, the first line begins a list of jobs that make up this workflow. In our example, the workflow consists of one job, which has a <job_id> of build.
This <job_id> could be easily be changed to something else, but in our scenario an ID of build makes sense, since the job’s purpose is to automatically build an EKS cluster on AWS.
The remainder of this workflow file is spent declaring the specifics of this build job.
jobs.<job_id>.runs-on
[D]efine(s) the type of machine to run the job on.
GitHub Actions depends on a runner machine to provide the compute resources that actually run a workflow.
The runs-on line under the build job declares that build will use a GitHub-hosted runner machine to run the various steps it contains. By specifying ubuntu-latest as the value for runs-on, this workflow file declares that the runner for the build job should use the latest stable Linux OS release offered by GitHub Actions.
jobs.<job_id>.steps
A job contains a sequence of tasks called steps
Each of the steps for a given <job_id> will run as its own process on the runner machine. The steps defined in the build job are:
step: Checkout
(build job→steps→Checkout)
1
2
3
- name: Checkout
uses: actions/checkout@v2
This first step of the build job introduces a pair of new workflow syntax keys:
jobs.<job_id>.steps[*].nameA name for your step to display on GitHub.
jobs.<job_id>.steps[*].usesSelects an action to run as part of a step in your job. An action is a reusable unit of code…. We strongly recommend that you include the version of the action you are using
Actions are either JavaScript files or Docker containers.
As specified by the name value, this step is named Checkout.
The uses line declares that v2 of the checkout action should run in this step. A version of the action is explicitly specified, in order to help prevent the overall workflow from breaking when major development changes are made to the checkout action.
As noted in its own GitHub README, the checkout action’s purpose is simply to check out our repository’s code to the action’s runner. Doing so makes our code accessible to the runner, and to other steps in the build job.
This should make sense, since the runner will need to read our infrastructure as code in order to automate the provisioning of real-world AWS infrastructure.
step: Configure AWS credentials
(build job→steps→Configure AWS credentials)
1
2
3
4
5
6
7
- name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials@v1
with:
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
aws-region: us-west-1
The next step in the workflow is named Configure AWS credentials. The uses line declares that in this step, the build job’s runner should run v1 of the aws-actions/configure-aws-credentials action.
Unsurprisingly, the purpose of the configure-aws-credentialsaction is to configure AWS credentials, so that its runner can use those credentials to authenticate to AWS. Without such authentication, GitHub Actions would be unauthorized to build any AWS infrastructure for us.
Notice how the with keyword is introduced in this job step:
jobs.<job_id>.steps[*].with[T]he input parameters defined by the action.
Input parameters are set as environment variables.
In this particular example, the values assigned to with are placeholders, referencing pre-stored AWS credentials that will be will be made available to the build job’s runner at runtime.
A few things to keep in mind when working with this step:
- It will require additional setup before it can successfully run, because:
- a) A valid AWS access key with appropriate permissions will need to be created first if not already available, and
- b) That access key’s ID and value will need to be saved as Action secrets within the repository where the workflow will run.
-
Using access keys is not considered AWS best practice. AWS recommends using temporary IAM credentials to access their services instead. The process of setting these up for GitHub Actions is detailed here.
- The value for the input parameter
aws-regionwas hard-coded by the author of DevOps the Hard Way and might need to be changed to match whichever region was chosen earlier during the initial EKS cluster deployment.
step: Setup Terraform
(build job→steps→Setup Terraform)
1
2
3
- name: Setup Terraform
uses: hashicorp/setup-terraform@v1
The next step of the build job is named Setup Terraform, and it runs v1 of the setup-terraform action from Hashicorp. As noted in its GitHub repository, this action installs the Terraform CLI to its runner. As a result, terraform commands are made available for other job steps to use.
This is important because the build job will use the Terraform CLI behind the scenes while automating AWS infrastructure creation.
step: Terraform Init
(build job→steps→Terraform Init)
1
2
3
4
5
6
7
- name: Terraform Init
working-directory: Terraform-AWS-Services/elasticsearch/elasticsearch_configuration/
run: |
terraform init \
-backend-config "bucket=terraform-states-monitoring-platform" \
-backend-config "key=elasticsearch-terraform.tfstate"
terraform workspace new dev || terraform workspace select dev
In this step, which is named Terraform Init, two new workflow keys are introduced:
jobs.<job_id>.steps[*].working-directory[S]pecify the working directory of where to run the command
jobs.<job_id>.steps[*].runRuns command-line programs
Because the Terraform CLI gets installed to the runner by the Setup Terraform step, this step, named Terraform Init, can directly run a terraform CLI command. The command is run from the specified working-directory on the runner.
The core command this step runs is the same terraform init command we’ve run manually multiple times in the Terraform section of this series of blog posts; it downloads and installs the AWS Terraform provider locally to enable direct communication between the Terraform CLI and the AWS API.
step: Terraform Format
(build job→steps→Terraform Format)
1
2
3
4
- name: Terraform Format
working-directory: Terraform-AWS-Services/elasticsearch/elasticsearch_configuration/
run: terraform fmt
This is the first time the terraform fmt command has appeared in this series of blog posts. Per the Terraform CLI documentation:
The
terraform fmtcommand is used to rewrite Terraform configuration files to a canonical format and style.
This step runs the terraform fmt command, taking the Terraform configuration files checked out to the runner from our repository and “cleaning them up,” making any recommended formatting and style fixes.
step: Terraform Plan
(build job→steps→Terraform Plan)
1
2
3
4
- name: Terraform Plan
working-directory: Terraform-AWS-Services/elasticsearch/elasticsearch_configuration/
run: terraform plan -var="environment=development" -var="elasticsearch_password=$"
The terraform plan command is familiar by now in this series of posts; it verifies what changes would be made to existing infrastructure after applying a given Terraform configuration.
In this case, the Terraform Plan step of the build job will check the current state of the real-world AWS infrastructure against what has been declared as code in our repository, and note any differences to be reconciled.
step: Terraform Apply
(build job→steps→Terraform Apply)
1
2
3
- name: Terraform Apply
working-directory: Terraform-AWS-Services/elasticsearch/elasticsearch_configuration/
run: terraform apply -var="environment=development" -var="elasticsearch_password=$" -auto-approve
The terraform apply command is also already familiar; it is the command that actually applies a Terraform configuration to make real-world AWS infrastructure changes.
If our infrastructure as code declares something different from what actually exists in our AWS cloud environment, then the terraform apply command will try to make the real-world changes required to match the declared Terraform config.
It is this Terraform Apply step of the build job that will perform any actual infrastructure changes on AWS.
That is the essence of what we are trying to accomplish with CI/CD in our scenario: whenever we push IaC changes to our repository, we want GitHub Actions to read our updated infrastructure code and automatically make any required changes to real-world cloud infrastructure.
In the next post, we will configure and test the Actions workflow that has been detailed throughout this post, against a live GitHub repository.