Avatar
Dr. Florian Rappl is Solution Architect for IoT and distributed web applications at smapiot. His main interest lies in the creation of innovative architectures that scale in development and usage. He won several prizes for his work and is a Microsoft MVP in the area of development technologies. He regularly blogs and writes articles for popular magazines and websites in the web development space.

Azure DevOps is Microsoft's solution for managing the full software development life cycle (SDLC). It contains everything a team needs to build outstanding products. This includes an issue tracker, dashboards and reporting, source control including an advanced editor, and other features, artifacts, test management, and more.

One crucial element of Azure DevOps is a system to manage builds and deployments. The system to manage builds of software and their releases is called Azure Pipelines.

While Azure Pipelines was initially set up as a GUI-driven solution, it's also developed into a code-driven solution over the years. With the most recent updates Azure Pipelines can now be completely controlled in a code-first way. This enables Infrastructure-as-Code (IaC) solutions that already start with a maintainable continuous integration/continuous delivery (CI/CD) system completely based on YAML files.

But let’s step back a bit for now and start with the basics first.

Build Versus Release

Originally, Azure Pipelines was divided into the following concepts:

  • Build jobs
  • Release definitions
  • Task groups
  • Libraries

Only libraries and build jobs are necessary to define truly reusable and flexible pipelines these days.

Let’s first define some terms:

  • Build jobs are a way of defining how software is produced. As a simple example you need to run a compiler on source code to produce an executable.
  • Triggers define actions that start jobs. Common examples include a push to a certain branch ("branch trigger") or a certain point in time ("schedule trigger").
  • Artifacts are the files created by a build job. That could be a simple .exe file or an .msi installer file.

A typical build job may contain everything to create and verify artifacts. This includes building the source code and running defined tests for it. A build job should not deploy anything directly, but could (and should) publish artifacts. These artifacts may then be used in a release definition, which contains different triggers.

The distinction between build job and release release definition allows reusing the build job without any changes for, say, a pull request validation. In contrast, the release definition may only be used when certain branches change. Here, an actual release will be triggered.

Below you see how a release definition with multiple stages could look in the classic approach.

The split of builds and releases is also useful for things besides pull request validation. This allows a quick and easy rollback mechanism. Since the build already happened, you retain build artifacts that can be released independently of any long-running build jobs.

Modern Pipelines

Modern pipelines are no longer created in graphical editors (even though this can still be done). Instead, a special file called azure-pipelines.yml is added to the root of a code repository. If this file is found, then a pipeline is automatically created (or updated).

The format of this file is YAML. To make the format human friendly, the significant parts of the language have been designed to make it very readable. Consequently, YAML is a language that relies on whitespaces. Most CI/CD services let their automated code pipelines work with YAML.

An example of a YAML declaration in Azure DevOps looks as follows:

pool:
  vmImage: 'ubuntu-16.04'

steps:
- task: NodeTool@0
  inputs:
	versionSpec: $(node_version)

- script: npm install

As with JSON, for YAML there is also a schema to define what keys are expected or allowed, and what kinds of values they can contain.

Before we dive into the technical side of the Azure Pipelines YAML schema, we should have a look at the hierarchy defined by a pipeline YAML:

  • Pipeline
    • Stage A
      • Job 1
        • Step 1.1
        • Step 1.2
      • Job 2
    • Stage B

According to the official specification, the following terms are defined:

  • A pipeline is one or more stages that describe a CI/CD process. Stages are the major divisions in a pipeline. The stages "build this app," "run these tests," and "deploy to preproduction" are good examples.
  • A stage is one or more jobs, which are units of work assignable to the same machine. You can arrange both stages and jobs into dependency graphs. Examples include "run this stage before that one" and "this job depends on the output of that job."
  • A job is a linear series of steps. Steps can be tasks, scripts, or references to external templates.

A pipeline does not need to make use of the full hierarchy. Instead, if only a series of steps is required, then there is no need to define jobs or stages.

Coming back to the topic of the Azure Pipelines YAML schema, we see the following key regions:

  • trigger to set if the pipeline should be running when branches changed
  • pr to set if the pipeline should be running for pull requests
  • schedules to set if the pipeline should be running at fixed times
  • pool to define where the build agent should be running
  • resources to define what other resources your pipeline needs to consume
  • variables to define variables and their values
  • parameters to define input parameters for the pipeline template
  • stages to define the stages (consists of jobs)
  • jobs to define the different jobs (consists of steps)
  • steps to define what steps to take
  • template to allow referencing other (YAML) files

There are a couple more, but these give us the most important parts to get everything done. For the full spectrum, visit the official documentation.

Maintainable Pipelines

The key for writing maintainable Azure Pipelines YAML files lies in the resources specifier. It allows you to also consider other repositories as input artifacts. This way you can "just obtain" another repository, which brings in additional files.

In practice, this all starts with a file like the following:

# azure-pipelines.yml

resources:
  repositories:
  - repository: pipeline
    type: git
    name: my-pipelines

trigger:
- develop
- release
- master

stages:
- template: deploy-service.yml@pipeline
  parameters:
    serviceName: my-test-service

Here you refer to a repository named my-pipelines that later on can be called by referencing pipeline. Note that this repository must be available in the same project as the pipeline on Azure DevOps.

Instead of specifying the different stages, you refer to template. This allows you to define the full content of the stages in another file. You pick the deploy-service.yml file coming from the pipeline resource. As you defined it, the pipeline resource refers to your my-pipelines repository.

Looking at the file, you won’t see any big surprises. A standard YAML file, right? But wait – you define the parameters in here, too:

# my-pipelines/deploy-service.yml

parameters:
  pool:
    name: my-pool
    demands:
    - TYPE -equals node
    - ID -equals 0
  serviceName: ''

stages:
- stage: Build
  pool: ${{ parameters.pool }}
  jobs:
  - template: /build/npm-docker-build.yml

- template: /deploy/deploy-stages.yml
  parameters:
    pool: ${{ parameters.pool }}
    serviceName: ${{ parameters.serviceName }}

The parameters give you a convenient way to define what needs to be known (but flexible) inside the YAML definition. By using the ${{ }} replacement syntax, you can refer to the given value of these parameters later.

The definition of the pool parameter is interesting. To avoid any misuse here, you fix the value using a demands section. The default value is given via name.

Other than that, you see that template keys are used again to fill some other sub-sections. While the build stage is fully defined (except the contained jobs), the additional publishing stages are all found in a different file at /deploy/deploy-stages.yml. Notice that you did not define a resource here, nor did you reference the file from a resource.

The reason is simple: this file is still in the same repository as the referrer deploy-service.yml.

Let's take a look at another referenced file here: the definition of the build job via npm-docker-build.yml.

The file is pretty much straightforward; it just contains a number of steps. Every step is, for reusability reasons, coming from a different YAML file.

# my-pipelines/build/npm-docker-build.yml

jobs:
- job: NPM.Docker.Build
  displayName: NPM Docker Build
  variables:
	- group: my-keyvault-vars
	- template: /vars/common.yml
  steps:
	- template: /build/tasks/npm-build.yml
	- template: /build/tasks/npm-test.yml
	- template: /build/tasks/keyvault-verify.yml
	- template: /build/tasks/docker-build.yml
	- template: /build/tasks/docker-push.yml

At this point, the composition pattern is quite clear. The given pipeline-as-code description is highly efficient and — thanks to Git as a version control system — resilient against things like switching Azure DevOps organizations.

Defining variables also does not need to be done via libraries. Instead, you can leverage features such as defining all the variables via YAML, too.

In the code above, you referred to /vars/common.yml for the whole variables section. Let's see how this looks in practice:

# my-pipelines/vars/common.yml

parameters:
  additionalVariables: {}

variables:
  # Build trigger for forced builds
  ${{ if and(eq(variables['Build.Reason'], 'Manual'), notIn(variables['Build.SourceBranch'], 'refs/heads/develop', 'refs/heads/release', 'refs/heads/master')) }}:
	VERSION: '-manual'
	ENV_SUFFIX: '-test'
	ENV_NAME: 'test'

  # Build trigger for pull request
  ${{ if eq(variables['Build.Reason'], 'PullRequest') }}:
	VERSION: '-pr'
	ENV_SUFFIX: '-test'
	ENV_NAME: 'test'

  # Build trigger when branch "release" is merged
  ${{ if eq(variables['Build.SourceBranch'], 'refs/heads/release') }}:
	VERSION: '-beta'
	ENV_SUFFIX: '-stage'
	ENV_NAME: 'stage'

  # Define some standard variables
  IMAGE_NAME: '$(Build.Repository.Name)'
  IMAGE_TAG: '$(Build.BuildId)$(VERSION)'

  # Insert additional variables
  ${{ insert }}: ${{ parameters.additionalVariables }}

The additionalVariables parameter allows you to bring in additional variables, if wanted. In addition, the code above demonstrates how to

  • include some fixed variables in case of a specific (manual) build
  • include some fixed variables for a pull request
  • include some fixed variables for a predefined branch
  • include some variables based on existing variables (or in generic on parameters, too)
  • include the additional variables via a custom spread operation using ${{ insert }}

The great thing is that this now forms a wonderful round circle regarding builds and releases. Whereas the screen was formerly split (as mentioned for good reason), it is now one.

You can see why the release is where it is and what has been done to actually create the artifact that was then pushed to the different release stages.

Approval Gates

Finally, in these stages you want to be able to define environments and approval steps, too.

You start by defining an environment. Ideally, this is a virtual machine or an Azure Kubernetes Services cluster. Otherwise, you can use an "empty" environment where you do not add any resources.

Now you only need to reference the environment. This can be done via the environment's name. If a defined environment name is not found, then an empty environment is automatically created for you.

The following YAML shows a fragment for deploying an Helm chart to Azure Kubernetes Services:

parameters:
  serviceName: ''
  namespace: 'test'
  envName: 'test'

jobs:
  - deployment: 'HelmDeploy${{ parameters.envName }}'
    displayName: 'Helm Deploy'
    variables:
    - template: /vars/common.yml
      parameters:
        serviceName: ${{ parameters.serviceName }}
	   environment: ${{ parameters.envName }}
    strategy:
  	runOnce:
    	deploy:
       steps:
       - checkout: self
       - script: helm repo update
         displayName: 'Helm: Update Helm Charts'
       - task: HelmDeploy@0
         displayName: 'Helm: Deployment'
         inputs:
         	namespace: ${{ parameters.namespace }}
           command: upgrade
           chartName: 'test/my-service-chart'
           releaseName: ${{ parameters.serviceName }}

Through the environment settings the gate approvals are determined without any fiddling in the intrinsic build steps. This is quite nice because it allows general administrators to adjust the approval conditions without touching the code. The separation is clean and straightforward.

Next Steps

There are two points I'd love to leave you with.

The first is that the new YAML-based definition for pipelines allow for defining reusable environments. With the earlier way, only release definitions could define "stages," which were used similar to environments. Reusability was ensured via templates, but due to a number of reasons, this was an insufficient mechanism.

Environments solve this and more. They allow coupling to actual resources, like Azure Kubernetes Services. Using environments, the Azure-hosted cluster resources can be seen and inspected directly. This makes it possible to get all the running pods, their status, and even log file output on a single screen.

The second thing is the ability to automatically create projects using these pipelines. Having the whole pipeline definition in code makes project scaffolding quite simple. Formerly, you needed to create all resources (build jobs, release definitions, and so on) either via some custom tooling, via the command line tooling, or via a dedicated pipeline that accepts input parameters.

All of these ways used the Azure DevOps API.

Even though build jobs and release definitions have been moved out of the equation, you may still want to use these tools to bring in some consistency. This way you can create things like branches, their policies, and coding boilerplate without much trouble.

Focusing on the branch policies, you can use the official Azure CLI extension for Azure DevOps to bring this functionality in.

The command can be as simple as

az repos policy create ./my-policy.json

The unfortunate part here is that the whole payload of the body must be configured via a file (here my-policy.json). Here’s an example of adding a (manual) build job to the branch policies for the master branch:

{
  "isBlocking": true,
  "isDeleted": false,
  "isEnabled": true,
  "revision": 1,
  "settings": {
	"buildDefinitionId": 22,
	"displayName": "Manual Queue Policy",
	"manualQueueOnly": true,
	"queueOnSourceUpdateOnly": false,
	"scope": [
  	{
    	"matchKind": "Exact",
    	"refName": "refs/heads/master",
    	"repositoryId": "the-repo-id"
  	}
	],
	"validDuration": 0
  },
  "type": {
	"displayName": "Build",
	"id": "0609b952-1397-4640-95ec-e00a01b2f659"
  }
}

The build policy ID and most of the settings should remain as-is. The build definition ID must be updated accordingly to refer to the right build ID.

How to work with us

  • Contact us to set up a call.
  • We will analyze your needs and recommend a content contract solution.
  • Sign on with ContentLab.
  • We deliver topic-curated, deeply technical content to you.

To get started, complete the form to the right to schedule a call with us.

Send this to a friend