CI/CD Workflows with Buddy

About Buddy

Buddy is the CI/CD platform that is used to build releases and deploy them to our live site on Pantheon.

This section describes in details the projects and pipelines that are used for this process.

Active Project

The 2022 utk-libraries project is the current project used in Buddy for deploying and updating our WordPress instance.

The project is based on the utk-libraries Github repository. A deploy key is defined in this repository in the Settings > Deploy Keys section. This key allows Buddy to read and push releases to this repository. Without this key, deployment will not work. Also, we use deploy keys so that this process is not tied to any one individual.

The Project Settings section in Buddy generated and defines the key that is used above in GitHub.

Other keys may be defined in the Variables, Keys, Assets section. If they are relevant, they will be defined below. This section also defines some global environmental variables including:

  1. $major: contains the base number for semantic versioning.

  2. $minor: contains the second number for semantic versioning.

  3. $patch: the last digit of the semantic versioning number for releases.

  4. $theme: the path that stores the location of the theme directory.

Pipelines

CI/CD Workflows in this project are controlled by its pipeline. There are currently two which are used in tandem and described below.

Because they are used in tandem, you CANNOT simply import one yml without the other. If you do this, you’ll need to tie them together in the end. That is also described below.

Current versions of the yamls are kept in a private repository. Sanitized versions of the pipelines are also stored here for documentation sake.

2022 Build, Test and Deploy Master Branch

This is the primary pipeline used for build and deploy. This pipeline is split into 10 separate steps.

Step 1: Install / Test

The first step of the pipeline is to build an environment where we can install and test our application with composer. To do this, we use the php docker image from docker hub that is tagged with the 7.2-stretch tag.

  - action: "Install/Test"
    type: "BUILD"
    working_directory: "/buddy/utk-libraries"
    docker_image_name: "library/php"
    docker_image_tag: "7.2-stretch"

Using this image, we build and run a container that:

  1. Turns off the memory limit for PHP

  2. Installs required Debian packages for PHP and composer

  3. Downloads and installs compose

  4. Configures and executes PHP extensions

These steps are reflected in the corresponding yaml:

    setup_commands:
      - "# Do not set a PHP Memory Limit"
      - "echo 'memory_limit=-1' >> /usr/local/etc/php/conf.d/buddy.ini"
      - ""
      - "# Install Packages from Apt"
      - "apt-get update && apt-get install -y git zip libfreetype6-dev libjpeg62-turbo-dev libpng-dev unzip zlib1g-dev"
      - ""
      - "# Install Composer"
      - "curl -sS https://getcomposer.org/installer | php -- --install-dir=/usr/local/bin --filename=composer"
      - ""
      - "# Configure php ext gd"
      - "docker-php-ext-configure gd --with-freetype-dir=/usr/include/ --with-jpeg-dir=/usr/include/ --with-png-dir=/usr/include/"
      - "docker-php-ext-install gd"
      - ""
      - "# Configure php ext zip"
      - "docker-php-ext-install zip"
      - ""
      - "# Configure php ext pdo_mysql"
      - "docker-php-ext-configure pdo_mysql --with-pdo-mysql"
      - "docker-php-ext-install pdo_mysql"

Once the container is built, composer is run inside of the utk-libraries git repository in the working directory:

  - action: "Install/Test"
    type: "BUILD"
    working_directory: "/buddy/utk-libraries"
    docker_image_name: "library/php"
    docker_image_tag: "7.2-stretch"
    execute_commands:
    - "cd /buddy/utk-libraries/$theme"
    - "composer clear-cache"
    - "composer install -vvv"

Step 2: Build React Apps & Sage 9 theme

Next, we build out the React components of the website. To do this, we use the node docker image from docker hub that is tagged with the 10 tag.

  - action: "Build React Apps & Sage 9 theme"
    type: "BUILD"
    working_directory: "/buddy/utk-libraries"
    docker_image_name: "library/node"
    docker_image_tag: "10"

We then setup the environment by installing gulp and grunt-cli.

    setup_commands:
    - "npm install -g gulp grunt-cli"

Finally, we build out the individual react applications and ultimately our theme based on sage:

    execute_commands:
    - "# build react apps"
    - "cd $react"
    - "cd accordion && yarn && yarn build"
    - "cd ../exhibits && yarn && yarn build"
    - "cd ../header && yarn && yarn build"
    - "cd ../panel && yarn && yarn build"
    - "cd ../social && yarn && yarn build"
    - "cd ../subsite-menu && yarn && yarn build"
    - ""
    - "# build theme"
    - "cd /buddy/utk-libraries/$theme"
    - "yarn install"
    - "yarn"
    - "yarn build:production-clean"
    - ""
    - "rm -rf node_modules"

Our action also defines the react path as a variable:

    variables:
    - key: "react"
      value: "wp-content/mu-plugins/utk_library/src/React"
      type: "VAR"

Step 3: Set Release Version

Next, we build a container to build our release. To do this, we use the ubuntu docker image from docker hub with the 18.04 tag.

  - action: "Set patch version"
    type: "BUILD"
    working_directory: "/buddy/utk-libraries"
    docker_image_name: "library/ubuntu"
    docker_image_tag: "18.04"

We then install git in this new container.

    setup_commands:
    - "apt-get update && apt-get -y install git"

And ultimately tag our new release by taking our global variables used for semantic versioning and adding a 1 to the patch number:

    execute_commands:
    - "# Set Path to Current Value Plus 1"
    - "((patch=patch+1))"
    - ""
    - "# Concatenate Semantic Version Number to $major.$minor.$patch"
    - "latest_version=$major\".\"$minor\".\"$patch"
    - ""
    - "# Create New Tag Equal to Current Semantic Versioning Number"
    - "git tag \"$latest_version\""

Like previous steps, all of this runs against our working directory that holds a copy of our github repository with the code base:

  - action: "Set patch version"
    type: "BUILD"
    working_directory: "/buddy/utk-libraries"
    docker_image_name: "library/ubuntu"
    docker_image_tag: "18.04"

Step 4: Push Release to Github

The following step is something that Mark believes should be reworked.

In this step, we push the tagged release we just built to GitHub, but oddly we do it in a new container that uses the same image that we had in the previous step.

  - action: "Push patch release"
    type: "BUILD"
    working_directory: "/buddy/utk-libraries"
    docker_image_name: "library/ubuntu"
    docker_image_tag: "18.04"

From a performance point of view, this definitely seems questionable. Yes, this is only an issue when changes are made or if you are building for the first time, but it still seems so wrong. In Mark’s the execute commands should be combined into the previous step.

The setup rules are exactly the same as the previous step.

    setup_commands:
    - "apt-get update && apt-get -y install git"

The steps that are executed simply add the SSH key defined by the utk-libraries deploy keys settings to the authorized_keys file in the user directory of this container so that the tagged released can be pushed. We then create a fake git config file for Buddy to more easily see the changes that are made by Buddy rather than an individual. Finally, we set our origin to our Github repo and push the tagged release.

    execute_commands:
    - "# ssh & git clone"
    - "echo -e 'ssh-rsa ###REPLACE_HASH### utk-libraries-1 Key\\n' >> ~/.ssh/authorized_keys"
    - "chmod 0600 ~/.ssh/authorized_keys"
    - "ssh-keyscan github.com >> /root/.ssh/known_hosts"
    - ""
    - "# Make Git User Buddy Bot"
    - "git config --global user.email \"dlcontact@utk.edu\""
    - "git config --global user.name \"UTK Buddy Bot\""
    - ""
    - "# Set remote to UTK GitHub and Push our new Tagged Release"
    - "git remote set-url origin git@github.com:utkdigitalinitiatives/utk-libraries.git"
    - "git push origin master --tags"

Step 5 and 6: Find and Replace for our Sage-based theme

This section describes 2 important steps that is not clear from reading the hashed yaml. It is critical that both of these run so that our Sage-based theme builds.

In the first step, we open the .gitignore file in our Sage-based theme directory and replace several lines so that they are no longer ignored.

  - action: "Find & replace"
    type: "REPLACE"
    local_path: "$theme/.gitignore"
    replacements:
    - replace_from: "dist/*"
      replace_to: ""
    - replace_from: "dist"
      replace_to: ""
    - replace_from: "vendor/*"
      replace_to: ""
    - replace_from: "vendor"
      replace_to: ""

In the next step, we take similar actions versus the wp-content/mu-plugins/utk_library/assets/.gitignore:

  - action: "Find & replace"
    type: "REPLACE"
    local_path: "wp-content/mu-plugins/utk_library/assets/.gitignore"
    replacements:
    - replace_from: "react"
      replace_to: ""

The reason for this is that Pantheon needs these files in order to build correctly. This becomes more clear in the next steps.

Step 7: Push to Pantheon Master

NOTE: In order to understand fully what’s going on in this step, you need to import the yaml into a buddy pipeline. If not, this will look like wizardry.

This step, takes the output of all preceding actions, commits them, and force pushes all files to the master branch in a git repository on Pantheon. The repository’s location is highlighted below but obscured for security purposes:

  - action: "Push to Pantheon utk-libraries (master)"
    type: "PUSH"
    git_auth_mode: "ENV_KEY"
    env_key: "secure!###REPLACED_VALUE###"
    push_url: "ssh://codeserver.dev.###REPLACED LOCATION###:2222/~/repository.git"
    comment: "Build $BUDDY_EXECUTION_ID of $BUDDY_EXECUTION_REVISION_MESSAGE  $BUDDY_EXECUTION_REVISION"
    trigger_conditions:
    - trigger_condition: "ALWAYS"

As stated above, by looking at the raw yaml, there is a lot of magic here. You can only fully appreciate what is happening by importing this to a Buddy pipeline so that the action is decrypted. Notice, that there is an environmental variable that is hashed here:

  - action: "Push to Pantheon utk-libraries (master)"
    type: "PUSH"
    git_auth_mode: "ENV_KEY"
    env_key: "secure!###REPLACED_VALUE###"
    push_url: "ssh://codeserver.dev.###REPLACED LOCATION###:2222/~/repository.git"
    comment: "Build $BUDDY_EXECUTION_ID of $BUDDY_EXECUTION_REVISION_MESSAGE  $BUDDY_EXECUTION_REVISION"
    trigger_conditions:
    - trigger_condition: "ALWAYS"

When you look at this in Buddy, this variable is decrypted to this:

echo -e 'ssh-rsa ###REPLACED_HASH### utk-libraries-1 Key\n' >> ~/.ssh/authorized_keys
chmod 0600 ~/.ssh/authorized_keys

So that env_key includes the SSH key that allows Buddy to write to Pantheon.

It’s not clear how this next part is hashed in the yaml, but there is a CRITICAL instruction also captured here that relates to the find and replace steps. This too only becomes clear on import to Buddy:

Buddy rule that captures all changes to a fresh commit pre-push

Now we can start to better understand what is happening here. In this step, we are:

  1. Adding an SSH key to Buddy that allows us to write to Pantheon

  2. Committing all files that have been created or modified by previous actions (Including those files that were originally ignored)

  3. Force pushing the output to the master branch on Pantheon

Step 8: Push to Pantheon Sandbox

NOTE: In order to understand fully what’s going on in this step, you need to import the yaml into a buddy pipeline. If not, this will look like wizardry.

This step, takes the output of all preceding actions, commits them, and force pushes all files to the sandbox branch in a git repository on Pantheon. The repository’s location is highlighted below but obscured for security purposes:

  - action: "Push to Pantheon utk-libraries (sandbox)"
    type: "PUSH"
    git_auth_mode: "ENV_KEY"
    env_key: "secure!###REPLACED_VALUE###"
    target_branch: "sandbox"
    push_url: "ssh://codeserver.dev.###REPLACED_LOCATION###.drush.in:2222/~/repository.git"
    comment: "Build $BUDDY_EXECUTION_ID of $BUDDY_EXECUTION_REVISION_MESSAGE  $BUDDY_EXECUTION_REVISION"
    trigger_conditions:
    - trigger_condition: "ALWAYS"

As stated above, by looking at the raw yaml, there is a lot of magic here. You can only fully appreciate what is happening by importing this to a Buddy pipeline so that the action is decrypted. Notice, that there is an environmental variable that is hashed here:

  - action: "Push to Pantheon utk-libraries (sandbox)"
    type: "PUSH"
    git_auth_mode: "ENV_KEY"
    env_key: "secure!###REPLACED_VALUE###"
    target_branch: "sandbox"
    push_url: "ssh://codeserver.dev.###REPLACED_LOCATION###.drush.in:2222/~/repository.git"
    comment: "Build $BUDDY_EXECUTION_ID of $BUDDY_EXECUTION_REVISION_MESSAGE  $BUDDY_EXECUTION_REVISION"
    trigger_conditions:
    - trigger_condition: "ALWAYS"

When you look at this in Buddy, this variable is decrypted to this:

echo -e 'ssh-rsa ###REPLACED_HASH### utk-libraries-1 Key\n' >> ~/.ssh/authorized_keys
chmod 0600 ~/.ssh/authorized_keys

So that env_key includes the SSH key that allows Buddy to write to Pantheon.

It’s not clear how this next part is hashed in the yaml, but there is a CRITICAL instruction also captured here that relates to the find and replace steps. This too only becomes clear on import to Buddy:

Buddy rule that captures all changes to a fresh commit pre-push

Now we can start to better understand what is happening here. In this step, we are:

  1. Adding an SSH key to Buddy that allows us to write to Pantheon

  2. Committing all files that have been created or modified by previous actions (Including those files that were originally ignored)

  3. Force pushing the output to the sandbox branch on Pantheon

Rationale

If you’re reading this, you may be wondering: why do we push a master and sandbox branch to Pantheon?

The master branch described in step 7 is the primary dev environment. It has its own database and is what we normally look at when we are testing. If we pull the database from test to dev, it updates this.

The sandbox branch described here exists because some stakeholders wanted a “development” area to try new things and we needed a way to insure that this wasn’t overwritten when the database was updated. As far as Mark knows, this was only ever used by Teaching and Learning.

Step 9: Push to Github Upstream

NOTE: Like the two preceding steps, in order to understand fully what’s going on in this step, you need to import the yaml into a buddy pipeline. If not, this will look like wizardry.

This step, takes the output of all preceding actions, commits them, and force pushes all files to the upstream repository in Github.

  - action: "Push to UT Libraries Pantheon upstream"
    type: "PUSH"
    push_tags: true
    git_auth_mode: "ENV_KEY"
    env_key: "secure!###REPLACED_VALUE###"
    target_branch: "master"
    push_url: "git@github.com:utkdigitalinitiatives/utk-libraries-upstream.git"
    comment: "Build $BUDDY_EXECUTION_ID of $BUDDY_EXECUTION_REVISION_MESSAGE  $BUDDY_EXECUTION_REVISION"

Like the two preceding steps, the yaml includes a hashed key that includes instructions for authorizing write access to the Github repo and instructions to commit files from all preceding actions that are only readable once this yaml has been imported into a Buddy pipeline:

echo -e 'ssh-rsa ###REPLACED_HASH### utk-libraries-1 Key\n' >> ~/.ssh/authorized_keys
chmod 0600 ~/.ssh/authorized_keys
Buddy rule that captures all changes to a fresh commit pre-push

This action makes some things clear:

  1. Deployment to Pantheon happens in Buddy prior to deployment to Github

  2. We keep a copy of things on Pantheon in Github for easy diffing between the original code repo and the deployment repo.

Step 10: Apply Updates

This last action is the reason why we get an error when we try to import this yaml in Buddy. The error message makes you think the import fails, but actually it works except that this action doesn’t come along. You have to do that yourself.

This action calls an entirely different pipeline and forces it to run:

  - action: "Run 2022 utk-libraries/Apply Upstream Updates"
    type: "RUN_NEXT_PIPELINE"
    comment: "Triggered by $BUDDY_PIPELINE_NAME execution #$BUDDY_EXECUTION_ID"
    revision: "INHERIT"
    next_project_name: "utk-libraries-1"
    next_pipeline_name: "Apply Upstream Updates"

This pipeline isnt’ described in detail yet, but should be.

Essentially, the action builds a PHP environment so that Pantheon’s terminus-installer can be installed.

Once installed, we login to the environment in Pantheon, clear cache on volumes, and apply updates to volumes.

execute_commands:
    - "terminus auth:login --machine-token=$machine_token"
    - ""
    - "terminus site:upstream:clear-cache utk-volumes"
    - "terminus upstream:updates:apply --updatedb --accept-upstream utk-volumes.dev"

Mark is still not sure why this is important.