RL_

Zero-Downtime Deployment: Mastering Release and Rollback

What is a Release and Rollback System, and What Problem Does It Solve?

When you update a live production environment, overwriting active files directly on the server is a major risk. Whether you are deploying a custom WordPress theme or updating a complex backend application, replacing active files can cause broken pages, missing assets, or failed transactions for users who are currently navigating the site. A release and rollback system solves this problem by using an atomic deployment strategy.

Instead of modifying the active folder, your CI/CD pipeline uploads the new code into a uniquely named and versioned directory. Once all the files are safely transferred, a symbolic link (or symlink) instantly updates to point to this new directory. This rapid switch guarantees zero downtime. If a critical bug makes it to production, you can roll back instantly. You simply repoint the symlink to the previous version’s directory on the server, completely bypassing the need to run a reverse deployment or restore from a backup.

Using a .pem Key for AWS Lightsail Deployments

When you provision a server instance on AWS Lightsail or similar cloud platforms, you receive a default .pem (Privacy Enhanced Mail) file. This file contains the RSA private key needed for secure SSH access. To automate your deployments using a GitHub Actions CI/CD pipeline, you must store this raw .pem content securely as a repository secret, such as SSH_KEY. This setup allows the automated runner to authenticate with your server without needing manual password prompts.

How to Set Up the SSH Key on a Linux VPS

For standard Linux VPS environments running web servers like Nginx, setting up key-based authentication requires placing the corresponding public key on your server. Here is the step-by-step process:

  • Generate an SSH key pair locally if you are not using the default .pem file from your cloud provider.
  • Append your public key to the server’s ~/.ssh/authorized_keys file for your specific deployment user.
  • Verify that the .ssh directory has 700 permissions and the authorized_keys file is restricted strictly to 600 permissions.
  • Save the private key content inside your GitHub Repository Secrets so the automated workflow can authenticate securely.

The Release Trigger and GitHub Actions Workflow

The deployment process starts automatically whenever you push a new Git tag matching a specific version format, like v1.0.0, to your repository.

Here is the robust release.yml configuration:

name: Release

on:
  push:
    tags:
      - 'v*.*.*'

permissions:
  contents: read

concurrency:
  group: production-deploy

jobs:
  release:
    name: Build and Deploy
    runs-on: ubuntu-latest

    steps:
      - name: Get Latest Code
        uses: actions/checkout@v7.0.1

      - name: Setup Node.js
        uses: actions/setup-node@v7.0.0
        with:
          node-version: '24'
          cache: 'npm'

      - name: Run Build
        run: |
          npm ci
          npm run build

      - name: Install WP-CLI
        run: |
          curl -O https://raw.githubusercontent.com/wp-cli/builds/gh-pages/phar/wp-cli.phar
          chmod +x wp-cli.phar
          sudo mv wp-cli.phar /usr/local/bin/wp
          # `wp package install` always registers the package's GitHub repo as
          # a Composer VCS source to resolve its tag, even for an
          # already-Packagist-published package — which needs a live GitHub
          # API call and fails under the runner's anonymous rate limit, and
          # a GITHUB_TOKEN doesn't fix it either (Composer's github-oauth
          # validator rejects Actions' newer ghs_-prefixed token format).
          # Installing straight from Packagist with plain Composer avoids
          # the GitHub API entirely.
          # Pinned: dist-archive-command's dev-main and v3.2.x require
          # wp-cli/wp-cli ^2.13/^3.0, neither of which is a released
          # stable core version yet (stable is 2.12.0) — v3.1.0 is the
          # newest tag that still only requires ^2.
          mkdir -p "$HOME/.wp-cli/packages"
          composer require --working-dir="$HOME/.wp-cli/packages" --no-interaction wp-cli/dist-archive-command:3.1.0

      - name: Package Theme
        run: |
          # wp dist-archive reads .distignore to strip dev-only files
          # (node_modules, src, package*.json, etc.) and wraps the result
          # in a single top-level folder — the same layout WordPress
          # expects when installing a theme zip.
          wp dist-archive . ./rehan-lodhi.zip
          mkdir -p ./dist
          unzip -q ./rehan-lodhi.zip -d ./dist
          echo "PACKAGE_DIR=$(find ./dist -mindepth 1 -maxdepth 1 -type d)" >> "$GITHUB_ENV"

      - name: Deploy to Production
        env:
          SSH_KEY: ${{ secrets.SSH_KEY }}
          SSH_USER: ${{ vars.SSH_USER }}
          SSH_HOST: ${{ vars.SSH_HOST }}
          RELEASES_ROOT: ${{ vars.PROD_RELEASES_ROOT }}
          THEME_LINK: ${{ vars.PROD_THEME_LINK }}
          KEEP_RELEASES: ${{ vars.PROD_KEEP_RELEASES }}
        run: |
          # 1. Start the SSH agent and add your private key
          eval $(ssh-agent -s)
          echo "$SSH_KEY" | tr -d '\r' | ssh-add -

          # 2. Create the .ssh directory just in case
          mkdir -p ~/.ssh
          chmod 700 ~/.ssh

          SSH_TARGET="${SSH_USER}@${SSH_HOST}"
          RELEASE_DIR="${RELEASES_ROOT}/${GITHUB_REF_NAME}"

          # 3. Rsync the extracted, already-filtered package into its own
          #    tag-named release directory — never into the live theme path
          #    directly, so the site keeps serving the previous release
          #    until the symlink flips below.
          ssh -o StrictHostKeyChecking=no "${SSH_TARGET}" "mkdir -p '${RELEASE_DIR}'"
          rsync -avz --delete \
            -e "ssh -o StrictHostKeyChecking=no" \
            "${PACKAGE_DIR}/" "${SSH_TARGET}:${RELEASE_DIR}/"

          # 4. Atomically point wp-content/themes/rehan-lodhi at the new
          #    release, then prune old releases beyond KEEP_RELEASES —
          #    but never the one currently live, so a rollback target
          #    can't be pruned out from under a manual re-point.
          ssh -o StrictHostKeyChecking=no "${SSH_TARGET}" bash -s <<EOF
            set -euo pipefail

            THEME_LINK="${THEME_LINK}"
            RELEASES_ROOT="${RELEASES_ROOT}"
            RELEASE_DIR="${RELEASE_DIR}"
            KEEP_RELEASES="${KEEP_RELEASES}"

            # First run only: the live path is a real directory from the
            # original deploy.sh copy, not yet a symlink — move it aside
            # instead of deleting it.
            if [ -e "\${THEME_LINK}" ] && [ ! -L "\${THEME_LINK}" ]; then
              mv "\${THEME_LINK}" "\${THEME_LINK}.pre-release-backup"
            fi

            ln -sfn "\${RELEASE_DIR}" "\${THEME_LINK}"

            # Sort by tag (semver), not mtime — rsync -a carries the
            # source dir's timestamp over, so mtime doesn't reflect
            # deploy order. Whatever the symlink now points at is always
            # protected, even if it's an older tag from a rollback.
            cd "\${RELEASES_ROOT}"
            CURRENT="\$(basename "\$(readlink -f "\${THEME_LINK}")")"
            ls -1d */ 2>/dev/null | sed 's#/\$##' | sort -rV | tail -n "+\$((KEEP_RELEASES + 1))" | while read -r old; do
              [ "\${old}" != "\${CURRENT}" ] && rm -rf "\${old}"
            done
          EOF

          # 5. Kill the agent for clean exit
          ssh-agent -k

How the release workflow operates: The CI/CD workflow listens for specific version tags to ensure that deployments only happen for deliberate milestones. It compiles the necessary assets using Node.js and packages a clean theme artifact. A vital fix in this configuration involves how WP-CLI packages are installed. Standard wp package install commands rely on GitHub APIs, which often fail because of rate limits applied to anonymous CI runners. To bypass this issue, the script installs the dist-archive-command directly from Packagist using Composer. This approach successfully avoids the GitHub API rate limit trap entirely.

During the actual deployment phase, the script authenticates with the server, creates a new directory named after the Git tag, and transfers the files. Finally, it runs an atomic symlink swap. This command points the active live environment to the newly released folder while automatically cleaning up older deployments to save server disk space.

The Rollback Trigger and Workflow

The rollback workflow acts as a manual safety net. By utilizing a workflow_dispatch trigger, a developer can type a specific version tag directly into the GitHub Actions interface to instantly revert the production environment.

Here is the rollback.yml configuration:

name: Rollback Production Theme

# Re-points the live wp-content/themes/rehan-lodhi symlink at a release
# that's still on disk under /home/rehanlodhi/releases/rehan-lodhi — no
# rebuild, no rsync, just an atomic symlink flip. Only the last 5 releases
# (per release.yml) are kept, so this only works for tags still retained.
on:
  workflow_dispatch:
    inputs:
      tag:
        description: 'Tag to roll back to, e.g. v1.2.0 (must still exist on the server)'
        required: true
        type: string

permissions:
  contents: read

concurrency:
  group: production-deploy

jobs:
  rollback:
    name: Repoint Theme Symlink
    runs-on: ubuntu-latest

    steps:
      - name: Roll back via SSH
        env:
          SSH_KEY: ${{ secrets.SSH_KEY }}
          SSH_USER: ${{ vars.SSH_USER }}
          SSH_HOST: ${{ vars.SSH_HOST }}
          RELEASES_ROOT: ${{ vars.PROD_RELEASES_ROOT }}
          THEME_LINK: ${{ vars.PROD_THEME_LINK }}
          TAG: ${{ inputs.tag }}
        run: |
          # Validate before it's embedded in a remote shell command below —
          # a stray quote in a hand-typed input could otherwise break out
          # of the heredoc and inject commands into the SSH session.
          if ! [[ "$TAG" =~ ^v[0-9]+\.[0-9]+\.[0-9]+$ ]]; then
            echo "Invalid tag '$TAG' — expected format vX.Y.Z" >&2
            exit 1
          fi

          eval $(ssh-agent -s)
          echo "$SSH_KEY" | tr -d '\r' | ssh-add -
          mkdir -p ~/.ssh
          chmod 700 ~/.ssh

          SSH_TARGET="${SSH_USER}@${SSH_HOST}"
          RELEASE_DIR="${RELEASES_ROOT}/${TAG}"

          ssh -o StrictHostKeyChecking=no "${SSH_TARGET}" bash -s <<EOF
            set -euo pipefail
            if [ ! -d "${RELEASE_DIR}" ]; then
              echo "No release at ${RELEASE_DIR} — it may have been pruned (only the last 5 are kept)." >&2
              echo "Available releases:" >&2
              ls -1 "${RELEASES_ROOT}" >&2
              exit 1
            fi
            ln -sfn "${RELEASE_DIR}" "${THEME_LINK}"
            echo "Rolled back ${THEME_LINK} -> ${RELEASE_DIR}"
          EOF

          ssh-agent -k

How the rollback workflow operates: Because the primary release workflow preserves a set number of previous deployments on the server, a rollback happens without any new file transfers or code recompilation. First, the script validates the format of the manually entered tag. After successful SSH authentication, it checks if the requested release directory still exists on the production server. If the directory is validated, the script simply updates the symlink to point back to that older directory. This resolves critical production issues in milliseconds.

Conclusion and Future Deployment Improvements

Implementing a symlink-based deployment strategy completely changes how you manage backend delivery. It drastically reduces deployment anxiety and mitigates operational risk.

Moving forward, you can improve this pipeline by adding a few extra features:

  • Automated HTTP Health Checks: Add a script to ping your site immediately after the symlink flips to ensure the server returns a successful status code.
  • Database Migration Safeguards: While this system handles file changes perfectly, adding a parallel process to safely manage and roll back database schema alterations will secure your entire application stack.
  • Automated Webhooks: Push deployment statuses to a Slack or Discord channel so your team has full operational visibility when changes go live.