CI/CD Pipeline

---

name: cicd-pipeline description: Create, debug, and manage CI/CD pipelines with GitHub Actions. Use when the user needs to set up automated testing, deployment, releases, or workflows. Covers workflow syntax, common patterns, secrets management, caching, matrix builds, and troubleshooting. metadata: {"clawdbot":{"emoji":"🚀","requires":{"anyBins":["gh","git"]},"os":["linux","darwin","win32"]}}

---

CI/CD Pipeline (GitHub Actions)

Set up and manage CI/CD pipelines using GitHub Actions. Covers workflow creation, testing, deployment, release automation, and debugging.

When to Use

Setting up automated testing on push/PR

Creating deployment pipelines (staging, production)

Automating releases with changelogs and tags

Debugging failing CI workflows

Setting up matrix builds for cross-platform testing

Managing secrets and environment variables in CI

Optimizing CI with caching and parallelism

Quick Start: Add CI to a Project

Node.js project

# .github/workflows/ci.yml
name: CI
on:
  push:
    branches: [main]
  pull_request:
    branches: [main]
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 20
          cache: npm
      - run: npm ci
      - run: npm test
      - run: npm run lint

Python project

# .github/workflows/ci.yml
name: CI
on:
  push:
    branches: [main]
  pull_request:
    branches: [main]
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
          cache: pip
      - run: pip install -r requirements.txt
      - run: pytest
      - run: ruff check .

Go project

# .github/workflows/ci.yml
name: CI
on:
  push:
    branches: [main]
  pull_request:
    branches: [main]
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-go@v5
        with:
          go-version: "1.22"
      - run: go test ./...
      - run: go vet ./...

Rust project

# .github/workflows/ci.yml
name: CI
on:
  push:
    branches: [main]
  pull_request:
    branches: [main]
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: dtolnay/rust-toolchain@stable
      - uses: Swatinem/rust-cache@v2
      - run: cargo test
      - run: cargo clippy -- -D warnings

Common Patterns

Matrix builds (test across versions/OSes)

jobs:
  test:
    strategy:
      fail-fast: false
      matrix:
        os: [ubuntu-latest, macos-latest, windows-latest]
        node-version: [18, 20, 22]
    runs-on: ${{ matrix.os }}
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: ${{ matrix.node-version }}
      - run: npm ci
      - run: npm test

Conditional jobs

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npm test
  deploy:
    needs: test
    if: github.ref == 'refs/heads/main' && github.event_name == 'push'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: ./deploy.sh

Caching dependencies

# Node.js (automatic with setup-node)
uses: actions/setup-node@v4
  with:
    node-version: 20
    cache: npm  # or yarn, pnpm
Generic caching
uses: actions/cache@v4
  with:
    path: |
      ~/.cache/pip
      ~/.cargo/registry
      node_modules
    key: ${{ runner.os }}-deps-${{ hashFiles('**/package-lock.json') }}
    restore-keys: |
      ${{ runner.os }}-deps-

Artifacts (save build outputs)

- uses: actions/upload-artifact@v4
  with:
    name: build-output
    path: dist/
    retention-days: 7
Download in another job
uses: actions/download-artifact@v4
  with:
    name: build-output
    path: dist/

Run on schedule (cron)

on:
  schedule:
    - cron: "0 6 * * 1"  # Every Monday at 6 AM UTC
  workflow_dispatch:  # Also allow manual trigger

Deployment Workflows

Deploy to production on tag

name: Release
on:
  push:
    tags:
      - "v*"
jobs:
  release:
    runs-on: ubuntu-latest
    permissions:
      contents: write
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 20
          cache: npm
      - run: npm ci
      - run: npm run build
      - run: npm test
      # Create GitHub release
      - uses: softprops/action-gh-release@v2
        with:
          generate_release_notes: true
          files: |
            dist/*.js
            dist/*.css

Deploy to multiple environments

name: Deploy
on:
  push:
    branches: [main, staging]
jobs:
  deploy:
    runs-on: ubuntu-latest
    environment: ${{ github.ref == 'refs/heads/main' && 'production' || 'staging' }}
    steps:
      - uses: actions/checkout@v4
      - run: npm ci && npm run build
      - run: |
          if [ "${{ github.ref }}" = "refs/heads/main" ]; then
            ./deploy.sh production
          else
            ./deploy.sh staging
          fi
        env:
          DEPLOY_TOKEN: ${{ secrets.DEPLOY_TOKEN }}

Docker build and push

name: Docker
on:
  push:
    branches: [main]
    tags: ["v*"]
jobs:
  build:
    runs-on: ubuntu-latest
    permissions:
      packages: write
    steps:
      - uses: actions/checkout@v4
      - uses: docker/setup-buildx-action@v3
      - uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}
      - uses: docker/build-push-action@v6
        with:
          push: true
          tags: |
            ghcr.io/${{ github.repository }}:latest
            ghcr.io/${{ github.repository }}:${{ github.sha }}
          cache-from: type=gha
          cache-to: type=gha,mode=max

npm publish on release

name: Publish
on:
  release:
    types: [published]
jobs:
  publish:
    runs-on: ubuntu-latest
    permissions:
      id-token: write
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 20
          registry-url: https://registry.npmjs.org
      - run: npm ci
      - run: npm test
      - run: npm publish --provenance
        env:
          NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}

Secrets Management

Set secrets via CLI

# Set a repository secret
gh secret set DEPLOY_TOKEN --body "my-secret-value"
Set from a file
gh secret set SSH_KEY < ~/.ssh/deploy_key
Set for a specific environment
gh secret set DB_PASSWORD --env production --body "p@ssw0rd"
List secrets
gh secret list
Delete a secret
gh secret delete OLD_SECRET

Use secrets in workflows

env:
  # Available to all steps in this job
  DATABASE_URL: ${{ secrets.DATABASE_URL }}
steps:
  - run: echo "Deploying..."
    env:
      # Available to this step only
      API_KEY: ${{ secrets.API_KEY }}

Environment protection rules

Set up via GitHub UI or API:

Required reviewers before deployment

Wait timers

Branch restrictions

Custom deployment branch policies

# View environments
gh api repos/{owner}/{repo}/environments | jq '.environments[].name'

Workflow Debugging

Re-run failed jobs

# List recent workflow runs
gh run list --limit 10
View a specific run
gh run view 
View failed job logs
gh run view  --log-failed
Re-run failed jobs only
gh run rerun  --failed
Re-run entire workflow
gh run rerun 

Debug with SSH (using tmate)

# Add this step before the failing step
uses: mxschmitt/action-tmate@v3
  if: failure()
  with:
    limit-access-to-actor: true

Common failures and fixes

"Permission denied" on scripts

- run: chmod +x ./scripts/deploy.sh && ./scripts/deploy.sh

"Node modules not found"

# Make sure npm ci runs before npm test
run: npm ci     # Install exact lockfile versions
run: npm test   # Now node_modules exists

"Resource not accessible by integration"

# Add permissions block
permissions:
  contents: write
  packages: write
  pull-requests: write

Cache not restoring

# Check cache key matches - use hashFiles for lockfile
key: ${{ runner.os }}-node-${{ hashFiles('**/package-lock.json') }}
NOT: key: ${{ runner.os }}-node-${{ hashFiles('package.json') }}

Workflow not triggering

Check: is the workflow file on the default branch?

Check: does the trigger event match? (push vs pull_request)

Check: is the branch filter correct?

# Manually trigger a workflow
gh workflow run ci.yml --ref main

Workflow Validation

Validate locally before pushing

# Check YAML syntax
python3 -c "import yaml; yaml.safe_load(open('.github/workflows/ci.yml'))" && echo "Valid"
Use actionlint (if installed)
actionlint .github/workflows/ci.yml
Or via Docker
docker run --rm -v "$(pwd):/repo" -w /repo rhysd/actionlint:latest

View workflow as graph

# List all workflows
gh workflow list
View workflow definition
gh workflow view ci.yml
Watch a running workflow
gh run watch

Advanced Patterns

Reusable workflows

# .github/workflows/reusable-test.yml
name: Reusable Test
on:
  workflow_call:
    inputs:
      node-version:
        required: false
        type: string
        default: "20"
    secrets:
      npm-token:
        required: false
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: ${{ inputs.node-version }}
      - run: npm ci
      - run: npm test

# .github/workflows/ci.yml - caller
name: CI
on: [push, pull_request]
jobs:
  test:
    uses: ./.github/workflows/reusable-test.yml
    with:
      node-version: "20"

Concurrency (prevent duplicate runs)

concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true  # Cancel previous runs for same branch

Path filters (only run for relevant changes)

on:
  push:
    paths:
      - "src/**"
      - "package.json"
      - "package-lock.json"
      - ".github/workflows/ci.yml"
    paths-ignore:
      - "docs/**"
      - "*.md"

Monorepo: only test changed packages

jobs:
  changes:
    runs-on: ubuntu-latest
    outputs:
      api: ${{ steps.filter.outputs.api }}
      web: ${{ steps.filter.outputs.web }}
    steps:
      - uses: actions/checkout@v4
      - uses: dorny/paths-filter@v3
        id: filter
        with:
          filters: |
            api:
              - 'packages/api/**'
            web:
              - 'packages/web/**'
  test-api:
    needs: changes
    if: needs.changes.outputs.api == 'true'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: cd packages/api && npm ci && npm test
  test-web:
    needs: changes
    if: needs.changes.outputs.web == 'true'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: cd packages/web && npm ci && npm test

Tips

Use workflow_dispatch on every workflow for manual triggering during debugging

Pin action versions to SHA for supply chain security: uses: actions/checkout@b4ffde...

Use continue-on-error: true for non-critical steps (like linting)

Set timeout-minutes on jobs to prevent runaway builds (default is 360 minutes)

Use job outputs to pass data between jobs: outputs: result: ${{ steps.step-id.outputs.value }}

For self-hosted runners: runs-on: self-hosted with labels for targeting specific machines

Use workflow_dispatch on every workflow for manual triggering during debugging

Pin action versions to SHA for supply chain security: uses: actions/checkout@b4ffde...

Use continue-on-error: true for non-critical steps (like linting)

Set timeout-minutes on jobs to prevent runaway builds (default is 360 minutes)

Use job outputs to pass data between jobs: outputs: result: ${{ steps.step-id.outputs.value }}

For self-hosted runners: runs-on: self-hosted with labels for targeting specific machines