How We Cut Our CI Pipeline from 45 Minutes to 8 Minutes

The Problem

Meet Alex, a backend engineer at a growing startup. Every time Alex pushes code, the same ritual plays out: hit git push, switch to Slack, wait. And wait. And wait.

It started as a joke in standup: “Go grab coffee, the CI will be done by the time you’re back.” Then it stopped being funny. Alex’s team was losing hours every day just waiting for builds. Their CI pipeline was taking 45 minutes on a good day. On bad days — dependency cache misses, flaky tests, runner queue waits — it pushed past an hour.

The pipeline was a monolith: one workflow file, one job, everything sequential. Lint, type-check, unit tests, integration tests, build, e2e tests, deploy preview. All in series. If linting failed at minute 2, everything after it was wasted compute. If the build succeeded at minute 40, Alex still had to wait for e2e tests to finish before seeing a preview.

Why this matters: Every minute your team spends waiting on CI is a minute they’re not shipping features. For a team of 10 developers pushing 10 PRs a day, a 45-minute pipeline burns 75 person-hours per week on waiting alone. That’s almost two full work weeks — gone.

The Investigation

Alex’s team started by instrumenting every step. They added timing logs and cache-hit reporting to each job. Here’s what the raw data looked like:

Step                    Duration    Cache hit?
──────────────────────────────────────────────
Install dependencies    4m 20s      ❌ (miss)
Lint                    2m 10s      N/A
Type check              3m 45s      N/A
Unit tests              8m 30s      N/A
Build                   6m 15s      ❌ (miss)
Integration tests       12m 40s     N/A
E2E tests               7m 20s      N/A
──────────────────────────────────────────────
Total:                  45m 00s

Let’s break down what each metric is telling you:

• Duration — How long that step took from start to finish. Simple enough.
• Cache hit? — Whether the step found a valid cached version of its dependencies. A “miss” means it downloaded and installed everything from scratch, which is almost always the slowest path.
• N/A — Steps like linting and tests don’t cache anything, so there’s nothing to hit or miss.

Two things jumped out at Alex:

1. Dependency caching was broken — they were missing 60% of the time. Every miss meant re-downloading the entire node_modules tree from scratch.
2. Everything ran in sequence — no parallelism at all. The build step sat idle while linting ran, even though they didn’t depend on each other.

Key lesson: You can’t optimize what you don’t measure. Before Alex’s team added instrumentation, they assumed the build step was the bottleneck. The data showed otherwise — dependency install was the real culprit.

The Solution

Step 1: Fix Dependency Caching

The root cause was naive cache keys. Alex’s team was using just the yarn.lock hash, but every PR branch changed the lockfile slightly. The fix was multi-layered:

- name: Cache dependencies
  uses: actions/cache@v4
  with:
    path: |
      **/node_modules
      ~/.cache/yarn
    key: >
      ${{ runner.os }}-yarn-
      ${{ hashFiles('yarn.lock') }}-
      ${{ github.base_ref }}
    restore-keys: |
      ${{ runner.os }}-yarn-${{ hashFiles('yarn.lock') }}-
      ${{ runner.os }}-yarn-

Here’s what each piece does:

• path — Tells GitHub Actions which folders to cache. Alex’s team cached both node_modules (where installed packages live) and Yarn’s global cache (where downloaded package tarballs live).
• key — The unique identifier for this cache entry. It combines the OS, a hash of your lockfile, and the target branch. If any of these change, it’s a new cache entry.
• restore-keys — Fallback keys. If the exact key doesn’t match, GitHub tries these in order. The first one that matches restores a partial cache, so you only download the packages that changed.

The key insight: Alex added github.base_ref to the key so PRs against main could share a cache, while restore-keys provided fallback to any prior cache. Cache hit rate went from 40% to 92%.

Step 2: Parallel Job Decomposition

Next, Alex broke the monolith into parallel jobs with explicit dependencies:

jobs:
  lint:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: ./.github/actions/setup
      - run: yarn lint

  typecheck:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: ./.github/actions/setup
      - run: yarn typecheck

  unit-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: ./.github/actions/setup
      - run: yarn test:unit

  build:
    needs: [lint, typecheck, unit-tests]
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: ./.github/actions/setup
      - run: yarn build

Here’s what each piece does:

• jobs — Defines independent units of work. Each job runs on its own runner, so they can execute in parallel.
• runs-on — Specifies the type of machine to use. ubuntu-latest is the standard choice for most workflows.
• needs — Declares dependencies between jobs. The build job won’t start until lint, typecheck, and unit-tests all finish successfully. This gives you a safety gate without serializing everything.
• uses: ./.github/actions/setup — A reusable composite action that Alex extracted to avoid repeating checkout, caching, and install steps across every job.

Now linting, type-checking, and unit tests all run at the same time. The build step waits for all three to pass, then runs on its own. Total wall-clock time dropped from 45 minutes to about 20 minutes with just this change.

Step 3: Test Splitting

Unit tests were still the longest parallelizable step. Alex split them across 4 runners:

unit-tests:
  strategy:
    matrix:
      shard: [1, 2, 3, 4]
    steps:
      - run: yarn test:unit --shard=${{ matrix.shard }}/4

This tells GitHub Actions to spin up 4 identical runners, each running a quarter of the test suite. The --shard flag tells your test runner which slice to execute. With 4 shards, the 8-minute unit test step dropped to just over 2 minutes.

Production pitfall: Fixed sharding works well when your tests are roughly equal in duration. If one shard consistently runs slower (e.g., it gets all the integration-heavy tests), you’ll end up waiting for that one straggler. Consider adaptive splitting tools like @jest/test-sequencer or split for more balanced distribution.

The Results

Metric	Before	After
Pipeline duration	45m	8m
Cache hit rate	40%	92%
Runner minutes/day	1,350	240
Developer wait time	~3h/day	~30min/day

What this means for you: Alex’s team went from spending 3 hours per developer per day waiting on CI to just 30 minutes. That’s 2.5 hours reclaimed per person, per day. For a team of 10, that’s 25 hours a day — more than enough to ship an extra feature or tackle tech debt every single sprint.

The runner minutes dropped too, which might seem counterintuitive (more parallelism usually means more compute). But the cache fix was the real hero here — fewer cache misses meant less redundant downloading, which more than offset the cost of parallel runners.

What to Watch Out For

Simplicity. One workflow file became a directory of composable actions. That’s more files to navigate, more YAML to understand. If your team is small or your CI is simple, the monolith might be fine. Don’t over-engineer.

Debugging ease. When jobs run in parallel, a failure in one job doesn’t stop the others. You get multiple error notifications, and tracing the root cause across parallel logs takes more effort. Alex’s team added a Slack notification that only fires when all jobs complete, summarizing pass/fail per job.

Runner cost. More parallel runners = more concurrent minutes. GitHub Actions bills by the minute, and parallel jobs multiply your bill. In Alex’s case, the cache improvements offset the cost, but your mileage may vary. Check your billing dashboard after making changes.

Cache invalidation. The multi-layered cache key works great until it doesn’t. If you upgrade a major dependency version, the lockfile hash changes, and everyone gets a cold cache for that first build. Plan for it — maybe schedule a weekly “cache warm” workflow that runs on main after hours.

Key lesson: Cache key design is the single most impactful optimization you can make in CI. A well-designed cache key with good fallbacks saves more time than any amount of parallelism. Start there, then parallelize.

Alex’s team learned something else too: always measure before optimizing. Their “slow build” assumption was wrong — dependency install was the real culprit. The data told them where to focus, and the results spoke for themselves. Your team’s bottleneck might be different, but the approach is the same: measure, identify, fix, repeat.