Sealing Secrets in CI: Stopping Token Drift in Container Build Pipelines

Hook: The Base Image That Carried Your AWS Keys

The platform team baked a new base image for GitHub Actions runners. It included a convenience script that exported AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY from the environment into /tmp/aws.env for local testing parity. A week later, someone triggered an innocuous docs build. The workflow cached the Docker build context, packaging /tmp/aws.env into the image layer. That image got pushed to the internal registry, pulled into staging, and an intern noticed AWS keys in the container history. No intrusion occurred, but the keys rotated, pipeline halted, and everyone wondered how a docs job grabbed production credentials.

Secret drift happens when tokens survive longer than the job that issued them. CI pipelines stitch together runner environments, caches, and images. If any stage writes secrets to disk or logs, the next stage can pick them up. This article shows how to stop the bleeding: isolate environments, scrub artifacts, enforce per-step token scopes, and test the pipeline with the same rigor you test code. We will use GitHub Actions and GitLab CI examples, cover both quick wins and deeper systemic fixes, and leave you with verification steps to catch regressions before they hit prod.

The Problem Deep Dive

Developers assume CI secrets vanish when the job ends. In reality, secrets leak via:

• Filesystem persistence. /tmp and workspaces persist across steps by default. Scripts writing env dumps create artifacts other steps inherit.
• Docker layer caching. When builds cache layers, any secrets copied into the image remain in intermediate layers even if later layers remove them.
• Shared runners. Self-hosted runners reuse machines. If cleanup hooks fail, environment variables and credential files linger.
• Verbose tooling. Package managers print tokens in logs when debug mode is on.

Consider this GitHub Actions snippet:

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Configure AWS
        run: |
          aws configure set aws_access_key_id ${{ secrets.AWS_KEY }}
          aws configure set aws_secret_access_key ${{ secrets.AWS_SECRET }}
      - name: Build image
        run: docker build -t internal/app .
      - name: Save cache
        uses: actions/cache@v3
        with:
          path: .
          key: build-${{ github.sha }}

aws configure writes credentials to ~/.aws/credentials. The cache step archives the entire workspace, including that file. The next job restoring the cache inherits the keys.

Technical Solutions

Quick Patch: Explicit Secret Scrubbing

After using a secret, delete it and avoid caching sensitive paths:

      - name: Configure AWS
        run: |
          mkdir -p ~/.aws
          cat <<'AWS_CFG' > ~/.aws/credentials
[default]
aws_access_key_id=${{ secrets.AWS_KEY }}
aws_secret_access_key=${{ secrets.AWS_SECRET }}
AWS_CFG
      - name: Build image
        run: docker build --secret id=aws,src=<(cat ~/.aws/credentials) -t internal/app .
      - name: Cleanup
        run: |
          rm -f ~/.aws/credentials
          shred -u ~/.aws/credentials || true
      - name: Save cache
        uses: actions/cache@v3
        with:
          path: node_modules
          key: deps-${{ github.sha }}

Durable Fix: Ephemeral Runners and Scoped Tokens

Provision per-job runners that destroy themselves after execution. On GitHub Actions, use actions-runner-controller with ephemeral mode:

runs-on: [self-hosted, ephemeral]

In GitLab CI, configure the Kubernetes executor so each job receives a fresh pod. Combine this with short-lived credentials using OIDC:

permissions:
  id-token: write
  contents: read

steps:
  - name: Assume AWS role
    uses: aws-actions/configure-aws-credentials@v4
    with:
      role-to-assume: arn:aws:iam::123456789012:role/build-role
      aws-region: us-east-1

OIDC removes static secrets and limits blast radius. Grant the assumed role only the permissions needed for that job.

Docker Build Secrets

Use build-time secrets so credentials never enter the layer cache:

# syntax=docker/dockerfile:1.4

RUN --mount=type=secret,id=aws,target=/root/.aws/credentials \
    aws s3 cp s3://internal-deps/config.json /app/config.json

Pass secrets at build time rather than copying files into the context:

docker build --secret id=aws,src=/tmp/aws-creds -t internal/app .

Workspace Policies

Limit caches and artifacts:

• Cache only dependency directories (node_modules, .m2, ~/.cache/pip).
• In GitLab, use cache: paths: - node_modules/ and avoid cache: paths: - ..
• Disable artifact upload for paths containing secrets.

Secret Scanners in CI

Run scanners after critical stages:

gitleaks detect --source . --no-git
trufflehog filesystem .

Scan container images before pushing:

docker run --rm ghcr.io/zricethezav/gitleaks:latest detect --source=oci://internal/app:latest

Alprina Policies

Configure Alprina to parse CI configs. Flag workflows that cache home directories, use static secrets instead of OIDC, or invoke Docker builds without --secret mounts. Fail the pipeline when policies are violated.

Testing & Verification

Add automated canary tests that fail when secrets leak:

      - name: Canary secret
        run: echo "CI_CANARY=${{ secrets.CI_CANARY }}" >> $GITHUB_ENV
      - name: Check workspace for canary
        run: |
          if grep -R "${CI_CANARY}" .; then
            echo "Canary found"
            exit 1
          fi

After the job, assert that the runner is destroyed. Track runner IDs and verify they disappear from your infrastructure provider. For Docker builds, run docker history internal/app:latest and ensure no layer contains sensitive strings.

Common Questions & Edge Cases

Do we really need ephemeral runners? If cost prohibits them, enforce cleanup scripts that wipe /tmp, home directories, and Docker caches post-job. Reimage shared runners on a schedule.

What about caching compiled assets? Store them in remote caches (S3, GCS) with server-side encryption. Avoid caching the entire workspace.

How do we debug failed jobs if environments vanish? Capture logs and artifacts explicitly, and provide controlled debugging jobs with short-lived credentials.

Can third-party actions leak secrets? Yes. Pin versions, review their code, and run them in isolated jobs with minimal permissions.

Is OIDC alone enough? No. Combine it with IAM policies, CloudTrail alerts, and secret scanners for visibility.

Conclusion

CI pipelines blend developer convenience with production power. Treat secrets as single-use tokens: inject them, consume them, and wipe every trace. Ephemeral runners, scoped credentials, secret-aware caching, and automated scanners keep credentials from hitchhiking in your artifacts and container layers.