Docker image layers store
every secret you ever added.

How Docker layers actually work at the filesystem level

A Docker image is not a flat filesystem. It is a stack of read-only layers, where each layer represents a filesystem diff produced by one Dockerfile instruction. When you run docker build, every RUN, COPY, and ADD instruction creates a new layer containing only the files that changed.

These layers are stored as tarballs and addressed by content hash (SHA-256). The image manifest is a JSON file that lists layer hashes in order. The Docker daemon combines them using a union filesystem (OverlayFS on most Linux hosts) to present a single coherent view. A file that is "deleted" in layer N is hidden from the union view — but the original content still exists, verbatim, in the layer tarball for layer N-1.

FROM python:3.12-slim

# Layer 1: base packages
RUN apt-get update && apt-get install -y git

# Layer 2: secrets copied in  ← THE PROBLEM
COPY .env /app/.env
COPY credentials.json /root/.aws/credentials

# Layer 3: use the secrets to install private packages
RUN pip install -r requirements-private.txt

# Layer 4: "clean up" — this does absolutely nothing to remove secrets
RUN rm /app/.env /root/.aws/credentials

COPY . /app/
CMD ["python", "-m", "app"]

Why "delete in a later layer" is the most dangerous Docker misconception

Stack Overflow answers from 2016-2020 frequently suggested this cleanup pattern as a solution for keeping images small. The advice was wrong about security even then. The confusion persists because it works visually: running docker exec into the container, you genuinely cannot see the deleted files. The union filesystem hides them. But the image tarball still contains every layer unchanged.

Consider the operational reality: you build the image, push it to ECR, and your CI pipeline scans the running container for secrets. The scanner finds nothing — because the union view hides the deleted files. The image gets a green tick and ships to production. Meanwhile, the ECR repository contains every layer. Any engineer with ecr:GetDownloadUrlForLayer permission can extract the raw layer bytes.

# Save the image as a tarball
$ docker save myapp:latest -o myapp.tar

# The tarball is a nested structure: manifest.json + per-layer tarballs
$ tar tf myapp.tar | head -20
manifest.json
repositories
sha256:3a7f8e.../
sha256:3a7f8e.../layer.tar     # Layer 1: base
sha256:b2c14d.../layer.tar     # Layer 2: .env was COPY'd here
sha256:9f1aa2.../layer.tar     # Layer 3: pip install
sha256:d43e8b.../layer.tar     # Layer 4: rm .env (whiteout only)

# Extract layer 2 and find the secret
$ mkdir layer2 && tar xf sha256:b2c14d.../layer.tar -C layer2
$ cat layer2/app/.env
DATABASE_URL=postgres://admin:[email protected]:5432/app
AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
STRIPE_SECRET_KEY=sk_live_4eC39HqLyjWDarjtT1zdp7dc

Three commands. No special privileges. No exploits. The secrets are in plain text. This is the exact sequence an attacker runs after pulling the image, or after gaining read access to an S3-backed registry.

Extraction in practice: docker history, dive, and Skopeo

Three tools make this trivially accessible. You don't need to write code. You don't need root. You just need the image.

docker history — the quick check

docker history --no-trunc <image> prints every layer's creation command. ARG values and inline secrets passed via RUN are visible here in full, unredacted.

$ docker history --no-trunc myapp:latest

IMAGE          CREATED BY
sha256:d43e8b  RUN /bin/sh -c rm /app/.env /root/.aws/credentials
sha256:9f1aa2  RUN /bin/sh -c pip install -r requirements.txt \
               --extra-index-url https://token:[email protected]/simple/
sha256:b2c14d  COPY .env /app/.env
sha256:3a7f8e  RUN apt-get update && apt-get install -y git

dive — interactive layer explorer

dive is an open-source tool that lets you interactively browse each layer's filesystem diff. It makes extracting secrets from any layer point-and-click: browse to the layer where .env was added, and its content is right there in the file tree pane. CI integration is available via CI=true dive <image>.

Skopeo — registry-level extraction without docker pull

Skopeo can copy registry images directly to disk in OCI layout without a Docker daemon. This matters because an attacker with registry read credentials (leaked in a CI log, for example) can extract every layer without ever running the container:

# Copy image from ECR to local OCI directory without docker pull
$ skopeo copy \
  docker://123456789.dkr.ecr.eu-west-1.amazonaws.com/myapp:latest \
  oci:/tmp/myapp-oci

# Navigate layer blobs directly
$ ls /tmp/myapp-oci/blobs/sha256/
3a7f8e...  b2c14d...  9f1aa2...  d43e8b...

# Extract the layer containing secrets
$ tar xf /tmp/myapp-oci/blobs/sha256/b2c14d... -C /tmp/extracted
$ cat /tmp/extracted/app/.env

Registry exposure: public ECR and Docker Hub repositories

The blast radius of this problem scales with registry accessibility. A private registry with tight IAM controls limits exposure to internal attackers or credential-theft scenarios. A public Docker Hub repository exposes every layer to the entire internet.

GitHub Actions workflows frequently push images to Docker Hub or public ECR repositories. The combination of "we made the repo public for easier deployment" and "we once copied a .env into a layer" produces a public credential exposure. Researchers scanning Docker Hub for exposed secrets have found AWS keys, database credentials, and API tokens in images from well-known companies, all extracted via the docker save method above.

ARG and ENV variable leakage via docker history

Even without copying a secrets file, passing secrets as build arguments or environment variables leaks them into layer metadata — visible via docker history --no-trunc forever.

FROM node:20-alpine

# ARG values are baked into layer metadata
ARG NPM_TOKEN
RUN echo "//registry.npmjs.org/:_authToken=${NPM_TOKEN}" > ~/.npmrc && \
    npm install && \
    rm ~/.npmrc

# The RUN layer's creation command in docker history will show:
# /bin/sh -c echo "//registry.npmjs.org/:_authToken=npm_AbCdEf123..." > ~/.npmrc

The rm ~/.npmrc at the end of the same RUN instruction is the correct pattern — it keeps the file creation and deletion in a single layer, preventing file-level persistence. But the ARG value is still recorded verbatim in the layer's creation command metadata, readable via docker history. This is a separate, lesser-known leak vector.

Patterns that actually prevent secrets from entering layers

Option 1: BuildKit secret mounts (recommended)

BuildKit's --mount=type=secret syntax provides a temporary, in-memory mount point available only during a single RUN instruction. The secret is never written to any layer, does not appear in docker history, and cannot be extracted by any image inspection tool.

# syntax=docker/dockerfile:1
FROM python:3.12-slim

RUN --mount=type=secret,id=pip_token \
    pip install \
      --extra-index-url "$(cat /run/secrets/pip_token)" \
      -r requirements.txt

# Secret is passed at build time, never stored in any layer
$ DOCKER_BUILDKIT=1 docker build \
  --secret id=pip_token,env=PIP_EXTRA_INDEX_URL \
  -t myapp:latest .

Option 2: Multi-stage builds to exclude the secret layer entirely

In a multi-stage build, the final image is copied from an intermediate builder stage. If the secret-using operations happen only in the builder stage and only non-secret artifacts are COPY --from=builder'd into the final stage, the secret never reaches the final image at all.

# Stage 1: builder (never pushed to registry)
FROM python:3.12-slim AS builder
ARG PIP_EXTRA_INDEX_URL
RUN pip install --prefix=/install -r requirements.txt

# Stage 2: final image — contains only the installed packages, not ARGs
FROM python:3.12-slim
COPY --from=builder /install /usr/local
COPY . /app/
CMD ["python", "-m", "app"]

Scanning container images in CI before they reach the registry

The fix is preventive: scan images for secrets before pushing to the registry. Several tools can do this as a CI gate.

• Trivy (trivy image --scanners secret myapp:latest) — scans layer content for secret patterns using a regex-based ruleset. Runs against the local daemon or an OCI archive.
• Grype combined with Syft — Syft generates an SBOM with file hashes; Grype checks vulnerabilities. Neither scans for secrets natively, so combine with a dedicated secrets scanner.
• AquilaX container scan — scans all image layers for secrets, PII, misconfigurations, and known CVEs in a single pipeline step, reporting before push.

- name: Build image
  run: docker build -t myapp:{{ github.sha }} .

- name: Scan image for secrets
  run: |
    trivy image \
      --scanners secret \
      --exit-code 1 \
      --severity HIGH,CRITICAL \
      myapp:{{ github.sha }}

# Push only happens if the scan step exits 0
- name: Push to registry
  run: docker push myapp:{{ github.sha }}

Remediation checklist

• Audit existing images with docker save | tar xf | grep -r SECRET_PATTERN or a secrets scanner before deciding what to rotate.
• Migrate all secret-passing patterns to BuildKit --mount=type=secret.
• Use multi-stage builds so the build-time environment (with credentials) is never part of the pushed image.
• Never pass real credentials via ARG — even for "temporary" builds.
• Add a Trivy or AquilaX secrets scan as a CI gate before every registry push.
• If a compromised image was ever public, rotate all credentials it touched immediately.