Why secrets leak β€” and why it matters

Secrets β€” API keys, database passwords, OAuth tokens, private keys, encryption keys β€” are the most targeted artefacts in a software supply chain attack. The 2024 GitGuardian State of Secrets Sprawl report found over 12.8 million secrets committed to public GitHub repositories in a single year. The real number, including private repositories and other platforms, is orders of magnitude higher.

The common entry points:

  • Committed to git: A developer adds a .env file or hardcodes a key during debugging and commits it. Even after removal, it lives in git history forever.
  • Leaked via logs: Exception handlers or debug logging expose environment variables or request headers containing tokens.
  • Exposed in build artefacts: Docker images, compiled binaries, or deployment packages embed secrets from the build environment.
  • Misconfigured CI/CD: Secrets injected as environment variables are printed by a verbose test runner or error message.
  • Shared credential stores: Credentials stored in wikis, Slack messages, or shared spreadsheets.

Git history is permanent: Removing a secret from the current HEAD does not remove it from history. Every clone of the repository contains the full history. You must assume the secret is compromised and rotate it immediately.

OWASP guidance on secrets management

OWASP addresses secrets management primarily through the Secrets Management Cheat Sheet and the broader OWASP Top 10 (A02:2021 – Cryptographic Failures and A05:2021 – Security Misconfiguration both cover credential exposure). The core principles:

  • Never store secrets in source code. Not even in private repositories. Code is shared, forked, cloned, and backed up in ways that bypass access controls.
  • Treat secrets as ephemeral. Secrets should have short lifetimes, be rotated automatically, and be revocable without requiring a code deployment.
  • Centralise secret storage. Use a dedicated secret manager (HashiCorp Vault, AWS Secrets Manager, GCP Secret Manager, Azure Key Vault). This gives you audit logs, access control, and rotation in one place.
  • Least privilege access to secrets. Each service or process should only be able to access the specific secrets it needs.
  • Audit secret access. Every read of a secret should be logged. Unusual access patterns (bulk reads, reads from unexpected IPs) are early indicators of compromise.

OWASP Secrets Management Cheat Sheet: The full reference is available at cheatsheetseries.owasp.org. It covers storage, transport, generation, rotation, and detection in detail.

Environment variables β€” the right way

Environment variables are the standard mechanism for injecting configuration into processes. They are better than hardcoded values, but they come with their own risks that are frequently misunderstood.

What environment variables protect against

  • Secrets appearing in source code
  • Secrets appearing in version control
  • Different secrets per environment (dev/staging/prod) without code changes

What environment variables do NOT protect against

  • Process inspection: any process running as the same user can read /proc/<pid>/environ on Linux
  • Logging: many frameworks log all environment variables on startup or in crash dumps
  • Child processes: environment variables are inherited by all child processes
  • .env files committed to git
.gitignoretext
# Always add .env files to .gitignore
.env
.env.local
.env.*.local
.env.production
.env.staging
# Also ignore any file that might contain secrets
*.pem
*.key
secrets.json
credentials.json

Safe .env file practices

  • Commit a .env.example with placeholder values as documentation
  • Never commit the actual .env file β€” add it to .gitignore before the first commit
  • Use different secrets for each environment; never share production secrets with development
  • Restrict file permissions: chmod 600 .env

Pre-existing .env in git: If a .env file was previously committed, adding it to .gitignore does not remove it from history. Use git filter-repo to purge the file from history and rotate all exposed secrets.

Secret vaults β€” centralised management

A secret vault is a dedicated system for storing, accessing, and auditing secrets. It replaces scattered environment variables, config files, and shared password databases with a single, controlled plane.

Key vault options

  • HashiCorp Vault: The most feature-rich open-source option. Supports dynamic secrets (generates short-lived database credentials on demand), fine-grained ACLs, audit logs, and multiple authentication backends.
  • AWS Secrets Manager: Best if your workloads run on AWS. Integrates with IAM for access control, supports automatic rotation for RDS, Redshift, and other AWS services.
  • GCP Secret Manager: Native to Google Cloud. Simple API, IAM-based access control, version management.
  • Azure Key Vault: Native to Azure. Supports secrets, keys, and certificates. Integrates with Azure Managed Identities for keyless authentication.
  • 1Password Secrets Automation: Good for teams that already use 1Password; bridges human and machine secret access.

Dynamic secrets: The most powerful vault feature. Instead of a static database password, Vault creates a unique credential for each application instance with a short TTL. When the TTL expires, the credential is automatically revoked. There is no long-lived secret to steal.

Secrets in CI/CD pipelines

CI/CD pipelines are a major source of secret exposure. Build systems need secrets to deploy, sign, and integrate β€” and they are frequently misconfigured.

Best practices for CI/CD secrets

  • Use the platform's native secret store: GitHub Actions Secrets, GitLab CI Variables (masked), CircleCI Contexts, Jenkins Credentials. Never put secrets in yaml pipeline files.
  • Mask secrets in logs: Most CI platforms mask registered secrets in logs, but this is not foolproof. Avoid printing secrets explicitly.
  • Use short-lived OIDC tokens instead of static credentials: GitHub Actions, GitLab CI, and most major CI platforms support OIDC federation with cloud providers. Your pipeline authenticates with a short-lived token rather than a static key.
  • Restrict secret access to specific branches and environments: Production secrets should only be accessible from the production deployment pipeline, not feature branches.
  • Audit pipeline logs: Regularly review CI logs for accidental secret exposure.
.github/workflows/deploy.ymlYAML
# OIDC authentication β€” no static AWS keys needed
permissions:
  id-token: write
  contents: read

steps:
  - name: Configure AWS credentials via OIDC
    uses: aws-actions/configure-aws-credentials@v4
    with:
      role-to-assume: arn:aws:iam::123456789012:role/GitHubActionsRole
      aws-region: us-east-1
# No AWS_ACCESS_KEY_ID or AWS_SECRET_ACCESS_KEY needed

Secret rotation and expiry

Static secrets that never change are the most dangerous kind. A secret that was compromised six months ago and never rotated is actively being used by an attacker right now. Rotation limits the window of exploitation.

  • Set maximum lifetimes: Define a maximum age for every secret type. API keys: 90 days. Database passwords: 30 days. Short-lived tokens: hours or minutes.
  • Automate rotation: Manual rotation does not happen consistently. Use vault rotation policies, cloud-native rotation (AWS Secrets Manager can rotate RDS credentials automatically), or Kubernetes External Secrets Operator.
  • Test rotation before you need it: Rotation under incident pressure is where mistakes happen. Practice rotating secrets in staging environments.
  • Revocation endpoints: For OAuth tokens and API keys, know how to revoke immediately. Document the revocation procedure for every secret type.

Best practice: The ideal secret lifetime is zero β€” meaning dynamic secrets that are created on demand and expire automatically. If you cannot achieve that, short rotation cycles are the next best thing.

Detecting secrets in code and history

Prevention is the goal, but detection is the safety net. Automated secret scanning should run at multiple points in your development workflow.

Where to run secret scanning

  • Pre-commit hooks: Catch secrets before they enter git history. Tools: detect-secrets, gitleaks, trufflehog.
  • Pull request / MR scanning: Scan every code change before it merges. GitHub Advanced Security, GitLab Secret Detection, and AquilaX all support this.
  • Full repository history scan: Run periodically to catch secrets that were missed earlier. trufflehog and gitleaks can scan the full git history.
  • IDE scanning: Real-time feedback in the editor before a developer even saves a file.
.pre-commit-config.yamlYAML
repos:
  - repo: https://github.com/Yelp/detect-secrets
    rev: v1.4.0
    hooks:
      - id: detect-secrets
        args: ['--baseline', '.secrets.baseline']

False positives: Secret scanners will flag test fixtures and example values. Maintain a baseline file or allowlist for known false positives β€” but review every item you add to the allowlist carefully.

Secrets management checklist

  • All secrets stored in a centralised vault, not in code or config files
  • .env files in .gitignore β€” only .env.example committed
  • CI/CD uses platform secrets store or OIDC federation β€” no static keys in pipeline YAML
  • Secret scanning runs at pre-commit, PR, and scheduled full-history scan
  • All secrets have defined maximum lifetimes with automated rotation
  • Secret access is logged and audited
  • Revocation procedures documented and tested for every secret type
  • Different secrets for each environment β€” no prod secrets in dev
  • Least privilege: each service can only access the secrets it needs
  • Build artefacts (Docker images, binaries) scanned for embedded secrets

Catch secrets before they reach production

AquilaX secret scanning detects API keys, tokens, and credentials in source code β€” in your IDE, on every PR, and across your full git history.

Explore secret scanning β†’