Secrets in Git, Done Right
Secrets in Git, Done Right
Across a few posts now I have circled the same question from different
sides. Storing passwords is about the secret at rest. Kubernetes Secrets
are not secret because base64 is not encryption. Azure managed identities
remove the stored secret by making the identity the credential. Terraform
state holds secrets in the clear. Each one gestures at a question none of
them answers directly: when you do have a secret, a database password, an
API token, a TLS private key, and your whole workflow is git, where does
it actually live?
The wrong answers are everywhere. Plaintext in the repo, which is a leak
the moment anyone clones it and forever in the history. A gitignored file
passed around the team over Slack, which is unversioned, unauditable, and
out of date on everyone's laptop. A secret nobody committed and nobody can
find when the one person who had it is on holiday. GitOps makes this sharp:
the whole model is "git is the source of truth," and a source of truth with
a hole where the secrets should be is not much of one.
SOPS with age is the answer I reach for, and it is elegant enough to be
worth a post of its own.
The Idea: Commit the Ciphertext
SOPS, Secrets Operations, started at Mozilla and is now a CNCF project. Its
idea is simple and exactly right: encrypt the secret values in a file,
commit the encrypted file to git, and decrypt it only at the moment of use.
The secret lives in your repository, versioned and reviewable like
everything else, but as ciphertext that is useless to anyone without the
key.
The detail that makes it pleasant rather than painful is that SOPS encrypts
only the values, not the whole file. An encrypted Kubernetes secret or a
config file keeps its structure: the keys, the field names, the shape all
stay readable plaintext, and only the sensitive values become
ENC[AES256_GCM,data:...]. That means a git diff still shows you that
someone changed the database password without showing you the password, and
a reviewer can see what kind of change happened. Compare that to encrypting
the whole file, where every change is an opaque blob and review is
impossible. SOPS keeps the thing git is good at, meaningful diffs, while
removing the thing git is bad at, holding secrets.
age: The Key Half, Kept Simple
SOPS needs something to do the actual encryption, and historically that
meant PGP, which is a notorious swamp of key management, expiry, and
tooling that fights you. The modern answer, and now the preferred one, is
age, by Filippo Valsorda. age is a small, modern encryption tool with no
configuration and no options to get wrong. A keypair is two short strings:
a public key you encrypt to, a private key you decrypt with. That is the
whole mental model.
The combination is the point. age provides dead-simple, modern public-key
encryption with keys you can actually manage; SOPS provides the
file-format awareness, the value-only encryption, and the git-friendly
workflow on top. You encrypt a file to one or more age public keys, commit
the result, and anyone holding a matching private key can decrypt it.
Nobody else can, including git itself, including whoever hosts your
repository.
Why This Fits a Team
The property that makes this work beyond one person is that SOPS can
encrypt a file to several recipients at once. Every team member generates
their own age keypair and shares only their public key. You encrypt the
secrets to all of their public keys, and each person decrypts with their
own private key. No shared master password that everyone knows and nobody
can rotate, no secret handed around, just a list of public keys that are
safe to commit.
This is where it gets genuinely nice operationally. Onboarding someone is
adding their public key to the config and re-encrypting. Offboarding is
removing their key and re-encrypting, after which they can no longer
decrypt new versions even though they still have the old private key,
because they are no longer a recipient. Your CI system gets its own
keypair, so the pipeline can decrypt at deploy time without any human key
being involved. A .sops.yaml file in the repo maps paths to keys, so
secrets under production/ encrypt to the production recipients and
staging/ to the staging ones, automatically, and nobody has to remember
which key to use. The whole access model becomes a reviewable list of
public keys in a file, which is exactly the kind of thing git and code
review are built to manage.
How It Runs in Practice
The workflow is small. A developer edits a secret with SOPS, which decrypts
it in memory, lets them change it, and re-encrypts on save, so plaintext
never touches the disk. They commit the encrypted file. At deploy time, the
decryption happens wherever the secret is actually needed: a GitOps
controller like Flux decrypts with a key held in the cluster and applies
the result, or a CI pipeline decrypts with its key and hands the value to
whatever consumes it. The plaintext exists only transiently, at the point
of use, and never in the repository or in git history.
That is the entire shape: encrypted in git, decrypted at the edge, in
memory, by whoever legitimately holds a key. The secret has a full version
history, every change is in the commit log attributable to a person, and
the file is as auditable as the rest of your infrastructure code, because
it is part of it.
SOPS or Vault: Not the Same Question
The reflexive objection is "why not just use Vault," and the answer is that
they solve different problems and the distinction matters.
SOPS keeps the secret in your git repository, encrypted, and decrypts it at
deploy. The secret is a file you own, with no service to run, no API to be
available at deploy time, nothing to keep highly available. HashiCorp Vault
is a running service that stores secrets behind an API and serves them
dynamically, and it does things SOPS cannot: dynamic short-lived
credentials minted on demand, leasing and automatic revocation, a rich
audit log of every access. That power costs you an always-on, highly
available service to operate, secure, and depend on at the exact moment you
are deploying.
The honest split: for a small platform, a homelab, a GitOps repo, a team
that wants secrets versioned alongside their code with no extra
infrastructure, SOPS with age is the right answer and Vault is
over-engineering. For dynamic database credentials, for a regulated
environment that needs per-access audit and automatic rotation of
short-lived secrets, Vault earns its operational weight. They also compose,
SOPS can use Vault as a key backend, but most teams reaching for SOPS are
reaching for it precisely because they do not want to run Vault, and that
is a legitimate choice, not a compromise.
The One Thing You Must Not Get Wrong
All of this rests on one point of failure: the age private keys, and
especially the master key or the CI key that can decrypt everything. SOPS
moves the problem from "protect every secret" to "protect the keys that
decrypt the secrets," which is a smaller and better-defined problem, but it
is still the problem, and the whole scheme is exactly as strong as how you
guard those keys.
The private key never goes in git, obviously, it is the one file that must
stay out, held in a password manager, a secure secret store, or a
hardware-backed key. The CI key is the sharp one, because it can decrypt
everything and lives in automation: scope it, store it in the CI platform's
protected secret store, and treat its exposure as a full compromise
requiring re-encryption of everything under new keys. And because
re-encrypting to a new recipient set is how both rotation and offboarding
work, make sure you can actually do it smoothly before you need to do it in
a hurry.
The Point
Secrets in git is not a thing to avoid, it is a thing to do correctly, and
SOPS with age is how. Encrypt the values, commit the ciphertext, keep
readable diffs, decrypt only at the point of use. age gives you modern
key management without the PGP misery; SOPS gives you the git-friendly,
value-only, multi-recipient workflow on top. A team's whole access model
becomes a reviewable list of public keys, onboarding and offboarding become
a re-encrypt, and your secrets get the same version history and audit trail
as the infrastructure code they belong with.