Sealed Secrets Kubernetes Tutorial: Encrypt Secrets for Git

You have a GitHub repo for your homelab Kubernetes cluster. You've built an HA control plane with kube-vip, planned your IPs, installed k3s, and mapped the bootstrap order. Now you need a way to manage credentials: API keys, database passwords, OIDC client secrets. These are things that should never be committed to source control.

Or should they?

Here's the thing that should make you nervous: Kubernetes Secrets are base64-encoded, not encrypted (Kubernetes docs, 2025). Base64 encoding is not an encryption method (Kubernetes Good Practices, 2025).

echo cGFzc3dvcmQ= | base64 -d
# password

That's it. One command. Zero secrets. If you commit a plain Kubernetes Secret to a public repo, you've published your passwords to the internet. Base64 is encoding. It's the same data in a different alphabet. It was never meant to protect anything.

What's this? Oh no! There are secrets in MY repo!

And yet, they're SealedSecrets. Ciphertext. Safe to commit, impossible to decrypt without your cluster's private key. This sealed secrets Kubernetes tutorial walks through the exact workflow from the homelab-k8s repo: deploy the controller, seal a secret, commit it, back up the key, and sleep easy knowing your credentials are safe even though your repo is public.

Key Takeaways

• Kubernetes Secrets are base64-encoded, not encrypted. Committing plain Secrets to a public repo is functionally identical to posting your passwords on Twitter (Kubernetes docs, 2025). Sealed Secrets replaces the data block with RSA-OAEP ciphertext that only your cluster can decrypt.
• Sealed Secrets (Bitnami, over 9,500 GitHub stars, listed on the CNCF Landscape) encrypts secrets with a key that lives INSIDE your cluster. One controller, one CLI tool, one key to back up, zero external services.
• kubeseal is a UNIX pipeline tool: stdin → stdout. Write your Secret, pipe it through kubeseal, commit the SealedSecret. The plaintext never touches a committed file.

How It Works in Practice

• The controller does the decryption transparently. Pods mount the Secret as usual and have no idea it came from an encrypted source.
• ArgoCD applies SealedSecrets like any other Kubernetes resource. The controller decrypts them out-of-band. The sync wave (-3) ensures the controller exists before any SealedSecret is applied. Without this ordering, the API server rejects the unknown resource.
• If you lose the private key, every SealedSecret in your repo becomes permanently unreadable. Back it up. One command. Store it in Bitwarden. If you rebuild your cluster, restore the key BEFORE applying SealedSecrets.

The Problem: Why You Can't Commit Kubernetes Secrets to Git

Kubernetes stores Secrets as base64-encoded data, not encrypted (Kubernetes docs, 2025). Anyone who can read the YAML can decode the secret in one shell command. The data block in a Secret is reversible by anyone with base64 (which is everyone), by design. Kubernetes Secrets protect against accidental exposure in kubectl describe output, not against intentional access to the YAML. This isn't a bug. It's just not what you think it is.

And your homelab repo may be public. Mine is. It's a portfolio piece, a reference for other homelabbers, an open-source contribution to the community. But it needs credentials. cert-manager needs DNS-01 API keys. Databases need passwords. Apps need OIDC client secrets. If those aren't in git, ArgoCD can't deploy them. If they ARE in git as plain Secrets, you've published your passwords to the internet.

The other options aren't great:

1. Don't commit secrets, create them manually. Works until you rebuild your cluster. Then you're recreating 20+ secrets from a runbook you may or may not have updated. Error-prone and slow.
2. Make the repo private. Limits sharing and portfolio value. The point of the public repo is that other homelabbers can learn from it. Plus, this violates Rule #1: NEVER COMMIT SECRETS TO SOURCE CONTROL. Ever. Period. Full Stop.
3. Encrypt the secrets so they're safe in a public repo. This is Sealed Secrets.

The GitOps requirement makes option 1 a non-starter. If secrets aren't in git, ArgoCD can't deploy them. You'd need a side-channel: a separate vault, manual kubectl commands, external scripts. This breaks the "git push deploys everything" promise. Sealed Secrets keeps the promise. Secrets ARE in git, ArgoCD applies them, the controller decrypts them, pods mount them. Nobody in the chain except the controller knows encryption was involved.

My repo is public. It has ALWAYS been public. Every credential for 18+ apps lives in that repo as a SealedSecret. Route53 API keys, database passwords, OIDC client secrets: all ciphertext, all committed, all deployed by ArgoCD. This isn't a tutorial pattern. This is my actual cluster. If you read [k3s Bootstrap Order: Why Sequence Matters]https://taegost.com/2026/06/k3s-bootstrap-order-why-sequence-matters-gitops-and-argocd-primer/), you know Sealed Secrets is step 1 after kube-vip. This post is the implementation.

How Does Sealed Secrets Encryption Work?

Sealed Secrets uses asymmetric encryption: the controller holds an RSA key pair that never leaves the cluster. kubeseal (a CLI tool) fetches the public key and encrypts your Secret into a SealedSecret custom resource. The SealedSecret YAML contains only ciphertext in its encryptedData block, safe to commit anywhere.

When you apply a SealedSecret to the cluster, the controller decrypts it using the private key and creates a standard Kubernetes Secret. Pods mount the Secret as usual. They have no idea it came from an encrypted source. The encryption is one-way: kubeseal encrypts, ONLY the controller decrypts. Even you, the cluster operator, cannot decrypt a SealedSecret without the controller.

Here's each piece of the flow:

The RSA Key Pair

The controller generates a key pair on first start. A Kubernetes Service exposes the public key at /v1/cert.pem on the controller's HTTP endpoint. The controller stores the private key as a Kubernetes Secret in kube-system. The public key encrypts, the private key decrypts. Standard RSA-OAEP with AES-256-GCM hybrid encryption (Bitnami Sealed Secrets crypto docs, RFC 8017, NIST SP 800-38D).

The kubeseal Encryption Step

kubeseal reads your plain Secret YAML, fetches the controller's public key (or uses a cached copy), encrypts each value in the data or stringData block, and outputs a SealedSecret resource. The encryptedData block replaces data. The SealedSecret is a custom resource (SealedSecret.bitnami.com/v1alpha1) that only the controller understands. To everyone else, it's opaque ciphertext.

The Controller Decryption Step

The controller watches for SealedSecret resources. When it sees one, it uses its private key to decrypt each entry in encryptedData, creates a standard Secret with the decrypted values in the same namespace, and sets the SealedSecret's status to Synced. The resulting Secret is identical to what you would have created manually. Same keys, same values, same namespace.

Namespace Scope

SealedSecrets are namespace-scoped. A SealedSecret in namespace traefik creates a Secret in namespace traefik. The controller can be configured to allow cross-namespace decryption via annotations, but the default is strict namespace isolation. This means a SealedSecret committed to the wrong namespace won't decrypt. The controller enforces namespace boundaries for you.

Key Rotation

The controller supports key rotation: generating a new key pair while keeping the old one for decryption. New secrets are sealed with the new key, old secrets continue to decrypt with the old key. Over time, re-seal old secrets with kubeseal --re-encrypt and remove the old key. This is a Day 2 operation covered in the homelab-k8s docs. This post focuses on initial setup.

How Do You Deploy Sealed Secrets on k3s?

Deploying Sealed Secrets is a single kubectl apply. The controller ships as a self-contained manifest with Deployment, RBAC, and CRDs. Apply it, wait for the controller to be ready, and install kubeseal on your workstation.

# Deploy the controller from the latest release manifest
kubectl apply -f apps/sealed-secrets/sealed-secrets-controller.yaml

This deploys the controller into kube-system. The manifest includes: the CRD (SealedSecret resource definition), the controller Deployment, RBAC (the controller needs permission to manage secrets in all namespaces), and a Service to expose the public key endpoint.

Wait for the controller to be ready:

kubectl rollout status deployment sealed-secrets-controller -n kube-system

Verify the controller is working before sealing anything. I learned this the hard way: the first time I deployed Sealed Secrets, I immediately sealed a secret before checking the controller logs. The controller hadn't finished generating its key pair yet. The seal succeeded (kubeseal fetched a partial response), but the controller rejected the ciphertext. Now I always run the verification commands first.

Check the logs:

kubectl logs -n kube-system -l app.kubernetes.io/name=sealed-secrets

You should see "controller started" and "fetching key." The controller generates its key pair automatically on first start if none exists.

Check that the key secret exists:

kubectl get secret -n kube-system -l sealedsecrets.bitnami.com/sealed-secrets-key

You should see one secret with a name like sealed-secrets-keyXXXXX. This is the private key, the master key for every SealedSecret in your cluster.

Back Up the Private Key

If you lose this key, every SealedSecret in your repo becomes permanently unreadable. The ciphertext is RSA-OAEP. There's no recovery, no backdoor, no "reset my password" link. The backup is one command:

kubectl get secret -n kube-system \
  -l sealedsecrets.bitnami.com/sealed-secrets-key \
  -o yaml > main.key

Store main.key in Bitwarden (or your password manager). Delete the local copy: rm main.key. The backup contains both the public and private keys. No separate backups needed.

The disaster recovery path. If you rebuild your cluster, restore the key before applying any SealedSecrets:

# After restoring main.key from Bitwarden
kubectl apply -f main.key   # Restore the key
kubectl apply -f apps/sealed-secrets/sealed-secrets-controller.yaml  # Deploy controller
# Now apply your SealedSecrets — the controller finds the existing key

If you forget to restore the key first, the controller generates a NEW key pair, and every existing SealedSecret fails to decrypt. The fix: delete the new key secret, apply the backup, restart the controller. The post includes this DR flow so you know what to do when — not if — you rebuild.

Install kubeseal

The kubeseal CLI is needed on your workstation to seal secrets.

# macOS
brew install kubeseal

# Linux (via apt)
sudo apt install kubeseal

# Or download the binary directly
# https://github.com/bitnami-labs/sealed-secrets/releases

Verify the install:

kubeseal --version

kubeseal supports two modes. Online mode (simplest): run kubeseal without --cert. It auto-fetches the public key from the controller at runtime. Offline mode (CI/CD, machines without cluster access): fetch and cache the cert once:

kubeseal --fetch-cert \
  --controller-namespace=kube-system \
  --controller-name=sealed-secrets-controller \
  > pub-cert.pem

Then use --cert pub-cert.pem for subsequent seals. Both modes produce identical output. The SealedSecret is the same ciphertext either way.

How Do You Seal a Secret with kubeseal?

The kubeseal workflow: write your Secret YAML, seal it with kubeseal, apply the SealedSecret, verify the controller decrypted it, then delete the plaintext. Never delete the plaintext before verifying the SealedSecret works. If the ciphertext is corrupt or you used the wrong public key, you'll need the plaintext to re-seal.

Naming Convention

Adopt a consistent naming pattern BEFORE you start sealing. The homelab-k8s repo uses:

Example: secret-authentik.yaml → seal → sealedsecret-authentik.yaml. This makes it obvious at a glance which is which.

The prefix convention is load-bearing for .gitignore. The repo blocks *-secret.yaml, *-secrets.yaml, *.secret.yaml, and secret-*.yaml (all plaintext patterns) while whitelisting !*-sealedsecret.yaml, !*-sealedsecrets.yaml, and !sealedsecret-*.yaml. The ! negation ensures SealedSecret files are allowed through despite matching the broader block. If you swap the naming convention, your .gitignore won't catch plaintext secrets.

Step 1 — Write the Secret YAML

Create a standard Kubernetes Secret. Use stringData for human-readable values (Kubernetes base64-encodes it at apply time):

# secret-myapp.yaml
apiVersion: v1
kind: Secret
metadata:
  name: myapp-credentials
  namespace: default
stringData:
  api-key: "sk-1234567890abcdef"
  db-password: "super-secret-database-password"

Do NOT commit this file. You'll delete it after verification.

Step 2 — Seal It

Online mode (with cluster access):

kubeseal --format yaml < secret-myapp.yaml > sealedsecret-myapp.yaml

kubeseal auto-fetches the controller's public key at runtime.

Offline mode (CI/CD, no cluster access):

kubeseal --format yaml --cert pub-cert.pem \
  < secret-myapp.yaml > sealedsecret-myapp.yaml

Both produce identical output: apiVersion: bitnami.com/v1alpha1, kind: SealedSecret, and an encryptedData block containing ciphertext. The stringData block is gone, replaced by encryptedData with values like api-key: AgB4k... (a long base64-encoded RSA ciphertext blob).

Step 3 — Apply the SealedSecret

Commit and push:

git add sealedsecret-myapp.yaml
git commit -m "Add sealed secret for myapp"
git push

If applying manually: kubectl apply -f sealedsecret-myapp.yaml. If using ArgoCD: ArgoCD picks up the new file on the next sync cycle. The SealedSecret is the committed artifact. It's ciphertext, safe for a public repo.

Step 4 — Verify Decryption

BEFORE deleting the plaintext, verify the controller created the Secret:

kubectl get secret myapp-credentials -n default

Check the decrypted value matches:

kubectl get secret myapp-credentials -n default \
  -o jsonpath="{.data.api-key}" | base64 -d

Should output sk-1234567890abcdef.

Check the SealedSecret status:

kubectl get sealedsecret myapp-credentials -n default

Should show Synced. If it shows ErrDecrypt, the plaintext is still on disk. Fix the issue and re-seal. Common causes: sealed with a different cluster's public key, malformed YAML, namespace mismatch.

Step 5 — Delete the Plaintext

Only after successful verification:

rm secret-myapp.yaml

The plaintext secret is a temporary local artifact. It exists for the duration of the sealing pipeline and is deleted once the round-trip is confirmed.

Step 6 — Protect with .gitignore

Add these rules to your repo's .gitignore:

# Block plaintext secret patterns
*-secret.yaml
*-secrets.yaml
*.secret.yaml
secret-*.yaml

# Allow SealedSecret patterns
!*-sealedsecret.yaml
!*-sealedsecrets.yaml
!sealedsecret-*.yaml

# Private key backup
main.key

This is a safety net, not a substitute for the workflow. Even with .gitignore, follow the seal, apply, verify, delete loop. The .gitignore protects against the one time you forget.

Common Mistakes

How Does ArgoCD Work with Sealed Secrets?

ArgoCD applies SealedSecret resources like any other Kubernetes manifest. It doesn't know or care that the resource contains encrypted data. The Sealed Secrets controller watches for SealedSecrets independently and decrypts them into Secrets. This means your GitOps pipeline has no idea encryption is involved.

ArgoCD sees a SealedSecret, applies it, and reports "Synced." The controller does the decryption asynchronously. ArgoCD doesn't wait for it and doesn't track it. If the controller fails to decrypt (wrong key, malformed ciphertext), the SealedSecret shows status: ErrDecrypt but ArgoCD still shows "Synced." The resource was applied, decryption is a runtime concern.

The sync wave (-3). Sealed Secrets deploys at sync wave -3, the lowest wave number in the stack. This ensures the controller exists and is running BEFORE any SealedSecret resource (wave -2 and above) is applied. If ArgoCD tried to apply a SealedSecret before the controller's CRD existed, the API server would reject the unknown resource and the sync would fail. Wave ordering eliminates this race condition entirely.

I learned this the hard way. On my first GitOps bootstrap, I deployed Sealed Secrets at the same wave as everything else. ArgoCD applied the Application for cert-manager (which referenced a SealedSecret) BEFORE the Sealed Secrets controller CRD was registered. The API server rejected the SealedSecret, ArgoCD marked the Application as degraded, and nothing else deployed. Moving Sealed Secrets to wave -3 fixed it. The other five components deploy in sequence. Sealed Secrets deploys first.

Verification after sync. After ArgoCD syncs, check the SealedSecret status:

kubectl get sealedsecret -A

All should show Synced. If any show ErrDecrypt, check the controller logs:

kubectl logs -n kube-system -l app.kubernetes.io/name=sealed-secrets --tail=50

Common cause: secret was sealed with a different cluster's public key. Fix: re-seal with the correct key.

The self-managed pattern. Once ArgoCD manages the cluster, ArgoCD manages the Sealed Secrets controller itself (via the apps/manifests/sealed-secrets.yaml Application resource). If you update the controller manifest, ArgoCD applies the update. If the controller is deleted, ArgoCD restores it. The controller manages secrets; ArgoCD manages the controller. This is infrastructure as code: the tool that protects your secrets is itself defined in git.

Frequently Asked Questions

What happens if I lose the Sealed Secrets private key?

Every SealedSecret in your repo becomes permanently unreadable. The ciphertext is RSA-OAEP. There's no practical way to decrypt without the private key. Your cluster continues running (existing Secrets aren't affected), but you can't create new Secrets from your committed SealedSecrets. Back up the key: kubectl get secret -n kube-system -l sealedsecrets.bitnami.com/sealed-secrets-key -o yaml > main.key. Store it in Bitwarden. This is not optional.

Can I use Sealed Secrets with multiple clusters?

Yes, but each cluster needs its own key pair. A SealedSecret encrypted for Cluster A cannot be decrypted by Cluster B. If you share a repo across clusters, maintain separate SealedSecret files per cluster, or use the controller's --additional-namespaces flag for cross-namespace decryption from a central controller. For a single homelab cluster: one cluster, one key pair.

How is this different from SOPS, Vault, or External Secrets Operator?

SOPS encrypts per-value with a cloud KMS key (AWS KMS, etc.), so you need a cloud account. Vault is a full secrets platform. Powerful, but requires running and maintaining Vault itself (HA, storage backend, unseal, backups). External Secrets Operator syncs secrets FROM an external provider (AWS, GCP, 1Password) TO Kubernetes. The external provider is the source of truth, not git. Sealed Secrets makes GIT the source of truth. The secret lives in your repo as ciphertext, and the controller creates the Kubernetes Secret from it. For a solo homelab with a public repo: one controller, one CLI tool, one key, zero external services.

Do I need to re-seal secrets when the encryption key rotates?

No. The controller keeps every private key it has ever generated, so existing SealedSecrets continue to decrypt indefinitely after a key rotation. New secrets are sealed with the new key; old secrets keep working with the old key. You can re-seal old secrets with kubeseal --re-encrypt if you want everything on the latest key (and eventually remove the old key from the controller), but it's optional cleanup, not a requirement. The homelab-k8s docs cover key rotation in detail.

Can I seal a secret without kubectl access to the cluster?

Yes. Fetch the public key once: kubeseal --fetch-cert > pub-cert.pem. After that, seal secrets offline: kubeseal --format yaml --cert pub-cert.pem < secret.yaml > sealedsecret.yaml. The public key is all kubeseal needs to encrypt. You can seal secrets from any machine, commit them, and push. The controller decrypts them when they're applied to the cluster. This is the operational detail that makes Sealed Secrets practical for CI/CD pipelines.

Next Steps

You now have a working Sealed Secrets controller on your k3s cluster, your first encrypted secret committed to a public repo, and the private key backed up. The credentials your cluster needs (API keys, database passwords, OIDC client secrets) can now live in git as ciphertext, deployed by ArgoCD, and decrypted transparently by the controller.

But Sealed Secrets is infrastructure. The first thing that actually USES it comes next.

Coming up: cert-manager. Your Sealed Secrets controller is running. cert-manager needs it. The Route53 DNS-01 challenge credentials that let cert-manager issue wildcard TLS certificates are stored as a SealedSecret. You'll deploy cert-manager, create that SealedSecret, and watch it issue your first Let's Encrypt certificate automatically. That's the next post: "cert-manager on k3s: Automatic TLS for Every Service."

If you haven't read the bootstrap order overview, start there. k3s Bootstrap Order: Why Sequence Matters explains WHY Sealed Secrets is step 1 after kube-vip and maps the full dependency chain from here to ArgoCD.

The homelab-k8s repo has the actual controller manifest at apps/sealed-secrets/, the SealedSecret examples for every app, and detailed docs at docs/sealed-secrets.md. Don't trust me. Go look at the repo. It's public. It's real. It's running 18+ apps right now, all with secrets committed as ciphertext.

You just committed an encrypted secret to a public repo and slept easy. That's not a small thing. That's production.