Skip to content

Secrets Management

Architecture

Secrets follow a three-layer model: Bitwarden (source of truth), SOPS+age (repo-encrypted), and Kubernetes Secrets (runtime).

flowchart TB
    BW["Bitwarden Vault<br/>(source of truth)"]
    SOPS["SOPS + age<br/>(repo-encrypted files)"]
    K8s["Kubernetes Secrets<br/>(runtime, encrypted at rest)"]
    TF["OpenTofu State<br/>(SOPS-encrypted)"]

    BW -->|"manual export"| SOPS
    SOPS -->|"ansible decrypt"| K8s
    SOPS -->|"tofu reads"| TF

    style BW fill:#1a5276,stroke:#2980b9,color:#fff
    style SOPS fill:#7d6608,stroke:#f1c40f,color:#fff

Bitwarden Items

All master credentials are stored in Bitwarden under the LRON folder:

Item Type Contents
Hetzner API Token API credential Cloud API token for OpenTofu
Hetzner DNS Token API credential DNS API token for zone management
age Secret Key Secure note SOPS decryption key (AGE-SECRET-KEY-...)
WireGuard Keys Secure note Hub + Home private/public key pairs
K3s Node Token Secure note Cluster join token
Rancher Bootstrap Login Initial admin password
Authelia Users Secure note User database YAML (hashed passwords)
SSH Deploy Key SSH key ed25519 key pair for all VMs
TOTP Seed TOTP Authelia TOTP seed (also in authenticator app)

SOPS + age Flow

Encryption

# .sops.yaml in repo root
creation_rules:
  - path_regex: secrets/.*\.yaml$
    age: "age1..."  # Public key (safe to commit)
  - path_regex: tofu/.*\.tfstate$
    age: "age1..."

# Encrypt a new secret
sops --encrypt --in-place secrets/wireguard.yaml

# Edit an encrypted file (decrypts in $EDITOR, re-encrypts on save)
sops secrets/wireguard.yaml

# Decrypt for use in scripts
sops --decrypt secrets/wireguard.yaml

How SOPS Encrypts YAML

SOPS encrypts values but leaves keys in cleartext, making diffs readable:

# What's committed to Git:
wireguard:
    private_key: ENC[AES256_GCM,data:...,iv:...,tag:...,type:str]
    public_key: ENC[AES256_GCM,data:...,iv:...,tag:...,type:str]
    preshared_key: ENC[AES256_GCM,data:...,iv:...,tag:...,type:str]
sops:
    age:
        - recipient: age1...
          enc: |
            -----BEGIN AGE ENCRYPTED FILE-----
            ...

Persistent State Paths

Only these paths contain state that must survive a full rebuild:

Path Location Contents Backup
secrets/ Git repo SOPS-encrypted secrets Git + GDrive mirror
tofu/terraform.tfstate Git repo SOPS-encrypted infra state Git + GDrive mirror
/var/lib/rancher/k3s/server/ Hub VM K3s server data, etcd Recovery procedure
/etc/wireguard/ Hub VM WireGuard config SOPS copy in repo
~/.beast-session.log Hub VM Beast cost tracking Expendable

Hub VM is the only pet

Hub and DMZ are long-lived but reproducible. Their /var/lib/rancher/k3s/ directories are the only state not fully captured in Git. The recovery procedure recreates everything else from code.

Key Rotation Procedures

age Key Rotation

  1. Generate new age key: age-keygen -o new-key.txt
  2. Add new public key to .sops.yaml
  3. Re-encrypt all files: find secrets/ -name '*.yaml' -exec sops updatekeys {} \;
  4. Re-encrypt tofu state: sops updatekeys tofu/terraform.tfstate
  5. Update Bitwarden with new private key
  6. Remove old public key from .sops.yaml
  7. Commit changes

WireGuard Key Rotation

  1. Generate new key pair: wg genkey | tee privatekey | wg pubkey > publickey
  2. Update secrets/wireguard.yaml via SOPS
  3. Run Ansible to deploy new config to Hub
  4. Update local WireGuard client config
  5. Update Bitwarden

K3s Node Token Rotation

  1. Stop K3s on all agent nodes
  2. Regenerate token on server: edit /var/lib/rancher/k3s/server/node-token
  3. Update secrets/k3s.yaml via SOPS
  4. Restart K3s server
  5. Re-join agent nodes with new token
  6. Update Bitwarden

SSH Key Rotation

  1. Generate new key: ssh-keygen -t ed25519 -f ~/.ssh/lron_ed25519_new
  2. Add new public key via Ansible to all VMs
  3. Update hcloud_ssh_key.deploy in OpenTofu
  4. Apply: tofu apply
  5. Verify access with new key
  6. Remove old key from authorized_keys via Ansible
  7. Update Bitwarden