Security Hardening

Defense in Depth

flowchart TB
    subgraph L1["Layer 1: Perimeter"]
        HFW["Hetzner Cloud Firewall"]
    end

    subgraph L2["Layer 2: Host"]
        UFW["UFW (stateful)"]
        CS["CrowdSec IDS"]
        SSH["SSH hardening"]
        EL["endlessh tarpit"]
    end

    subgraph L3["Layer 3: Kubernetes"]
        K3sH["K3s hardening flags"]
        SE["Secrets encryption"]
        CNP["Cilium NetworkPolicy"]
    end

    subgraph L4["Layer 4: Application"]
        Auth["Authelia TOTP"]
        TLS["TLS everywhere"]
        SC["Security contexts"]
    end

    L1 --> L2 --> L3 --> L4

    style L1 fill:#4a235a,stroke:#7d3c98,color:#fff
    style L2 fill:#1a5276,stroke:#2980b9,color:#fff
    style L3 fill:#7d6608,stroke:#f1c40f,color:#fff
    style L4 fill:#1e8449,stroke:#27ae60,color:#fff

CrowdSec + endlessh

CrowdSec

CrowdSec runs on Hub and DMZ as a log-based intrusion detection system.

Installed parsers:

Parser	Purpose
sshd-logs	SSH brute-force detection
caddy-logs	HTTP abuse detection (DMZ only)
linux-syslog	System authentication failures

Installed scenarios:

Scenario	Trigger	Ban Duration
ssh-bf	5 failed SSH in 30s	4 hours
ssh-slow-bf	10 failed SSH in 10min	24 hours
http-bad-user-agent	Known malicious UA	24 hours
http-path-traversal	Path traversal attempt	48 hours
http-bf-wordpress	WP login brute-force	24 hours

Bouncer: cs-firewall-bouncer adds iptables DROP rules for banned IPs.

Central API: Enrolled in CrowdSec Central API (free tier) for community blocklist sharing. Contributes local detections, receives global threat intelligence.

endlessh

endlessh runs on port 22 (Hub and DMZ) as an SSH tarpit. It sends an infinitely slow SSH banner, trapping automated scanners.

Parameter	Value
Listen port	22 (standard SSH port)
Real SSH port	2222
Max clients	4096
Delay	10000ms between lines
Max line length	32

Tarpit economics

endlessh wastes attackers' time and resources while consuming almost zero resources on the server. Most automated scanners target port 22 -- they get trapped while real SSH runs on 2222.

UFW Egress Rules

Each VM runs UFW with explicit egress rules (default deny outbound):

Hub Egress

Destination	Port	Protocol	Purpose
Any	443	TCP	HTTPS (Rancher, Helm, CrowdSec API)
Any	80	TCP	HTTP (package updates, Let's Encrypt)
Any	53	UDP/TCP	DNS
10.0.1.0/24	Any	Any	Private network (inter-node)
Any	51820	UDP	WireGuard (listen)
Any	123	UDP	NTP

DMZ Egress

Destination	Port	Protocol	Purpose
Any	443	TCP	HTTPS (Let's Encrypt, CrowdSec API)
Any	80	TCP	HTTP (package updates)
Any	53	UDP/TCP	DNS
10.0.1.0/24	Any	Any	Private network (inter-node)
Any	123	UDP	NTP

Beast Egress

Destination	Port	Protocol	Purpose
Any	443	TCP	HTTPS (container pulls, package updates)
Any	80	TCP	HTTP (package updates)
Any	53	UDP/TCP	DNS
10.0.1.0/24	Any	Any	Private network (inter-node)
Any	123	UDP	NTP

SSH Configuration

Applied via Ansible to all VMs (/etc/ssh/sshd_config):

Port 2222
PermitRootLogin no
PasswordAuthentication no
PubkeyAuthentication yes
AuthorizedKeysFile .ssh/authorized_keys
MaxAuthTries 3
LoginGraceTime 30
ClientAliveInterval 300
ClientAliveCountMax 2
X11Forwarding no
AllowTcpForwarding no
PermitTunnel no
AllowAgentForwarding no

Setting	Value	Rationale
Port	2222	Avoid port 22 scanners (endlessh runs there)
Root login	Disabled	All access via deploy user + sudo
Password auth	Disabled	Key-only authentication
Max auth tries	3	Limit brute-force window per connection
Login grace time	30s	Short window to authenticate
Client alive	300s interval, 2 retries	Drop idle connections after 10 minutes
Forwarding	All disabled	No tunneling or agent forwarding

K3s Hardening Flags

Flag	Purpose
--secrets-encryption	Encrypt secrets at rest in etcd using AES-CBC
--protect-kernel-defaults	Enforce required kernel parameters (CIS benchmark)
--kube-apiserver-arg="audit-log-path=..."	API audit logging
--kube-apiserver-arg="audit-log-maxage=30"	Retain audit logs for 30 days
--disable=traefik	Remove unnecessary attack surface
--disable=servicelb	Remove unnecessary attack surface

Secrets Encryption

K3s encrypts Kubernetes secrets at rest using an encryption config:

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      - aescbc:
          keys:
            - name: key1
              secret: <base64-encoded-key>
      - identity: {}

Encryption key management

The K3s secrets encryption key is stored in /var/lib/rancher/k3s/server/cred/encryption-config.json on the Hub VM. This file is critical -- if lost, encrypted secrets cannot be decrypted. It is backed up via SOPS in the repository.