feat(bastion): add cluster-wide SSH bastion as Terraform - #208
Merged
Conversation
The linuxserver/openssh-server image runs `sshd -f /config/sshd/sshd_config` and only includes drop-ins from /config/sshd/sshd_config.d/, so the config mounted at /etc/ssh/sshd_config.d/ was silently ignored: AllowTcpForwarding stayed `no` (bastion could not port-forward/ProxyJump) and none of the hardening applied. It also appended keys to a PVC-persisted authorized_keys and never removed them, so key removal did not revoke access. - Mount the sshd drop-in under /config/sshd/sshd_config.d/ (verified via sshd -T) - Serve an authoritative concatenated authorized_keys via AuthorizedKeysFile instead of the append-only PUBLIC_KEY_DIR flow, so removals revoke access - Add checksum annotations so key/config edits roll the pod - Pin storage_class_name=managed-csi (avoid Pending PVC on missing default) - Set externalTrafficPolicy=Local to preserve client IPs for the audit log - Single-source the login user (USER_NAME/AllowUsers) via local.bastion_user
The image boots sshd even when our drop-in is ignored (wrong mount path, an image change to the include behaviour, etc.), silently serving with insecure defaults. Add exec probes that dump the running config (`sshd -T`) and require every directive we ship to be present: - readinessProbe: pull the pod from the LoadBalancer endpoints on mismatch, so no connections are routed to a misconfigured bastion - startupProbe + livenessProbe: crash-loop the pod instead of serving The expected directive list is derived from the drop-in (lowercased, comment-stripped) so the assertion can't drift from the config. Verified against the image: passes with the correct mount, fails on wrong path / no drop-in / a tampered directive.
`sshd -T` without `-h` loads host keys from the stock location (root-owned 0600) and exits "no hostkeys available" when the exec probe runs as a non-root user, which made the startup probe report every directive missing and crash-loop a correctly-configured pod. Pass the host keys explicitly (as the image's service run script does) so the config dump succeeds regardless of the probe's user. Verified as both root and uid 1000: passes with the correct mount, fails on wrong path / no drop-in / tampered directive.
The bastion had grown into ~280 lines of raw kubernetes_* HCL with an inline sshd config heredoc and an embedded shell assertion script — awkward in HCL and inconsistent with every other cluster component here (traefik, argocd, cert-manager, ...), which are helm_release. Extract it into a local chart and apply it the same way. - base-infrastructure/charts/ssh-bastion: ns (via create_namespace), configmaps, statefulset, service; sshd drop-in and the fail-closed assertion are real files - the assertion now derives expected directives from the mounted drop-in at runtime, so it can't drift from the shipped config - image pin (renovate-annotated) and team public keys live in values.yaml - helm-unittest suite locks in the drop-in mount path, AuthorizedKeysFile, probe wiring, checksum-driven rollout, storage class and service shape - bastion.tf shrinks to the public IP + a helm_release passing environment, cpu request, loadBalancerIP and resource group Behaviour is unchanged; verified via helm lint, helm unittest and terraform validate.
Per review: the authorized keys stay in resources/bastion.tf (passed to the chart as values), and cluster/cloud-specific settings are injected by Terraform rather than baked into the chart: - chart values default keys: [], storageClass: "" (omitted -> cluster default StorageClass), and a generic service.annotations map (no hardcoded Azure key) - Terraform passes the keys list, storageClass=managed-csi, the reserved loadBalancerIP and the azure-load-balancer-resource-group annotation - tests updated: storageClassName present when set / omitted when unset, and the LB annotation via the generic map Verified: helm lint, helm unittest (9/9), terraform validate.
Drop the single-use locals block; the keys list lives directly in the values passed to the chart.
Validated the chart on a local kind cluster (NodePort, real SSH login and port-forward through the bastion) and ran authenticated + black-box security audits. Black-box surface is solid (key-only, root unreachable, MaxAuthTries 3, no user enumeration, PQ crypto). Applied the safe authenticated-audit fixes: - AllowAgentForwarding no — the comment claimed agent forwarding was intentionally off but it was never actually set; enforce it (also now part of the fail-closed assertion, since that derives from the drop-in) - automountServiceAccountToken: false — the bastion never calls the K8s API, so don't mount a cluster credential on an internet-exposed host - memory requests/limits as integer bytes (50Mi/200Mi); 0.05Gi/0.2Gi rendered as fractional-byte "…m" quantities that Kubernetes warns about - service.type/nodePort are now configurable (default LoadBalancer) so the chart can run on clusters without a cloud LB (kind, bare-metal) Deferred (need per-cluster decisions / image constraints, documented for follow-up): PermitOpen allowlist + egress NetworkPolicy to narrow the pivot surface, restrictive pod securityContext, host-key ownership (linuxserver image chowns /config to the login user), and pruning ECDSA host key / SHA-1 MACs. Verified: helm lint, helm unittest (9/9), terraform validate.
… limits) HIGH #1 (host-key exfil): the image runs sshd as the unprivileged login user, so the host private key is necessarily readable by that account — it can't be hidden from the user the shell runs as. Remove the exfil vector instead: make the bastion forwarding-only (forwardingOnly=true, default). PermitTTY no + ForceCommand /bin/false block interactive shell, command exec, and (ForceCommand also overrides subsystems) SFTP/SCP, while -N/-L/-D/-J/-W keep working. A key holder can forward but cannot read anything on the box. HIGH #2 (broad cluster pivot): - sshd.permitOpen: optional forward-destination allowlist (default unrestricted), sshd-enforced. - networkPolicy (opt-in; enabled in the Terraform values): ingress limited to the SSH port, egress limited to an allowlist (default RFC1918) so the public internet and cloud metadata (169.254.169.254) are denied by construction. Requires an enforcing CNI (azure/calico); harmless no-op otherwise. Also: service.type/nodePort configurable (default LoadBalancer) for non-cloud clusters. Verified on a local kind cluster: pod Ready (fail-closed assertion accepts the new directives), interactive shell + scp + sftp all blocked, host key NOT exfiltrable, port-forward to the kube API still works, PermitOpen allows the listed target and resets a non-listed one, NetworkPolicy renders and is accepted by the API. helm lint, helm unittest (15/15), terraform validate all pass.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
Breaking change 🚨