From c1bb90a659eae90bae799e15af8edcc9dfe56f6d Mon Sep 17 00:00:00 2001 From: Boobalan Rathinam Date: Mon, 17 Aug 2026 15:50:18 -0700 Subject: [PATCH 1/2] feat(support-bundle): add consolidated support-bundle.sh collector Consolidate support-bundle-edge.sh (host + cluster) and support-bundle-infra.sh (cluster-only) into a single capability-driven script, per CONSOLIDATION-DESIGN.md. The legacy scripts are kept intact as fallbacks until the consolidated script is widely adopted. Highlights: - Never aborts mid-run: every collector runs through run_step and records OK / SKIP / FAIL / DENIED into collection-summary.txt inside the bundle; the only fatal errors are temp-dir/archive failures. - Advisory RBAC coverage: denied namespaces/resources are pruned and recorded with remediation guidance (namespace-coverage.txt) instead of exiting 1. - Capability probe: API-server reachability (10s timeout), edge-host and agent-mode detection, distro detection, and an ordered non-fatal kubeconfig resolver (flag, env, /run/kubeconfig, admin.conf, HOME, SUDO_USER home, rke2/k3s/canonical paths). - New flags on top of the union of both legacy flag sets: -k kubeconfig, -K host-only, -H cluster-only (no root required), -q quiet, -v version. - Archive to CWD by default (-d overrides), bundle named -- degrading to -; cluster name applied at tar time so the console.log tee pipe is never broken. - Fixes ported in passing: quoted DISTRO checks, ${DISTRO:-runtime} output paths, compgen-guarded pod-log globs, O(n^2) namespace pruning replaced with single-listing rebuild, rke2 now collects var-log-pods, cleanup can no longer delete the tarball it just produced, LOGNAME sanitized against shell-hostile hostnames. Docs: README-support-bundle.md added; README.md indexes the new script. Co-Authored-By: Claude Fable 5 --- support-bundle/README-support-bundle.md | 115 ++ support-bundle/README.md | 28 +- support-bundle/support-bundle.sh | 2052 +++++++++++++++++++++++ 3 files changed, 2193 insertions(+), 2 deletions(-) create mode 100644 support-bundle/README-support-bundle.md create mode 100644 support-bundle/support-bundle.sh diff --git a/support-bundle/README-support-bundle.md b/support-bundle/README-support-bundle.md new file mode 100644 index 0000000..6eeed84 --- /dev/null +++ b/support-bundle/README-support-bundle.md @@ -0,0 +1,115 @@ +# Consolidated Support Bundle Collection Script + +This Bash script (`support-bundle.sh`) collects diagnostic information from SpectroCloud environments. It consolidates the edge (`support-bundle-edge.sh`) and infrastructure (`support-bundle-infra.sh`) collectors into a single script that covers both scopes: host-level OS state and Kubernetes cluster state. + +Collection is **capability-driven**: every collector is gated on the capabilities it actually needs (root, journald, a container runtime, a reachable API server, …) and the script **never aborts mid-run** — a partial bundle always beats no bundle. The only fatal errors are failure to create the temporary directory or write the archive. + +## Quick Start + +```bash +# Full bundle on an edge or cluster host (host collection requires root) +sudo bash support-bundle.sh + +# Cluster-only bundle from any machine with kubectl access (no root needed) +bash support-bundle.sh -H + +# Host-only bundle (no Kubernetes collection) +sudo bash support-bundle.sh -K +``` + +## Prerequisites + +* **Host collection** (default): run as root (`sudo`). Use `-H` for a cluster-only bundle that does not require root. +* **Kubernetes collection** (default): `kubectl` access to the cluster. The kubeconfig is resolved automatically (see below) or can be passed explicitly with `-k`. +* All other tools (`journalctl`, `crictl`, `chronyc`, `etcdctl`, `helm`, GPU tooling, …) are optional — collectors that need a missing tool are recorded as `SKIP` and the run continues. + +## Available Flags + +All flags are optional: + +| Flag | Description | Example | +|------|-------------|---------| +| `-d` | Output directory for temporary storage and .tar.gz archive | `-d /var/tmp` | +| `-K` | Skip all Kubernetes collection (host-only bundle) | `-K` | +| `-H` | Skip all host collection (cluster-only bundle; no root required) | `-H` | +| `-q` | Suppress per-resource progress output, keep the summary | `-q` | +| `-v` | Print the support bundle version and exit | `-v` | +| `-s` | Start day of journald log collection (days before now) | `-s 7` | +| `-e` | End day of journald log collection (days before now) | `-e 5` | +| `-S` | Start date of journald log collection | `-S 2024-01-01` | +| `-E` | End date of journald log collection | `-E 2024-01-01` | +| `-l` | Number of log lines to collect from journald/crictl logs | `-l 500000` | +| `-j` | Additional journald logs to collect | `-j cloud-init,systemd-resolved` | +| `-k` | Path to an explicit kubeconfig | `-k /etc/kubernetes/admin.conf` | +| `-n` | Additional namespaces to collect | `-n hello-universe,hello-world` | +| `-r` | Additional namespace scoped resources | `-r certificates.cert-manager.io` | +| `-R` | Additional cluster scoped resources | `-R clusterissuers.cert-manager.io` | + +## Kubeconfig Resolution + +The first readable entry wins; resolution is never fatal: + +1. `-k ` flag +2. `$KUBECONFIG` environment variable +3. `/run/kubeconfig` +4. `/etc/kubernetes/admin.conf` +5. `$HOME/.kube/config` +6. The invoking user's `~/.kube/config` when running under `sudo` +7. `/etc/rancher/rke2/rke2.yaml`, `/etc/rancher/k3s/k3s.yaml`, `/var/snap/k8s/current/credentials/admin.conf` + +If none is found (or the API server is unreachable within a 10s probe), Kubernetes collection is recorded as `SKIP` and the rest of the bundle is still produced. + +## Collection Summary + +Every collector reports its outcome into `collection-summary.txt` inside the bundle (and to stdout at the end of the run): + +| Status | Meaning | +|--------|---------| +| `OK` | Collector ran successfully | +| `SKIP` | Precondition absent (tool not installed, not an edge host, scope disabled, …) | +| `FAIL` | Collector ran and errored | +| `DENIED` | Blocked by RBAC | + +## RBAC Coverage (advisory) + +Before Kubernetes collection, the script checks `kubectl auth can-i` for the required cluster-scoped resources and pod access per targeted namespace, and writes the result table to `namespace-coverage.txt` inside the bundle. + +Unlike previous script versions, **insufficient RBAC never aborts the run**: denied namespaces are pruned from collection, recorded as `DENIED`, and remediation guidance (a minimal ClusterRole snippet) is printed — then collection continues with what is accessible. + +## Output + +The script creates a compressed tarball named: + +* `--.tar.gz` when a cluster was reachable +* `-.tar.gz` otherwise + +The archive is written to the **current working directory** by default, to `-d ` when given, and to the temporary base directory as a last resort if neither is writable. Every bundle contains `console.log` (the full run transcript), `.support-bundle` (version, scopes, detected capabilities), and `collection-summary.txt`. + +## Collected Information + +Identical in scope to the union of the two legacy scripts: + +* **Host tier** (requires root; skipped with `-H`): system info, chronyd/time sync, networking (iptables/nft/ip/ss/CNI), `/var/log` and `/var/log/spectrocloud`, journald units (including previous-boot kernel log), storage state (block devices, LVM, NVMe SMART, device-mapper), GPU state (AMD ROCm + NVIDIA, with `kubectl exec` fallbacks into operator pods), edge agent files (`/oem`, `/run/stylus`, cloud-config, installer logs, bundle checksums), container runtime (crictl), and helm releases. +* **Kubernetes tier** (skipped with `-K`): cluster info and dump, cluster- and namespace-scoped resources, custom resources, helm release secrets, metrics, previous pod logs, and — on Enterprise/PCG clusters — MongoDB replica set status, per-pod disk usage, and database/collection sizes. +* **Distro tier** (host + detected distribution): kubeadm manifests/certs/etcd, k3s/rke2 pod logs and certs, Canonical snap k8s files/dqlite state. + +Secrets are not collected, except helm release secrets for the spectro namespaces. Certificates are captured parsed (`openssl x509 -text -noout`), never as raw keys. + +## Environment Variables + +* `KUBECONFIG`: Path to the Kubernetes configuration file (see resolution order above) +* `DEV`: When set, bypasses the root requirement (development/testing only) + +## Relationship to the Legacy Scripts + +`support-bundle-edge.sh` and `support-bundle-infra.sh` remain available as fallbacks until this script is widely adopted: + +* `support-bundle.sh` ≈ `support-bundle-edge.sh` (full scope, root required) +* `support-bundle.sh -H` ≈ `support-bundle-infra.sh` (cluster-only, no root) + +Behavioral differences vs. the legacy scripts: + +* Insufficient RBAC and a missing/unset `KUBECONFIG` no longer abort the run (previously fatal in both scripts). +* The archive is written to the current working directory by default (the edge script wrote it to the temporary base directory). +* The bundle name includes the cluster name when available (previously edge used hostname only, infra used cluster name only). +* Every run produces `collection-summary.txt` and, when a cluster is reachable, `namespace-coverage.txt`. diff --git a/support-bundle/README.md b/support-bundle/README.md index ce5c7e7..fb1420d 100644 --- a/support-bundle/README.md +++ b/support-bundle/README.md @@ -4,6 +4,24 @@ This repository contains a collection of scripts designed to gather diagnostic i ## Available Scripts +### Consolidated Support Bundle (preview) +- **Script**: `support-bundle.sh` +- **Purpose**: Single script covering both the edge (host + cluster) and infrastructure (cluster-only) scopes. Collection is capability-driven and never aborts mid-run: each collector records OK / SKIP / FAIL / DENIED into `collection-summary.txt` inside the bundle, and RBAC gaps are reported (with remediation guidance in `namespace-coverage.txt`) instead of failing the run. +- **Documentation**: [README-support-bundle.md](README-support-bundle.md) +- **Quick Start**: + ```bash + # Full bundle on an edge/cluster host (host collection requires root) + sudo bash support-bundle.sh + + # Cluster-only bundle from any machine with kubectl access (no root needed) + bash support-bundle.sh -H + + # Host-only bundle (no Kubernetes collection) + sudo bash support-bundle.sh -K + ``` +- Supports all flags of both legacy scripts, plus `-k `, `-K` (skip Kubernetes), `-H` (skip host), `-q` (quiet progress), and `-v` (print version). +- The legacy scripts below remain available as fallbacks until this script is widely adopted. + ### Edge Environment Support Bundle - **Script**: `support-bundle-edge.sh` - **Purpose**: Collects logs and diagnostic information from edge hosts and their Kubernetes clusters @@ -38,13 +56,19 @@ This repository contains a collection of scripts designed to gather diagnostic i Each script has its own detailed documentation: -1. [README-edge.md](README-edge.md) - Complete documentation for the edge support bundle script +1. [README-support-bundle.md](README-support-bundle.md) - Complete documentation for the consolidated support bundle script + - Collection scopes (`-K` / `-H`) and full flag reference + - Kubeconfig resolution order + - Collection summary statuses and advisory RBAC coverage + - Differences from the legacy scripts + +2. [README-edge.md](README-edge.md) - Complete documentation for the edge support bundle script - Detailed usage instructions - Available flags and options - Collection details - Prerequisites and dependencies -2. [README-infra.md](README-infra.md) - Complete documentation for the infrastructure support bundle script +3. [README-infra.md](README-infra.md) - Complete documentation for the infrastructure support bundle script - Detailed usage instructions - Configuration options - Collection details diff --git a/support-bundle/support-bundle.sh b/support-bundle/support-bundle.sh new file mode 100644 index 0000000..13e643a --- /dev/null +++ b/support-bundle/support-bundle.sh @@ -0,0 +1,2052 @@ +#!/bin/bash +# Copyright 2026 Spectro Cloud +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# Consolidated SpectroCloud support bundle collector. +# Supersedes support-bundle-edge.sh (host + cluster) and +# support-bundle-infra.sh (cluster only), which are kept alongside as +# fallbacks until this script is widely adopted. +# +# Collection is capability-driven: every collector is gated on the +# capabilities it needs (root, journald, crictl, a reachable API server, …) +# and records OK / SKIP / FAIL / DENIED into collection-summary.txt. +# A partial bundle always beats no bundle: the only fatal errors are +# failure to create the temp directory or write the archive. + +SB_VERSION=20260817+dev + +DEFAULT_KUBECONFIG="/run/kubeconfig" + +JOURNALD_LOGS=( + # edge-cluster + stylus-agent stylus-operator palette-tui + # agent-mode + spectro-stylus-agent spectro-stylus-operator spectro-init spectro-palette-agent-start spectro-palette-agent-initramfs spectro-palette-agent-boot spectro-palette-agent-network spectro-palette-agent-bootstrap + # system + systemd-timesyncd chronyd + # k8s + containerd spectro-containerd kubelet k3s k3s-agent rke2-server rke2-agent + # Canonical Kubernetes (snap-based) common units + k8s.kubelet k8s.kube-apiserver k8s.kube-controller-manager k8s.kube-scheduler k8s.kube-proxy + snap.k8s.kubelet snap.k8s.kube-apiserver snap.k8s.kube-controller-manager snap.k8s.kube-scheduler snap.k8s.kube-proxy + # Kairos specific services + cos-setup-boot + ) + +SYSTEM_NAMESPACES=(amd-gpu-operator capa-system capi-kubeadm-bootstrap-system capi-kubeadm-control-plane-system capi-system capi-webhook-system cert-manager default gpu-operator grafana harbor headlamp keycloak konveyor-forklift kube-system kube-public kubernetes-dashboard kubevirt launchpad-ai longhorn-system metallb-system os-patch palette-system piraeus-system reach-system rook-ceph spectro-system spectro-task system-upgrade traefik victoria-metrics vm-dashboard zot-system) + +API_RESOURCES=(apiservices clusterroles clusterrolebindings crds csr mutatingwebhookconfigurations namespaces nodes priorityclasses pv storageclasses validatingwebhookconfigurations volumeattachments) + +API_RESOURCES_NAMESPACED=(apiservices configmaps cronjobs daemonsets deployments endpoints endpointslices events hpa ingress jobs leases limitranges networkpolicies poddisruptionbudgets pods pvc replicasets resourcequotas roles rolebindings services serviceaccounts statefulsets) + +VAR_LOG_LINES=500000 + +# Collection scopes; -H disables host, -K disables kubernetes +COLLECT_HOST=true +COLLECT_K8S=true +QUIET=false + +# --------------------------------------------------------------------------- +# lib — logging, status registry, step runner +# --------------------------------------------------------------------------- + +function timestamp() { + date "+%Y-%m-%d %H:%M:%S" +} + +function techo() { + echo "$(timestamp): $*" +} + +STATUS_REGISTRY=() + +# record-status [note] +function record-status() { + STATUS_REGISTRY+=("$1|$2|${3:-}") +} + +# Collectors call `skip ` (or set STEP_STATUS/STEP_NOTE directly) +# before returning 0 to report a non-OK outcome to run_step. +function skip() { + STEP_STATUS=SKIP + STEP_NOTE="$*" + return 0 +} + +# run_step