Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
117 changes: 117 additions & 0 deletions support-bundle/README-support-bundle.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,117 @@
# Consolidated Support Bundle Collection Script

This Bash script (`support-bundle.sh`) collects diagnostic information from SpectroCloud environments. It consolidates the edge (`support-bundle-edge.sh`) and infrastructure (`support-bundle-infra.sh`) collectors into a single script that covers both scopes: host-level OS state and Kubernetes cluster state.

Collection is **capability-driven**: every collector is gated on the capabilities it actually needs (root, journald, a container runtime, a reachable API server, …) and the script **never aborts mid-run** — a partial bundle always beats no bundle. The only fatal errors are failure to create the temporary directory or write the archive.

## Quick Start

```bash
# Full bundle on an edge or cluster host (host collection requires root)
sudo bash support-bundle.sh

# Cluster-only bundle from any machine with kubectl access (no root needed)
bash support-bundle.sh -H

# Host-only bundle (no Kubernetes collection)
sudo bash support-bundle.sh -K
```

## Prerequisites

* **Host collection** (default): run as root (`sudo`). Use `-H` for a cluster-only bundle that does not require root.
* **Kubernetes collection** (default): `kubectl` access to the cluster. The kubeconfig is resolved automatically (see below) or can be passed explicitly with `-k`.
* All other tools (`journalctl`, `crictl`, `chronyc`, `etcdctl`, `helm`, GPU tooling, …) are optional — collectors that need a missing tool are recorded as `SKIP` and the run continues.

## Available Flags

All flags are optional:

| Flag | Description | Example |
|------|-------------|---------|
| `-d` | Output directory for temporary storage and .tar.gz archive | `-d /var/tmp` |
| `-K` | Skip all Kubernetes collection (host-only bundle) | `-K` |
| `-H` | Skip all host collection (cluster-only bundle; no root required) | `-H` |
| `-q` | Suppress per-resource progress output, keep the summary | `-q` |
| `-v` | Print the support bundle version and exit | `-v` |
| `-s` | Start day of journald log collection (days before now) | `-s 7` |
| `-e` | End day of journald log collection (days before now) | `-e 5` |
| `-S` | Start date of journald log collection | `-S 2024-01-01` |
| `-E` | End date of journald log collection | `-E 2024-01-01` |
| `-l` | Number of log lines to collect from journald/crictl logs | `-l 500000` |
| `-j` | Additional journald logs to collect | `-j cloud-init,systemd-resolved` |
| `-k` | Path to an explicit kubeconfig | `-k /etc/kubernetes/admin.conf` |
| `-n` | Additional namespaces to collect | `-n hello-universe,hello-world` |
| `-r` | Additional namespace scoped resources | `-r certificates.cert-manager.io` |
| `-R` | Additional cluster scoped resources | `-R clusterissuers.cert-manager.io` |

## Kubeconfig Resolution

The first readable entry wins; resolution is never fatal:

1. `-k <path>` flag
2. `$KUBECONFIG` environment variable
3. `/run/kubeconfig`
4. `/etc/kubernetes/admin.conf`
5. `$HOME/.kube/config`
6. The invoking user's `~/.kube/config` when running under `sudo`
7. `/etc/rancher/rke2/rke2.yaml`, `/etc/rancher/k3s/k3s.yaml`, `/var/snap/k8s/current/credentials/admin.conf`

If none is found (or the API server is unreachable within a 10s probe), Kubernetes collection is recorded as `SKIP` and the rest of the bundle is still produced.

## Collection Summary

Every collector reports its outcome into `collection-summary.txt` inside the bundle (and to stdout at the end of the run):

| Status | Meaning |
|--------|---------|
| `OK` | Collector ran successfully |
| `SKIP` | Precondition absent (tool not installed, not an edge host, scope disabled, …) |
| `FAIL` | Collector ran and errored |
| `DENIED` | Blocked by RBAC |

## RBAC Coverage (advisory)

Before Kubernetes collection, the script checks `kubectl auth can-i` for the required cluster-scoped resources and pod access per targeted namespace, and writes the result table to `namespace-coverage.txt` inside the bundle.

Unlike previous script versions, **insufficient RBAC never aborts the run**: denied namespaces are pruned from collection, recorded as `DENIED`, and remediation guidance (a minimal ClusterRole snippet) is printed — then collection continues with what is accessible.

## Output

Bundle naming and archive location match the legacy scripts:

| Run scope | Bundle name | Archive location |
|-----------|-------------|------------------|
| Host in scope (default, or `-K`) | `<hostname>-<timestamp>.tar.gz` | Temporary base directory (`mktemp -d`, typically under `/tmp`; `-d <dir>` relocates it) — edge hosts often have read-only partitions, so the archive is never written to the current working directory |
| Cluster-only (`-H`) | `<cluster-name>-<timestamp>.tar.gz` | `-d <dir>` if given, else the current working directory, else the temporary base directory if neither is writable |

Every bundle contains `console.log` (the full run transcript), `.support-bundle` (version, scopes, detected capabilities), and `collection-summary.txt`.

## Collected Information

Identical in scope to the union of the two legacy scripts:

* **Host tier** (requires root; skipped with `-H`): system info, chronyd/time sync, networking (iptables/nft/ip/ss/CNI), `/var/log` and `/var/log/spectrocloud`, journald units (including previous-boot kernel log), storage state (block devices, LVM, NVMe SMART, device-mapper), GPU state (AMD ROCm + NVIDIA, with `kubectl exec` fallbacks into operator pods), edge agent files (`/oem`, `/run/stylus`, cloud-config, installer logs, bundle checksums), container runtime (crictl), and helm releases.
* **Kubernetes tier** (skipped with `-K`): cluster info and dump, cluster- and namespace-scoped resources, custom resources, helm release secrets, metrics, previous pod logs, and — on Enterprise/PCG clusters — MongoDB replica set status, per-pod disk usage, and database/collection sizes.
* **Distro tier** (host + detected distribution): kubeadm manifests/certs/etcd, k3s/rke2 pod logs and certs, Canonical snap k8s files/dqlite state.

Secrets are not collected, except helm release secrets for the spectro namespaces. Certificates are captured parsed (`openssl x509 -text -noout`), never as raw keys.

## Environment Variables

* `KUBECONFIG`: Path to the Kubernetes configuration file (see resolution order above)
* `DEV`: When set, bypasses the root requirement (development/testing only)

## Relationship to the Legacy Scripts

`support-bundle-edge.sh` and `support-bundle-infra.sh` remain available as fallbacks until this script is widely adopted:

* `support-bundle.sh` ≈ `support-bundle-edge.sh` (full scope, root required)
* `support-bundle.sh -H` ≈ `support-bundle-infra.sh` (cluster-only, no root)

Behavioral differences vs. the legacy scripts:

* Insufficient RBAC and a missing/unset `KUBECONFIG` no longer abort the run (previously fatal in both scripts).
* Every run produces `collection-summary.txt` and, when a cluster is reachable, `namespace-coverage.txt`.

Bundle naming and archive locations are unchanged from the legacy scripts (see [Output](#output)).
28 changes: 26 additions & 2 deletions support-bundle/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,24 @@ This repository contains a collection of scripts designed to gather diagnostic i

## Available Scripts

### Consolidated Support Bundle (preview)
- **Script**: `support-bundle.sh`
- **Purpose**: Single script covering both the edge (host + cluster) and infrastructure (cluster-only) scopes. Collection is capability-driven and never aborts mid-run: each collector records OK / SKIP / FAIL / DENIED into `collection-summary.txt` inside the bundle, and RBAC gaps are reported (with remediation guidance in `namespace-coverage.txt`) instead of failing the run.
- **Documentation**: [README-support-bundle.md](README-support-bundle.md)
- **Quick Start**:
```bash
# Full bundle on an edge/cluster host (host collection requires root)
sudo bash support-bundle.sh

# Cluster-only bundle from any machine with kubectl access (no root needed)
bash support-bundle.sh -H

# Host-only bundle (no Kubernetes collection)
sudo bash support-bundle.sh -K
```
- Supports all flags of both legacy scripts, plus `-k <kubeconfig>`, `-K` (skip Kubernetes), `-H` (skip host), `-q` (quiet progress), and `-v` (print version).
- The legacy scripts below remain available as fallbacks until this script is widely adopted.

### Edge Environment Support Bundle
- **Script**: `support-bundle-edge.sh`
- **Purpose**: Collects logs and diagnostic information from edge hosts and their Kubernetes clusters
Expand Down Expand Up @@ -38,13 +56,19 @@ This repository contains a collection of scripts designed to gather diagnostic i

Each script has its own detailed documentation:

1. [README-edge.md](README-edge.md) - Complete documentation for the edge support bundle script
1. [README-support-bundle.md](README-support-bundle.md) - Complete documentation for the consolidated support bundle script
- Collection scopes (`-K` / `-H`) and full flag reference
- Kubeconfig resolution order
- Collection summary statuses and advisory RBAC coverage
- Differences from the legacy scripts

2. [README-edge.md](README-edge.md) - Complete documentation for the edge support bundle script
- Detailed usage instructions
- Available flags and options
- Collection details
- Prerequisites and dependencies

2. [README-infra.md](README-infra.md) - Complete documentation for the infrastructure support bundle script
3. [README-infra.md](README-infra.md) - Complete documentation for the infrastructure support bundle script
- Detailed usage instructions
- Configuration options
- Collection details
Expand Down
Loading